When a CVE Fix Breaks Your Clients: Ceph RGW, SigV4 and an Unsigned Content-Type

On August 19th the Ceph project published 20.2.4 and 19.2.6, two hotfix releases closing four security issues. One of them, CVE-2026-54330, let anyone holding a pre-signed upload URL attach headers the issuer never authorised. We treated it as urgent and started rolling it out the next day, region by region and RGW node by RGW node, monitoring and testing between each: HEL1, then EYL1, then FRA2.

The upgrade looked clean. Our own tooling kept working: rclone, the AWS CLI, MinIO's mc, everything we run ourselves. Monitoring stayed green. What none of it showed was that a small share of requests on that cluster, around 5 percent, had started failing, while the hundreds of customers whose clients signed the request correctly carried on without noticing a thing.

A couple of hours after the last gateway finished upgrading, one customer opened a ticket. Uploads failing with 403 AccessDenied, roughly one in three of theirs, no pattern they could see. Same credentials, same bucket, objects that had uploaded fine the day before. Reads worked. Listings worked. Deletes worked.

The Cause

AWS Signature Version 4 has the client declare which headers it signed, in a SignedHeaders list, and the server recomputes the signature over exactly those. CVE-2026-54330 was that RadosGW still honoured extra x-amz-* headers that were absent from that list and carried no signature, so someone holding a pre-signed URL could bolt on, for example, x-amz-acl: public-read. The fix in 20.2.4 and 19.2.6 is blunt: if a header is present but not in SignedHeaders, reject the request.

That reached further than the vulnerability. Content-Type is not an x-amz-* header and carries no authorisation meaning, but it is still a header, and many clients send it without signing it. The AWS spec says a present Content-Type should be added to the canonical headers, but S3 has never enforced it, so a whole class of clients has sent it unsigned for years. aws-sdk-php only began signing it in 3.385.3. On 20.2.4, uploads from anything older get a 403.

The gateway log made it look like the customer's problem: no user, no bucket, nothing received, because signature verification failed before the request was ever tied to an account. Their credentials were fine. A packet capture showed the real cause, a Content-Type: image/webp header that the client had left out of its SignedHeaders.

We keep one gateway permanently held back on the previous release, out of service, for exactly this kind of question. So we could replay the byte-identical request against old and new, with the same key, seconds apart:

20.2.3   PUT, unsigned Content-Type   ->  200 OK
20.2.4   PUT, unsigned Content-Type   ->  403 AccessDenied
20.2.4   PUT, signed Content-Type     ->  200 OK

The Fix

The gateway ships an option, rgw_sigv4_insecure, that restores the previous header-verification behaviour. It applied at runtime, with no restart, and the customer's uploads recovered immediately. We chose it over reverting the release outright, which would have dropped the three other security fixes that shipped alongside it.

This is a deliberately temporary, controlled measure. The corrected code is merged in Ceph's development branch, but it has not yet been backported to the release line we run, and no backport is queued for it yet. So there is no fixed build for us to move to today. We are waiting on a future upstream release, tracking the backport, and holding the mitigation under our own control until it lands.

We also looked at stripping the unsigned Content-Type at the proxy. It would have fixed every upload and silently rewritten stored objects from image/webp to binary/octet-stream. The customer serves those images to browsers straight from the bucket, so that cure was worse than the problem.

What We Are Changing

The gap was not that our tests were thin. It was that they tested our software with our own clients, which sign correctly, so they could never reproduce a client-side mistake. We built zero-z3-prober: hourly, per gateway, it replays a fixed set of signed and unsigned requests and checks each against an expected result, so a per-node behaviour change like this one surfaces within the hour instead of in a customer ticket.

We are honest about its limit. It encodes the failures we already understand, and it will not catch the next tricky bug, the one we have not thought to write down. So we are also adding an alert on each gateway's own error rate against its recent baseline, so an unexplained jump like this 5 percent pages us on its own, without needing a matching test case.