[de-west-01] SZ1 & S3 Ceph Storage Maintenance

Scheduled Maintenance Report for WIIT AG

Postmortem

Summary

After upgrading our Ceph cluster, customers reported S3 requests failing with HTTP 403 (Access Denied). The cause is a behavioral change introduced by the fix for CVE-2026-54330, a flaw in RGW's SigV4 signature verifier. RGW now rejects SigV4 requests where the host header or any x-amz-* header is not part of the signed header subset.

The stricter validation is a legitimate tightening of signature verification, but it surfaced pre-existing non-conformant signing behavior in client applications.
Correctly signed requests were never affected.

Mitigation

We disable the additional header validation and restored the previous behavior, resolving the 403 errors.

Action Required by Customers

The workaround is temporary. Please ensure your S3 clients include the host header and all x-amz-* headers in the SigV4 signed header list. Current AWS SDKs and S3 client libraries do this by default — the likely sources of non-conformant requests are custom signing implementations, older SDK versions, and proxies that inject headers after signing.

Please treat this change as urgent; we will inform you about the next steps as soon as possible.

We apologise for the inconveniences caused.

Posted Aug 27, 2026 - 17:09 CEST

Completed

The scheduled maintenance has been completed.
Posted Aug 27, 2026 - 16:00 CEST

In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Aug 27, 2026 - 10:30 CEST

Scheduled

Description:
We will perform a emergency maintenance on the Ceph storage cluster in the
SZ1 Availability Zone of the de-west-01 region to resolve a critical
security issue.

Impact:
- No impact is expected while the individual OSD nodes are upgraded sequentially.
- Towards the end of the maintenance window, the Ceph RGW component
serving the S3 and Swift services will be restarted. Clients may
experience brief connection resets during this phase.

Required Action:
No action is required. We recommend ensuring your applications are
configured to handle transient connection errors with automatic retries.

We apologize for the short notice and any inconvenience this may cause.
Posted Aug 25, 2026 - 14:15 CEST
This scheduled maintenance affected: de-west-01 OpenStack Services (OpenStack Storage (S3 /Swift)) and de-west-01 OpenStack SZ1 (OpenStack Block Storage SZ1).