# Illustrative incident review: image delivery backlog

On 17 April, publishers in the EU region saw image uploads stall between 09:14 and 10:07 UTC. Existing images still loaded from the CDN, but new posts remained in a processing state. At the peak, 1,842 image jobs were waiting and the oldest job had been queued for 19 minutes. No source documents or published posts were lost.

## Timeline

At 09:14, the media queue alert crossed its ten-minute threshold. Support received the first report six minutes later. The on-call engineer checked CDN error rates and regional network health because the user-visible symptom was a missing image. Both were normal. At 09:32, a worker log sample showed repeated storage 403 responses. The retries were safe to replay, but they filled the worker pool and hid the original authorization failure behind queue delay.

## Cause

The credential rotator promoted a new storage key at 08:58. The upload API read the new key immediately, while long-lived image workers kept the previous key in memory. Storage rejected their writes after the old key expired at 09:11. Each failed job retried three times, so the queue grew faster than the workers could drain it. The rotation runbook checked the upload API but did not exercise a background image job.

## Recovery

At 09:41, the team paused new retries, restarted the image workers so they loaded the current key, and replayed the idempotent jobs from the queue. Storage writes succeeded on the first attempt. Queue age returned below one minute at 10:07 UTC, and support confirmed that the affected posts rendered their images. The team kept the incident open for another hour to watch error rates and queue depth.

## Follow-up

The media team owns a worker-side credential reload before the next rotation. The platform team will add a rotation test that uploads and processes an image, plus an alert on storage 403 responses before queue age rises. The incident review is scheduled for 21 April; no customer action is required.