Customer impact

Between July 30, 2026 at 10:00 pm UTC and July 31, 2026 at 1:28 am UTC, a subset of customers using Helix models for Document Understanding in the US experienced degraded performance, including failed requests returning HTTP 503 errors and client-side timeouts.

The impact lasted approximately 3 hours and 28 minutes. Some requests continued to process successfully, but affected customers may have seen retries, delayed processing, or failed document extraction and classification jobs.

Root cause

The root cause was duplicate re-processing of the same document pages during a period of high retries. Successful processing results were cached; when the queue was full, failed submissions were not cached. Repeated retries re-entered processing instead of being deduplicated. During the incident window, duplicate processing accounted for the majority of compute work, which drove sustained resource saturation and caused additional 503 responses - which increased load and led to additional 503 responses.

Detection

Automated alerting detected the first queue overflow errors began around 10:00 pm UTC.

Response

At 10:44 pm UTC, we posted an investigation notice for degraded Extraction and Classification performance in the US. By 11:26 pm UTC, we freed capacity by scaling down an unused processing path and freeing additional capacity for the affected Document Understanding workload.

Follow up

1. We plan to implement short-lived failure caching for repeated document submissions, so that duplicate retries after a queue overflow are rejected early instead of re-entering expensive processing.
2. We are improving monitoring and alerting to detect this class of issue sooner.