Customer Impact

Between April 22, 2026 1:28 pm UTC and April 22, 2026 1:52 pm UTC, some customers in the US region were unable to perform document extractions using IXP. During this window, all extraction and prediction requests against pinned model versions returned HTTP 404 Not Found errors. We sincerely apologize for the disruption this caused to your operations.

Root Cause

The incident was triggered by a deployment of a new release of IXP in the impacted region. A targeted code change that required a manual preparation step to be completed before deployment was released before the manual step was run. In environments with more frequent release schedules, this step was handled automatically by an existing part of the deployment process. However, the US and EU regions follow a less frequent, consolidated release schedule, so this automation was not present and the required manual step was missed.

As a result, several components of the model management system began reading data from an incorrect location. Because the required data did not exist, requests for extractions and predictions against pinned model versions returned 404 Not Found errors.

Detection

Automated monitoring detected the issue after the deployment completed, at approximately 1:28 pm UTC on April 22, 2026. These monitoring checks, which continuously validate core user workflows including extraction against pinned model versions, began failing as soon as the service started returning errors.

Response

Upon detection, our engineering team identified the root cause and manually ran the required step against the affected environment, restoring the model management system to read from the correct location.

After a sustained monitoring period to verify stable recovery—including confirmation that automated monitoring checks resumed passing and that traffic returned to normal response patterns—the incident was confirmed as fully resolved at 3:22 pm UTC.

Follow-Up

The IXP delayed release process will be reviewed, with a specific focus on why this issue was not seen in earlier regular releases.