Customer Impact

Between 2026-01-21 18:00 UTC and 2026-01-23 15:43 UTC, background re-inference of IXP Unstructured and Complex Documents (IXP-UCD) design-time predictions was unavailable in EU and US regions. During this period, customers viewing IXP-UCD documents did not see pre-computed predictions, and background prediction error rates appeared as 100%.

Root Cause

The incident was caused by a significant increase in IXP-UCD design-time load in the EU and US regions. This increased load led to a growing backlog of background inference jobs, which in turn caused newly uploaded documents to take over an hour to appear in the IXP-UCD documents table.

The root issue was that document uploads and background re-inference processing were coupled together, meaning that when background jobs became backlogged, they blocked new documents from appearing promptly for users.

Detection

Slow document uploads were spotted by the engineering team through internal use of the platform.

Response

Upon detection:

• The engineering team identified that background re-inference was causing significant delays in document availability.
• To prioritize core functionality, background re-inference of design-time predictions was temporarily disabled. This restored timely document uploads but eliminated pre-computed predictions during the affected period.
• The team then worked to re-enable automatic updating of design-time IXP-UCD predictions so users could see pre-computed predictions and meaningful error rates.
• A fix was deployed to ensure that newly uploaded documents appear immediately in the documents table without being affected by background processing.

Services were fully restored by 2026-01-23 15:43 UTC, with both document uploads and background predictions functioning normally.

Follow-Up

To prevent similar incidents in the future, we are taking the following steps.

Short term:

• Decouple document uploads from background re-inference to ensure core workflows are not impacted by background processing.

Long term:

• Improve load management and capacity handling for IXP-UCD design-time workflows to ensure consistent performance during traffic spikes.
• Introduce additional safeguards and circuit breakers to prevent background processing from impacting core user workflows.