Customer Impact

Between June 17, 2026 (~06:15 UTC) and June 18, 2026 at 14:22 UTC, customers using IXP in the Switzerland region were unable to reliably request document extractions. Affected users saw failed extractions and error messages when annotating documents, and no model training was triggered.

The impact was limited to the Switzerland region and affected a single customer, as regional traffic was very low during this period. Customer-facing failures were occurring from the morning of June 17; the issue was formally tracked and resolved as an incident on June 18.

Root Cause

The incident was caused by a deployment configuration issue in the Switzerland region that prevented several GPU-backed IXP service components from starting correctly. This left key document processing services unavailable, resulting in failed document extraction and annotation requests. Existing monitoring did not identify the regional service degradation, delaying detection until the affected customer reported the issue.

Detection

This issue was not caught by automated monitoring as expected. Our monitoring reported a number of lower-severity issues during this period but did not surface the more significant region-level outage, so the failure was instead first identified through a customer report on June 17 describing failed document annotation in a Swiss tenant. A formal incident was declared on June 18 at 10:46 UTC, and a public status-page update followed at 11:17 UTC.

Response

Following the customer report on June 17, engineering and SRE teams began investigating the issue affecting GPU-backed IXP services in the Switzerland region. Initial attempts to restore service by restarting the affected instances were unsuccessful, and the team continued working directly with the impacted customer while investigating the underlying cause.

On June 18, a formal incident was declared and a public status update was published. The investigation identified unhealthy service instances caused by an incorrect deployment configuration. The affected services were restarted with the corrected configuration, a configuration patch was applied to the remaining components, and a corrected service release was deployed. Service health was monitored after recovery, and the incident was resolved once the platform remained stable

Follow-Up

Planned

• Improve monitoring: Implement region-level health monitoring and alerting to detect similar failures automatically, reducing reliance on customer reports for detection.
• Strengthen deployment validation: Add deployment validation and regression tests to detect missing or invalid configuration before changes reach production.
• Enhance release safeguards: Review and improve deployment processes to ensure required configuration is consistently applied across all regional service instances.