Customer Impact

Between March 17, 2026 at 04:00 UTC and March 17, 2026 at 10:52 UTC, a subset of customers experienced intermittent slowness and error responses (HTTP 503) when running automation workflows that use our document understanding and communications mining features. The total impact window spanned approximately 6 hours and 52 minutes, though failures were not continuous: they occurred in three distinct waves at approximately 04:00, 07:00 and 09:00 UTC, and the first two waves recovered on their own without intervention. Affected customers saw automations run slowly, and some workflows failed to complete. The issue affected customers in the US region.

Root Cause

Before an automation can process documents, it must confirm the user is allowed to run it. That permission check goes to a separate service, and document processing waits up to 15 seconds for an answer before treating the request as failed. One of the servers handling those checks stopped responding for about three minutes, while absorbing an hourly surge in traffic. Normally traffic moves to a healthy server automatically, but three protections failed to do so: the check that removes an unhealthy server was too slow to react, the automatic switch to another server had been turned off earlier while we addressed an unrelated problem, and there was no cap on how many requests one server could be asked to handle — so retries from other systems piled up on the server already struggling. Permission checks then sat waiting 20 to 30 seconds, even though the service answered each one in a fraction of a second once it received them. That exceeded the 15-second limit, so document processing gave up and returned errors, which customers saw as HTTP 503 responses and failed workflows.

Detection

The issue was detected at 07:54 UTC on March 17, 2026 by automated alerting on request failures in our document processing system, and an incident was declared immediately. That was approximately 3 hours and 54 minutes after the earliest occurrence at 04:00 UTC, which was identified retrospectively during post-incident analysis. The delay occurred because the first two waves of failures resolved on their own before any alert threshold was sustained long enough to fire. Existing alerting monitored request failures in the calling system rather than latency between internal services, so the underlying slowdown was not visible until it produced enough failures to trigger an alert.

Response

At 07:57 UTC, the team confirmed the scope as a performance degradation in the US region affecting automations that depend on the permission-check path, with slowness and HTTP 503 responses observed.

By 09:29 UTC, analysis of individual sample transactions established that request routing was taking 20, 30 and 25 seconds, while the authorization service's own processing time in those same transactions was 5.3 milliseconds, 30 milliseconds and 739 milliseconds. This conclusively ruled out the authorization service as the bottleneck and directed the investigation to the routing layer between the two systems. Network and infrastructure engineers joined the response at 09:43 UTC to examine routing-layer behaviour.

At 10:24 UTC, a mitigation was applied: the document processing system's timeout for permission-check requests was increased from 15 seconds to 60 seconds, so that transient delays would slow requests rather than fail them. A service hotfix was also prepared during the incident. Following the mitigation, telemetry showed no further failures, and the incident was marked resolved at 10:52 UTC.

Follow Up

• The timeout for permission-check requests has been increased so that transient routing delays no longer cause workflow failures.

• Capacity benchmarking of the authorization service, with traffic limits applied at the routing layer so that no single instance can be overwhelmed by concurrent requests.

• Tighter timeouts in the authorization service's routing configuration, so a slow instance fails fast and traffic moves to a healthy one instead of queueing.

• A startup health check on the authorization service, so an instance is only sent traffic once it is genuinely ready to serve it.

• Investigation into the recurring hourly traffic surges that preceded the degradation, to establish their origin and remove them as a trigger.