Degraded Performance – Orchestrator & Document Understanding Services (East US)

Incident Report for UiPath

Postmortem

Customer impact

Between March 26, 2025, at 14:47 UTC, and March 26, 2025, at 20:00 UTC, two UiPath Automation Cloud services had partial outages in the U.S. region. Document Understanding customers may have experienced errors or increased latency while using the digitization capabilities. In Orchestrator, a subset of the tenants saw latency and some failed requests.

Root cause

Azure declared an incident with their Virtual Machine service at 13:30 UTC. This did not immediately impact Automation Cloud. But later, our traffic increased during U.S. business hours. Document Understanding and Orchestrator were unable to scale up to handle this extra traffic. By 14:47 UTC, this lack of capacity had caused latency and errors.

Detection

The increased latencies alerted us at 15:02 UTC. We immediately started investigating. During the investigation, customers reached out to Support to report degraded performance.

Response

Azure's incident prevented the allocation of new resources.

Document Understanding moved traffic to existing resources that had spare capacity. It was running at full capacity by 17:45 UTC, and had finished processing the backlog.

Orchestrator re-created their infrastructure using resources that Azure's issue did not affect. They then switched traffic to the new instance. The Orchestrator latency issues were mitigated by 20:00 UTC.

Follow up

We are working with Azure to see what can be done to avoid these sorts of incidents. We are also looking at our own tools and processes to make it faster to detect and fix infrastructure problems.

Posted Apr 04, 2025 - 16:59 UTC

Resolved

This issue has been mitigated. Thank you for your patience.
Posted Mar 26, 2025 - 21:26 UTC

Monitoring

The issue has been mitigated. We will continue to monitor closely to ensure stability. Thank you for your patience.
Posted Mar 26, 2025 - 20:36 UTC

Update

We are actively working to resolve the issue. Thank you for your patience.
Posted Mar 26, 2025 - 19:56 UTC

Investigating

We are currently experiencing degraded performance with the Orchestrator and Document Understanding services in the East US region due to an issue with a third-party dependency.
Posted Mar 26, 2025 - 16:32 UTC
This incident affected: Orchestrator and Document Understanding.