From 2024-04-03 23:45 UTC to 2024-04-04 02:35 UTC our customers experienced errors when accessing some of the services located in the US region of Automation Cloud. Impacted products include Automation Cloud, Orchestrator, Automation Hub, Automation Ops, Document Understanding, Serverless Robots, Cloud Robots - VM, Solutions Management, and Insights.
UiPath makes extensive use of Azure SQL. At the beginning of the outage, Microsoft performed routine SQL maintenance in the East US region. Typically this is done without any visible impact to our customers. But for some reason this maintenance caused the SQL Databases in this region to become unavailable. We are still waiting for a root cause from Microsoft and will update this document once we receive it.
Automated alerts immediately detected the issue and notified UiPath on-call engineers. They confirmed the scope of the outage and updated status.uipath.com.
After a brief investigation, we determined that the problem was with Azure SQL. We reached out to Microsoft Support to request assistance.
For the US region of all UiPath products, we place the primary database in Azure’s East US region and a failover database in Azure’s West US region. By default, Azure will failover from primary to secondary after the primary is unavailable for 60 minutes.
During this incident, most databases automatically failed to the secondary region. Unfortunately, the Orchestrator, Automation Hub and Insights databases did not. The UiPath engineers investigated the databases and began to trigger a manual failover, but by that time Microsoft had resolved the underlying issue in the East US region.