Multiple services are experiencing intermittent issues across multiple regions

Customer Impact

Between February 4, 11:34 UTC and February 4, 03:00 UTC, UiPath customers located in the UK region, as well as customers connecting via VPN endpoints with UK-based IP addresses, experienced service disruptions affecting Orchestrator and related services.

During this period, users encountered multiple errors, including failed authentication, asset retrieval failures, and general unavailability of core automation services, resulting in degraded platform access.

Root Cause

The outage happened because our cloud provider changed their API Gateway in the UK. This update added a system that limits how much traffic goes through the system. This caused UiPath traffic to slow down and cause problems with service.

Detection

Automated monitoring quickly detected the issue, alerting Engineering staff to elevated rates of HTTP 503 errors in the UK region.

Response

To mitigate the impact, we updated our network configuration, and re-routed all UK traffic to our European API Gateway, which had not yet received the new update. This ensured that customer traffic was no longer subject to throttling behavior and restored service availability.

Follow-up and preventive actions

We are working with our cloud provider to learn more about the new throttling feature.

To stop similar events from happening again, we have done the following.

• Requested our cloud provider to pause the rollout of the new feature and disable the throttling functionality in regions where it has already been deployed.
• Moved all production API Gateways into the cloud provider's delayed rollout group. We also assigned alpha API Gateways to the early rollout group. This lets us find and fix problems earlier in the release cycle.