Customer impact

From 2024-08-14 8:02:00 UTC to 2024-08-14 11:00 UTC our customers experienced errors when accessing some of the services located in the EU region of Automation Cloud. Impacted products include Automation Cloud, Orchestrator, Automation Hub, Automation Ops, Document Understanding, Serverless Robots, Cloud Robots - VM, Solutions Management, and Insights.

Root cause

A critical identity component that is managing the JWT token signing keys failed to initialize on startup because of a missing configuration. This startup failure was ignored and led to an incomplete OIDC connect discovery document (document was missing the "jwks_uri" link). As a result, UiPath services that depended on the OIDC discovery document to validate JWT tokens, were unable to load the signing keys and therefore were unable to perform authentication.

Detection

Automated alerts immediately detected the issue and notified UiPath on-call engineers. They confirmed the scope of the outage and updated UiPath Status .

Response

After a brief investigation, we determined that the issue was caused by a subset of the Identity pods and the faulty pods were restarted, restoring UiPath service.

Follow up

• Fixed the startup code to handle missing configuration correctly
• Added new Kubernetes startup probe to ensure that Identity pods are fully initialized before they are brought into the rotation