Background context

UiPath replicates authorization data from Orchestrator and Data Service into a centralized authorization store as a building block for a unified administration experience across UiPath services, which is used in deployment, publishing and listing of UiPath Apps.

Customer impact

67 organizations in Europe and 58 organizations in our community environment experienced an increase in access denials to Apps due to this incident. These customers saw authorization errors while deploying, publishing, or executing UiPath Apps.

Please note that the regions indicate the region the organization is hosted, not where the tenant is hosted.

Root cause

During the incident, we found that the authorization data was not correctly syncronized between Orchestrator, Data Service, and the central Authorization service. This resulted in an incomplete permission set for the UiPath Apps use cases.

We believe that the data divergence was caused by a race condition within the sync process. Specifically in a tool that is designed to proactively auto-heal any issues between the various UiPath services.

Detection

Unfortunately, UiPath did not self-detect this issue. It was reported by customers, and we reported it in the UiPath status page when more than 1 customers reported this issue.

Response

The team used telemetry to see which organizations had elevated authorization failures for apps publishing, and we remediated the issue by forcing a sync of all the permission data to the centralized authorization store.

Follow up

To ensure this doesn't happen again, we plan to make many improvements to the service

1. Improve detection by adding the necessary telemetry and monitoring to immediately catch and root cause any data de-synchronization.
2. Review the sync architecture, and build more robustness into the sync architecture that prevents the divergence.
3. Invest in better stress testing of the synchronization process to expose race conditions in our architecture.