UiPath uses feature flags to control the rollout of features separately from the rollout of new versions of our code. This gives us more control for canarying, testing, dogfooding, slow rollouts, and more.
Currently, we are migrating from one feature flag system to another.
At 22:23:49 UTC on Tuesday, Jun 25, 2024, customers across all regions were unable to log into the UiPath Automaiton Cloud Portal. They received the error message “Cannot find organization info.” The issue was resolved 28 minutes later.
The feature flag migration was first deployed successfully in alpha, then staging, then community, and onwards through each of our regions. After the migration was fully rolled out, a cleanup was performed on the feature flag confiuration files. At this time a typo was accidentally introduced. Our automatic validation did not catch the problem before deployment.
The issue was detected by our automatic monitoring within minutes. The on-call engineer was notified and immediately began troubleshooting.
The invalid feature flag config change was rolled back. The change was propagated to all production regions within 10 minutes.