Customer impact

From 15:00 to 16:38 UTC on 2026-10-02, Cloud Robots jobs in the US region took longer than usual to start, up to 22 minutes at the peak around 16:20 UTC. Orchestrator stayed available, no job was lost, no other region was affected. A small number of customers received a temporary limit error on new submissions while their earlier jobs were waiting.

Detection

Automated alerts fired at 15:07 UTC, seven minutes after impact began. Checks in the US region failed from about 15:32 UTC and the incident response started.

Root cause

At 14:55 UTC one customer's workflow began submitting jobs at about 20 times its normal rate. The service that admits jobs keeps a per-customer concurrency count in the database and updates it under a database lock on every job start and end. At that rate the lock queue for the customer's record grew until updates timed out. The waiting updates occupied the shared pool of workers that starts jobs for all customers in the region, so every customer's jobs waited behind them. The mechanism meant to isolate one customer's load became the bottleneck.

Response

The team identified the overloaded admission path and the workload behind it. A first configuration change at 15:46 UTC targeted the wrong record and had no effect. The correct change at 16:10 UTC exempted that customer's job type from the database lock and took effect at about 16:20 UTC. Start times dropped from about 13 seconds to under one second and the backlog of about 5200 jobs drained in 18 minutes. Job starts were normal at 16:38 UTC. The customer's trigger was switched off with the customer at about 16:40 UTC.

Follow up

The database-locked rate limiting is replaced by in-memory high performance rate limiting mechanism, so one customer's load cannot block job starts for others. The change is complete, tested in pre-production at the same load, and being rolled out.