Customer impact

Between July 20, 2026 at 4:28 pm UTC and 7:32 pm UTC, a subset of customers experienced degraded Orchestrator Jobs performance in the U.S. region, with delays in job processing during that window.

Root cause

The incident was triggered by a routine product deployment to Orchestrator that included a schema change to an existing column on a core Orchestrator database table. Specifically, a change to the column's size. The large size of the Jobs table made this schema change operation significantly more resource-intensive and longer-running than anticipated, causing it to still be in progress once normal U.S. traffic picked up, an overlap that had not been anticipated. The combination of the ongoing schema change and concurrent production traffic generated increased database load, which caused latency and delays in job processing. As the change rolled out, the affected databases were impacted progressively rather than all at once.

The script had been reviewed and tested prior to rollout. However, the resource impact observed here did not reproduce during pre-deployment testing, as it depended on production-scale traffic conditions that were not present in the test environment.

Detection

Our automated monitoring identified the issue at 4:28 pm UTC on July 20, 2026, and the alert was acknowledged immediately. As the extent of the customer impact became clear, the incident was formally declared customer-impacting at 7:12 pm UTC.

Response

As the schema change progressed, the resulting database load increased progressively across the affected databases rather than all at once. The team mitigated the issue in two stages: a first round of database capacity increases around 5:00–5:30 pm UTC addressed the databases affected at that point, and as impact continued to build on the remaining databases, a second round of capacity increases around 7:00–7:30 pm UTC was applied, fully resolving the elevated load.

At 7:19 pm UTC, a public status update confirmed the cause had been identified: degraded performance affecting Orchestrator Jobs in the U.S. region, with users potentially seeing delays when executing jobs. Mitigation actions were already under way at that point.

The database schema change script was stopped, and database capacity was increased as a mitigation measure. During the response call, the engineering team monitored database utilization, which was observed dropping and then stabilizing. By the end of the call, all databases were underutilized and the issue was confirmed mitigated. At 7:32 pm UTC, the incident was marked resolved, and the public status page confirmed that Orchestrator performance had returned to expected levels

Follow-up

Several actions will be taken following this incident:

• We are building a safer, more automated process for running database schema changes that reliably schedules resource-intensive changes outside of peak traffic hours, rather than relying on manual timing.
• We are extending our monitoring systems to more accurately include customer-visible slowdowns in parts of the platform that rely on Orchestrator (Maestro, Agents).