Customer impact

Between July 1, 2026 at 3:38 pm UTC and July 1, 2026 at 6:58 pm UTC, a subset of customers experienced increased latency and degraded performance when using Orchestrator in U.S. region. Some requests may have timed out during this period. The customer-impacting period lasted about 40 minutes.

Root cause

During a routine service update, a frequently-used database query was improved for better performance. The update to the query was correct; however, it caused the database to generate a new execution plan for that query. This new plan was built using a non-representative dataset, so it was not well-suited to production-scale data. As a result, the query began executing inefficiently, which drove a sustained increase in database CPU. The elevated CPU led to the query timeouts, increased request latency, and job execution delays that customers experienced during the incident.

Detection

The issue was detected through an automated availability alert at July 1, 2026 at 3:23 pm UTC. A customer-impacting incident was declared on July 1, 2026 at 3:39 pm UTC.

Response

By July 1, 2026 at 3:39 pm UTC, the affected Orchestrator database was identified as needing additional capacity. At 3:41 pm UTC, the decision was made to scale the database and update the public status page to indicate degraded performance and possible timeouts. The first public status page update was posted at 3:50 pm UTC, noting increased latency and degraded performance for Orchestrator in the U.S. region.

By 3:59 pm UTC, the database scaling had completed and latency was confirmed to have returned to normal. At 4:02 pm UTC, the incident was marked as mitigated. A monitoring update was posted to the public status page at 4:10 pm UTC, confirming that mitigation had been applied and performance was improving, while noting that users might still experience some degraded performance.

Monitoring continued after mitigation. The incident was formally resolved at 8:01 pm UTC.

Follow-up

• Activating query changes independently of deployment, so execution plans are generated against representative production data.
• Tooling to detect and clear inefficient execution plans automatically, so the database can regenerate a better one.
• Monitoring to flag when a single query consumes a disproportionate share of database CPU, so we can act before performance is affected.