Customer impact

Between July 2, 2026 at around 20:57 UTC and July 3, 2026 around 01:01 am UTC, some Maestro workflows in Australia and U.S. regions using instance-level operations like Retry, Pause, Resume, Cancel, Migrate, and patch variables were potentially impacted, with workflows appearing to be stuck state.

Root cause

Maestro workflows keep a detailed record of every step they execute, and rely on this record to resume a workflow from where it left off. A recent change introduced inconsistencies into this record for some workflows. As a result, those workflows could not resume and appeared to be stuck.

Detection

The incident was caught on an internal report. While we had alerts in place, the rate of errors was not enough to trigger the alerts.

Response

A cross-team bridge was set up on July 2, 8.57 pm UTC, and engineering started looking into the errors being thrown from workflows from Telemetry. Soon enough, on an analysis of telemetry and commits that were included as a part of the latest release payload, the root cause was identified.

Once that was done, a hotfix was developed and validated to confirm that it addressed the issue. The issue was mitigated once the hotfix was deployed to all affected rings and affected customer workflows were retried to ensure they were running.

Follow-up

We are implementing enhanced monitoring and alerting to detect such issues before they affects customers and implementing guardrails to ensure that these issues do not make it into production code.