Customer Impact

On March 13, 2026, between approximately 13:01 and 13:55 UTC, customers using Studio Web in the Enterprise Australia environment experienced a service outage. Users encountered HTTP 503 errors and were unable to access Studio Web.

The incident occurred at midnight local time in Australia (AEDT) or 1 PM UTC, which significantly limited customer exposure.

Root Cause

A routine change included a security hardening change that updated the internal port used by the service. During the staged deployment process, the updated port configuration was applied to the service routing layer before the running service instances were replaced with the new version. This created a mismatch: the routing layer directed traffic to the new port, but the existing service instances were still listening on the old port, making them unreachable.

This was an edge case in our staged deployment tooling, which does not account for port changes between versions. The port change was validated in pre-production environments, but the specific interaction with the staged deployment process was not anticipated.

Detection

The issue was detected automatically by our monitoring systems soon after the deployment entered its validation phase.

Response

The team quickly identified the configuration mismatch and applied a mitigation by synchronizing the routing configuration with the running service instances. All services completed their updates by 13:32 UTC, restoring full functionality. The service was confirmed fully restored at approximately 13:55 UTC after a stability monitoring period.

Follow-Up

1. Implement pre-deployment validations for detecting such configuration changes.
2. Investigate and improve the automatic failover mechanism for the Australia environment, which did not trigger as expected during this incident.
3. Add service endpoint validation to verify that traffic actually reaches service instances through the routing layer, not just that instances are healthy.
4. Develop a deployment strategy for configuration changes that affect service routing, ensuring they are applied atomically with the service update.