Customer Impact

Starting from 2025-12-22 15:00 UTC until 2025-12-22 19:00 UTC, some enterprise customers in U.S. region using the Serverless Robots service may have experienced workloads remaining in a pending state for extended periods.

The issue initially affected a small number of customers. As pending workloads accumulated due to limited processing capacity, the impact expanded to additional customers in the region.

After mitigation actions were completed, all pending workloads were processed and service performance returned to normal with no further degradation observed.

Root Cause

The incident was caused by an outage in our cloud provider’s infrastructure:

• We detected failures when allocating new virtual machines in the U.S. region’s Availability Set, as reported by our cloud provider, which triggered critical severity alerts.
• As a result, the workload processing infrastructure was unable to scale up to meet increased demand, leading to delayed job processing and degraded service performance.
• We also identified gaps in our automatic unavailability detection and failover mechanisms, which, if functioning as intended, would have reduced customer impact.

Detection

The issue was initially detected by automated critical severity alerts indicating failures during infrastructure upscaling. Shortly thereafter, additional alerts were triggered related to CPU and memory saturation, as well as delays in pod scheduling within the affected region.

Response

Upon receiving the critical alerts:

• We confirmed that the impact was limited to the U.S. region and opened an incident investigation to track and manage the response.
• The engineering team identified an outage in the cloud provider’s infrastructure and engaged provider support through dedicated escalation channels.
• Manual failover on secondary cloud provider was initiated to temporarily mitigate the issue, which reduced the load by processing the pending customer jobs in batches, until the workload returned to normal.

Once failover was fully active, all pending workloads were processed and normal service performance was restored.

Follow-Up

To prevent similar incidents in the future, we are taking the following steps:

• Enhancing automatic detection and failover for cloud service unavailability to support near-continuous workload processing and faster recovery.
• Improving the clarity and prioritization of cluster management alerts to enable faster and more effective engineering response.