Customer Impact

On November 25, 2025, between 9:14 am UTC and 9:34 am UTC, a subset of customers in the Europe region experienced degraded service while accessing apps platform. The issue stemmed from incorrect CDN URLs generated during deployment, which made some artifacts essential to load a web application inaccessible and impacted customers. Other regions and services remained unaffected during this event.

Root Cause

The incident was caused by an incorrect configuration introduced in the deployment pipeline. This configuration resulted in the system generating invalid CDN URLs, leading to missing assets and degraded application performance for affected customers.

Detection

The issue was detected through internal monitoring systems, which flagged degraded performance shortly after deployment. The investigation promptly confirmed an invalid configuration as the root cause.

Mitigation

To resolve the issue, the team immediately rolled back to a previously stable build of the application, removing the invalid configuration responsible for the CDN URL errors. By reverting to the older build, broken assets were restored, and application functionality returned to normal. The rollback completed successfully, restoring full service to affected customers by 9:34 am UTC.

Follow-Up

To prevent similar incidents in the future, we are implementing the following measures:

1. Automated Validation: Enhance deployment pipelines by introducing automated validation checks for configurations to detect invalid setups before they reach production.
2. Faster Rollback Framework: Implement a streamlined rollback process to reduce response times and ensure all systems can return to a stable state more quickly during emergencies.