Insights service is not accessible for some customers in EUS & NE regions

Customer impact

Between May 05, 2025 21:13 UTC and May 06, 2025 00:07 UTC customers in Delayed Enterprise US and Europe regions were not able to access Insights. All other regions were unaffected.

Root cause

This was a deployment issue where some files failed to copy to their target locations. The problem was identified and fixed on the latest release, but it was not backported to the prior release that was targeting the Delayed Enterprise regions. The issue occurred upon deployment to those delayed regions.

Detection

The issue was discovered by automated alert at 21:33 UTC.

Response

Once identified, we implemented the fix on the previous release and redeployed.

Our teams also identified that notifications of the incident were delayed due to human error. Our initial posts to the status page were published without notifications enabled.

Follow up

To prevent similar issues in the future, we are taking the following steps:

• Improving documentation: Our team will update our documentation related to hot-fix deployments.
• Enhancing checks: Our team will add checks to ensure that fixes are implemented and rolled out to each environment.
• Enhancing notification processes: To address our notification delay, our team will be updating both our manual and automated processes so that notifications are not unintentionally disabled.