Customer impact

Between 2026-01-13 16:47 UTC and 2026-01-13 18:24 UTC (approximately 1 hour and 37 minutes), customers in the Canada region experienced an outage of the Solutions Management UI following an Orchestrator deployment.

During this period, the Solutions Management interface failed to load, preventing affected customers from accessing or managing Solutions through the UI. All Solutions backend APIs, Orchestrator services, and running automations remained fully operational, and no data loss occurred. The impact was limited strictly to the user interface.

Root Cause

The incident occurred due to a missing frontend resource required by the Solutions Management UI after an Orchestrator version update.

As part of the Orchestrator deployment, the system attempted to load a newer version of the Solutions frontend from the content delivery network (CDN). However, this specific frontend resource had not been uploaded to the CDN beforehand. As a result, when the UI attempted to load, it failed due to the missing file.

This condition went undetected earlier because a temporary deployment configuration had been in place that masked the issue until the Orchestrator version upgrade triggered the frontend version switch.

Detection

The issue was detected by internal synthetic monitoring, which alerted the on-call engineering team shortly after the deployment when the Solutions Management UI became unavailable.

While monitoring successfully detected the UI outage, there was no automated pre-deployment validation in place to verify that the required frontend resource was available on the CDN before the version switch occurred.

Response

• 2026-01-13 16:47 UTC – Customer impact began following the Orchestrator deployment.
• Shortly after detection – The on-call engineering team investigated the issue and identified that the Solutions frontend was failing to load due to a missing CDN resource.
• Mitigation initiated – The team rolled back the Solutions frontend to a previously compatible version, restoring access to the UI.
• 2026-01-13 18:24 UTC – The Solutions Management UI was fully restored for all affected customers.

The following day, the correct frontend resource was uploaded to the CDN, and normal deployment processes were resumed after validation.

Follow-Up

Short-term fixes

• Ensure the correct Solutions frontend artifact is uploaded to the CDN before any Orchestrator version switch.
• Add automated validation steps to confirm frontend resource availability prior to deployment completion.

Long-term improvements

• Improve coordination and validation between Orchestrator and Solutions deployments to prevent version mismatches.
• Expand automated checks and canary coverage to detect frontend availability issues earlier, even in edge cases involving deployment configuration exceptions.
• Improve deployment visibility and alerting to ensure missing resources are detected before customer impact occurs.

These improvements will strengthen deployment validation and reduce the likelihood of similar issues impacting customers in future deployments.