Customer impact

Between May 1, 2026 at 9:45 pm UTC and May 2, 2026 at 11:20 am UTC, customers using a specific Automation Hub GxP environment were unable to sign in. All sign in attempts failed with HTTP 500 errors, blocking access to all Automation Hub features requiring authentication.

Scope: Only one specific Automation Hub GxP environment was affected. All other Automation Hub environments remained fully operational throughout.

Root cause

The incident was caused by the expiration of a credential used by Automation Hub's sign in service to authenticate with the identity provider. The credential had been rotated the previous week and the updated value was correctly stored in the secure configuration vault. However, the application services in the affected environment were not restarted after the rotation. These services load credentials into memory at startup and do not refresh them automatically, so they continued presenting the expired credential.

The outage was delayed by in-memory token caching—the cached token remained valid for a period after the credential expired. The outage began when the cache attempted to refresh and the identity provider rejected the expired credential. No documented procedure existed requiring a service restart after credential rotations, which directly contributed to the oversight.

Detection

Automated monitoring detected the issue at 9:45 pm UTC on May 1, 2026, when sign in failure alerts triggered for the affected environment. Investigation confirmed that all sign in attempts were returning HTTP 500 errors isolated to the authentication service. The engineering team began coordinating incident response at 10:08 pm UTC.

Response

At 10:28 pm UTC, the engineering team convened to investigate. They confirmed that the updated credential was already in the vault and valid, but the running services had not been restarted to load it.

The team redeployed all Automation Hub services in the affected environment using the current production version, ensuring no unreleased features were introduced. Redeployment began at approximately 11:14 am UTC on May 2, 2026. By 11:20 am UTC, sign in was restored and monitoring confirmed a 0% error rate. The incident was formally resolved at 11:25 am UTC.

Follow-up

To prevent recurrence, we are implementing the following improvements:

1. Automated post-rotation restarts: Adding automated validation to ensure application services are restarted whenever credentials are rotated, eliminating the manual step that was missed.
2. Proactive credential mismatch detection: Deploying monitoring to detect discrepancies between credentials in the vault and those in use by running services, enabling early identification of credential drift.
3. Proactive expiration alerting: Adding alerts for credentials approaching expiration that have not been loaded by running services.
4. Updated operational procedures and documentation: Updating maintenance procedures to require service restarts after credential changes and enhancing credential management documentation for validated environments.

These changes are designed to strengthen our authentication infrastructure, reduce reliance on manual steps, and ensure credential rotations are handled seamlessly.