Data Service Workflows execution and creation affected in Delayed US region

Customer impact

Between May 15, 2025, at 14:18 UTC and May 15, 2025, at 15:08 UTC, Data Service in the Delayed Enterprise U.S. region experienced downtime. Most of the Data Service queries failed because the service was not available.

Root cause

A previous deployment introduced a database column exclusively in the Delayed Enterprise U.S. region. Later, we removed this unused column. However, we didn’t know the application code still had references to the column. This caused the incident.

Detection

The problem was detected immediately through our alerting systems. Customers also reported the issue.

Response

The engineering team was promptly engaged. They removed the references to the deleted column. This fixed the Data Service.

Follow-up

To prevent recurrence of this issue:

• We will add a validation step to our deployments to ensure backward and forward compatibility between the application code and the database schema.
• We will add automated tests to find schema-related issues before deployment.
• We will fine-tune our alerting for faster detection of similar problems.