Customer Impact

On April 23, 2026, from 3:00 pm UTC to 11:42 pm UTC, some customers with IP restriction policies enabled experienced agent jobs remaining pending or failing to start. More than 150 jobs were affected across multiple regions.

Customers without IP restrictions and agents using the serverless execution path were not affected.

Root Cause

A messaging infrastructure update introduced a regression that prevented required IP restriction headers from being forwarded between internal services during agent job initiation.

Without these headers, downstream authorization checks rejected requests from organizations using IP restrictions. Serverless agents were unaffected because they use a separate execution path.

Detection and Response

The incident was detected through customer reports and monitoring of pending jobs. It was declared at 3:24 pm UTC, and the public status page was updated at 3:26 pm UTC.

The investigation established that:

• Only organizations with IP restrictions enabled were affected.
• Serverless agents were unaffected.
• Temporarily disabling IP restrictions allowed an affected job to complete, confirming the connection to IP restriction handling.

At 5:26 pm UTC, the team identified the header propagation regression. A fix was validated and progressively deployed across production regions. Customers began confirming recovery by 11:00 pm UTC, and all regions were operational by 11:42 pm UTC.

Preventive Actions

We are taking the following actions:

1. Add deployment checks that verify critical security headers are propagated across internal service calls.
2. Expand integration tests to cover allowed and denied requests for organizations using IP restrictions.
3. Introduce phased rollouts and environment-verification checks for messaging infrastructure changes.
4. Improve monitoring and escalation for pending agent jobs, including failures associated with security configurations.

We apologize for the disruption and are strengthening our testing, deployment, and monitoring processes to reduce the risk of recurrence.