Customer Impact

On July 8, for about five and a half hours (roughly 12:00 to 17:40 UTC), a subset of Agents customers in the US and EU intermittently received HTTP 500 errors on LLM completion calls. Only calls routed to one affected instance failed, so retries often succeeded. No data was lost and no other functionality was affected.

Root Cause

The failures were caused by SNAT port exhaustion on a single node in one of our shared production clusters. That node reached its fixed limit of outbound ports, which are shared by all pods running on it, so it could no longer open new outbound connections. As a result, the agents calls to the LLM Gateway failed at the network layer and returned 500 to customers.

The exhaustion was driven by the combined outbound connection volume of several services running on that node. We have identified the code that causes this problem, and are implementing a fix.

Response

Detection was delayed because of an independent under-reporting issne in telemetry, so our alerts stayed below paging severity. Once identified, the affected node was drained, which moved the pods onto healthy nodes and stopped the errors immediately. Draining a shared production node is a last-resort action, since it also evicts other teams' workloads from that node, which is why it was applied late rather than early.

Follow-Up

1. Harden the code against SNAT leaks.
2. Create a new alert that would help us identify such issues quickly on the service sde.