Customer impact

Between October 1, 2026 at 03:00 UTC and October 1, 2026 at 04:10 UTC, a subset of customers experienced intermittent failures when using AI-powered features, including Context Grounding, Agent Builder and Agents, Autopilot for Developers, generative AI activities, and Communications Mining.

Impact affected requests served by or routed through our Europe AI services infrastructure. Affected regions include UAE, Korea, Switzerland, UK, India, Singapore, Australia and Europe. Customers saw failed requests, including service-unavailable errors. Failures were concentrated during two periods 03:00 UTC to 03:45 UTC and 04:00 UTC to 04:08 UTC for a total incident window of approximately 1 hour and 10 minutes.

Root cause

A single image-processing workload sent a large batch of requests with very large image payloads to our Europe AI services infrastructure. Those oversized request bodies caused service instances to exhaust available memory, become unresponsive, and restart. As healthy capacity dropped, our systems returned service-unavailable errors for customer traffic. When traffic was automatically rerouted to a second Europe location, the same failure pattern occurred because both locations received the same oversized workload.

Detection

Our monitoring systems generated multiple concurrent alerts shortly after failures began at approximately 03:00 UTC. During the initial investigation, our engineering team confirmed that Europe AI service failures were causing customer-facing errors across multiple products and regions.

Response

During the investigation, our engineering team identified the source workload and defined mitigation options: stopping the workload at its source, or blocking its traffic before it reached the affected services.

As a short-term stabilisation measure, service memory limits were increased. Telemetry confirmed two failure periods 03:00 UTC to 03:45 UTC and 04:00 UTC to 04:08 UTC. By 04:10 UTC, service-unavailable errors had dropped to near zero and no additional service restarts were observed.

A traffic block was then applied to prevent the source workload from reaching the affected services. The source workload was confirmed to have stopped at approximately 04:46 UTC, and no further traffic from it was observed afterward.

Follow up

1. We are adding enforcement at the service level to reject or safely handle oversized image payloads before they can exhaust service memory.
2. We are adding a per-organization concurrency limit for Context Grounding image processing to prevent a single workload from affecting service availability for other customers.