The Orchestrator service experienced performance issues on 09/07/2023, between 5:04 AM and 6:08 PM UTC in United States, affecting one of the two clusters in the region. The issue manifested as a gradual increase in latency and timeouts for a number of key APIs, leading to general slowness of the web interface.

The cause was identified to be a robot trying to send a large payload to Orchestrator, which was not trimmed on either the robot or Orchestrator. This large payload hit a part of our code that was not optimized to handle such large data, triggering a behavior that consumed a lot of resources, affecting the latency of all requests processed on that particular pod. Since the request was not successful, the robot retried submitting the payload twice a minute, affecting different pods every time.

Upon detecting the issue, we first moved to block the robot sending the large payload. We then deployed a hotfix that implements a mechanism for trimming payloads above a reasonable size.

Following up, we are investing into better detection mechanisms. Our alerting systems failed to detect the issue in time mainly because only the latency of a small subset of pods was affected at a time. Aggregated metrics were only slightly affected and failure rate remained fairly constant. We are adding alerting that will be capable to detect degradation for single pods better. We are also auditing Orchestrator for similar cases of unrestricted payload sizes and either limiting them or investing into optimized handling.