Added latency in working with Data Service in US Region

Incident Report for UiPath

Postmortem

Customer impact

Users experienced added latency for DataService in US region between 2025-04-08 0723 UTC and 2025-05-07 1930 UTC. All other regions were unaffected.

Root Cause

This was an infrastructure issue where user requests were routed to the farthest compute rather than the closest compute. The issue occurred upon infrastructure upgrades specific to DataService in US region.

Detection

The issue was discovered by our routine review of service performance on 2025-05-07 1128 UTC.

Response

Once identified, we manually overrode the routing configuration to route traffic to the closest compute in US region. Gradual recovery of system performance was observed after this change.

Follow-up

To prevent similar issues in the future, we are taking the following steps:

  • Improving Detection Mechanism: Fine-tune our detection mechanism to be more sensitive to latency changes.
  • Improving checks: Additional validation checks for safe infrastructure updates.
Posted May 23, 2025 - 15:07 UTC

Resolved

This incident has been resolved.
Posted May 08, 2025 - 04:05 UTC

Update

We are continuing to monitor for any further issues.
Posted May 08, 2025 - 03:38 UTC

Update

We are continuing to monitor the effects of changes applied.
Posted May 07, 2025 - 19:23 UTC

Update

We are continuing to monitor the effects of changes applied.
Posted May 07, 2025 - 17:16 UTC

Update

We are continuing to monitor the effects of changes applied.
Posted May 07, 2025 - 16:06 UTC

Update

We are continuing to monitor the effects of changes applied.
Posted May 07, 2025 - 14:01 UTC

Monitoring

We've updated the routing to collocate compute and storage within the same region in the US. We will continue monitoring traffic during US timezone hours to observe improvements.
Posted May 07, 2025 - 12:43 UTC

Investigating

Users may be experiencing added latency Dataservice. Our Dataservice engineering team is currently investigating issues related to internal network latency between compute and storage.
Posted May 07, 2025 - 11:25 UTC
This incident affected: Data Service.