Insights cloud is experiencing failed requests in Japan and Australia regions
Incident Report for UiPath
Postmortem

Root Cause Analysis – UiPath Insights Outage on June 30 2022

Customer impact

On June 30, 2022 from 00:03 UTC until 01:02 UTC UiPath Insights was non-functional for all customers in Australia and Japan. Customers in those regions who attempted to use Insights during this time would receive an error.

Root Cause

UiPath Insights utilizes Looker for visualizations. On June 25 Looker sent us an email instructing us to update our DNS records which are used to make calls to them. Unfortunately, these emails were sent to an address that was no longer being monitored. As a result, these changes were missed.

Detection & Response

Our automated alerting system immediately detected the issue and notified the oncall engineers. It was immediately clear that communications with Looker were failing, but it was not immediately obvious why. The oncall engineers simultaneously filed a support ticket with Looker and escalated to the UiPath team responsible for our DNS entries. While investigating the issue, they realized that they could update the config used by our services to directly refer to Looker’s URL, rather than the DNS CNAME that we normally use. This allowed us to bypass the issue and deploy a mitigation while we were still getting to the bottom of the root cause.

Follow Up

After the incident was mitigated, we continued to work with Looker to determine the root cause. Once the root cause was found, we made the DNS changes they recommended as well as updated the email addresses they contact to include two different team aliases to ensure that we don’t miss similar communications in the future.

Posted Jul 05, 2022 - 22:11 UTC

Resolved
This incident has been resolved.
Posted Jun 30, 2022 - 01:27 UTC
Update
We are continuing to monitor for any further issues.
Posted Jun 30, 2022 - 01:14 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jun 30, 2022 - 01:13 UTC
Investigating
We are currently investigating this issue.
Posted Jun 30, 2022 - 00:53 UTC
This incident affected: Insights.