On June 30, 2022 from 00:03 UTC until 01:02 UTC UiPath Insights was non-functional for all customers in Australia and Japan. Customers in those regions who attempted to use Insights during this time would receive an error.
UiPath Insights utilizes Looker for visualizations. On June 25 Looker sent us an email instructing us to update our DNS records which are used to make calls to them. Unfortunately, these emails were sent to an address that was no longer being monitored. As a result, these changes were missed.
Our automated alerting system immediately detected the issue and notified the oncall engineers. It was immediately clear that communications with Looker were failing, but it was not immediately obvious why. The oncall engineers simultaneously filed a support ticket with Looker and escalated to the UiPath team responsible for our DNS entries. While investigating the issue, they realized that they could update the config used by our services to directly refer to Looker’s URL, rather than the DNS CNAME that we normally use. This allowed us to bypass the issue and deploy a mitigation while we were still getting to the bottom of the root cause.
After the incident was mitigated, we continued to work with Looker to determine the root cause. Once the root cause was found, we made the DNS changes they recommended as well as updated the email addresses they contact to include two different team aliases to ensure that we don’t miss similar communications in the future.