Between December 13, 2024, at 16:00 UTC and December 14, 2024, at 08:30 UTC, some customers with tenants hosted in the U.S. and E.U. region may have experienced errors or increased latency while using the generative AI capabilities in Document Understanding. Customers in the delayed rings were not affected.
Document Understanding leverages Azure OpenAI GPT to power features that require large language models (LLMs). UiPath partners with Azure to secure a specific capacity for these advanced AI services. However, this Azure capacity is limited. Sometimes obtaining additional resources takes time.
Shortly before the start of the incident, we saw a large increase in usage of the Gen AI features of Document Understanding. This caused us to run out of Azure OpenAI capacity. Azure did not have additional quota in these regions, so usage was heavily throttled. This has impacted all customers in these regions.
Our monitors found the problem and told our engineers in a few minutes.
Once the spike in requests subsided, our systems returned to their normal state.
To prevent similar issues in the future and enhance our service reliability, we are implementing several key improvements: