Customer impact

Between 2025-11-13 11:25 UTC and 2025-11-13 20:38 UTC, some customers experienced increased latency and intermittent failures when accessing Automation Cloud services hosted in the EU region.

Initially, only internal test organizations were affected, but as the issue progressed, a subset of community and trial customers encountered failures when:

• Signing up for new accounts
• Requesting or activating trial licenses

Once the issue was mitigated, service performance returned to normal and no further degradation was observed.

Root cause

The incident was caused by a database performance degradation in the Licensing Resource Manager (LRM) service.

A specific test organization used by internal automation tests had accumulated large numbers of stale entries in a DB table that contains license records due to:

• Repeated tenant creation/deletion tests that soft-deleted service licenses instead of hard-deleting them
• Absence of proper optimizing mechanism on that specific license records DB table

As a result, queries targeting this organization became extremely slow, causing high CPU utilization (100%) and blocked queries across the database.

When retries accumulate, they further exacerbate blocking, impacting other services including Automation Cloud and Orchestrator.

Detection

The issue was first detected by our engineering teams via automated critical severity alerts indicating sustained 100% CPU usage on the database.

Additional alerts were triggered shortly afterward by the Licensing service, showing latency and request failures in the affected region.

Response

Upon detection:

1. Database scaling was initiated to temporarily mitigate the load.
2. The engineering team identified that a test organization used for automated testing was causing abnormally resource-intensive queries.
3. The automated tests responsible for tenant creation/deletion were disabled via feature flag, reducing ongoing load.
4. A new query-optimizing mechanism was introduced on the responsible table
5. When performance did not fully recover, teams performed a cleanup of the large test organization’s service licenses, significantly reducing table size and unlocking blocked queries.

After cleanup completion, resource utilization and latency dropped immediately. Services were fully restored..

Follow-up

To prevent similar incidents in the future, we are taking the following steps:

• Optimizing the specific query that contributed to the database connection pool saturation, ensuring it performs efficiently under high traffic conditions.
• Tuning database connection pool configurations to improve resilience and reduce the likelihood of contention during traffic spikes.
• Enhancing monitoring and alerting mechanisms to proactively detect unusual query patterns and potential resource exhaustion scenarios before they impact customers.