Customer Impact

On September 10, 2026, between 10:00 UTC and 17:40 UTC, customers using Automation Cloud in Europe region experienced intermittent failures of unattended robot job executions on the serverless runtime.

Affected jobs failed shortly after starting, before any automation work was performed, with errors indicating that a required program file could not be found. Only automations using a specific set of software libraries were affected; the majority of jobs in the region continued to run normally. The job failure rate rose from a normal baseline of approximately 6% to a peak of 30-37%, and approximately 312 job executions failed that would otherwise have succeeded. Affected automations could be re-run successfully once the issue was resolved. No data was lost, and no job produced incorrect results.

Root Cause

To make automations start quickly we keep a pre-built cache of commonly used software packages, so robots do not download them at run time. Two changes combined to cause this incident.

A routine update to an internal build tool changed which Application runtime version the cache was built for. Packages ship files for several runtime versions and the cache keeps only those needed, to stay a manageable size; after the update it kept only the newest runtime's files and deleted the ones our robots use.

Separately, we store previous dependency calculations so repeat jobs start faster. Those stored results pointed at the deleted files, and the check validating them confirmed only that the package folder existed, not its contents. The folders were intact and the correct size, so the check passed, the stale result was reused, and the step that would have caught the missing files was skipped.

Affected jobs were handed a path that no longer existed and failed on startup. Without the second factor the robot would simply have downloaded the package at run time - slower, but not a failure. That is what happened in every other region, where the same cache was in use but this optimisation was not enabled; it was on only in Europe region, which is why impact was confined there.

Detection

The issue was not detected by automated monitoring. It was identified after a customer-reported support case. Three gaps contributed, all addressed below: there was no alert on elevated job failure rate per region, so a sustained rise from 6% to over 30% went unnotified for more than seven hours; the cache build had no validation that the files required by the robot runtime survived the size-reduction step, so it completed successfully and deployed automatically; and existing cache integrity checks verify that packages are present and record their sizes, which all passed, because every package was present at full size and only files inside them were missing.

Response

All times UTC on 10 September 2026.

• 09:37 and 09:49 - the newly built cache became active in the region.
• 10:00 - customer job failures begin.Following the customer report, engineering identified that the failures were caused by files missing from the package cache, and traced the change that removed them.
• 17:12 - mitigation begins; the cache is rolled back to the last version built before the change.
• 17:23 - the rolled-back cache becomes active.
• 17:40 - failure rate returns to normal. Customer impact ends.
• 17:46 - the stored-result optimisation is disabled as a secondary safeguard, so that even a faulty cache would slow job starts rather than fail them.
• 18:08 - the rollback completes across the region. Later that evening - the automated cache build pipeline was disabled, preventing a new cache from reintroducing the problem before a permanent fix is in place.

Recovery was verified by confirming that jobs starting after the rollback took effect returned to the region's normal failure rate, rather than inferring recovery from the overall rate subsiding.

Follow-up

• Short-term, delivered. The cache now keeps the files for every runtime version in use, instead of keeping only the newest and discarding the rest. This removes the condition that caused the incident, because a cache rebuild can no longer delete files that robots depend on. The capability was added on 11 September and switched on for both runtime versions on 14 September.
• Short-term, to do. Add automated end-to-end test coverage to the deployment pipeline that validates package cache integrity before rollout, blocking releases when tests fail.
• Short-term, to do. Add automated monitoring and alert to cover NuGet package cache missed that fail to be resolved either from cache or from the source.
• Longer-term, in progress. Stored dependency results are validated against their cache and discarded if referenced files are missing. Stale results won't be reused against a mismatched cache — if files are missing, jobs fall back to downloading at runtime instead of failing. This optimization will be rolled out gradually to verify and deliver the work.
• A related recurrence of the same underlying cause affected solution publishing on 11 September and was mitigated the same day; it is tracked separately. The fixes above address the common cause and are intended to prevent recurrence of both in any deployment region.