Customer impact

Between July 30, 2026 at 4:32 am UTC and 8:12 am UTC (approximately 3 hours and 40 minutes), a subset of customers in the Asia-Pacific region - Japan, Australia, and Singapore - experienced degraded Insights behavior for recently created queue items. These items could not be processed by the Insights data pipeline because their folder association was missing. No data was lost, and all affected items were recovered.

Root cause

The root cause was a planned database maintenance intended to clean up folder references that pointed to deleted folders or classic folders. This work was meant to proactively fix recurring tenant-migration problems and reduce the remaining references to classic folders, a prerequisite for removing them.

Unintentionally, the change left some active queue definitions without a folder association, and newly created queue items copied that missing association. As a result, the Insights data pipeline could not process those queue items until the associations were restored.

Detection

The issue was detected by automated alerting on July 30, 2026 at 4:32 am UTC, when an alert reported an extended pipeline failure in the Insights backend. Detection occurred at the same time the incident was opened. The public status page was updated at 4:42 am UTC.

Response

By 5:14 am UTC, analysis confirmed that the issue affected a subset of recently created queue items, that no data had been lost, and that all affected items could be recovered. At 6:17 am UTC, a recovery script was prepared to restore valid folder associations on queue definitions. At 6:22 am UTC, validation of the fix began before applying it to affected queue items.

Recovery was applied region by region. Japan was fixed by 7:16 am UTC and confirmed recovered at 7:42 am UTC. Australia was completed at 7:49 am UTC, and all three impacted regions were reported resolved at 7:55 am UTC. By 8:12 am UTC, the fix had been applied to all impacted regions and affected items had been recovered.

As a precaution, the same recovery was then applied to remaining regions where the cleanup had run. This precautionary work was completed by 11:00 am UTC, monitoring began at 11:12 am UTC, and the incident was resolved at 12:30 pm UTC after monitoring confirmed normal operation.

Follow up

Our goal is to ensure that routine database maintenance can no longer leave an active queue definition without a valid main folder association, and that any similar issue is detected and recovered before it affects customers. To achieve this, the following preventive actions are being tackled:

1. Add pre-execution validation and staged, region-by-region rollout to folder-cleanup maintenance, so active queue definitions always retain a valid main folder association.
2. Maintain a validated recovery procedure to quickly restore folder associations if a similar issue occurs.