Customer Impact

Some customers in the Japan region could not change their IXP data and projects at the design stage. Affected operations included taxonomy updates, changes to model configuration, and project renaming/deletion. Three projects were affected in total. There was no run-time impact for any customer.

Root Cause

The IXP team is moving customer data from one database to another. This process has been going on for a few months. In January, all documents were moved to the new database. Two small bugs in the code which performed dataset operations in the new database resulted in the possibility of datasets getting stuck in a “locked” state in which updates could not be made. These bugs were not found during testing because the steps to get to this locked state are rarely used. For example, the same changes were made in all production environments, and this code path was only hit in Japan.

Detection

The incident was first detected through automated monitoring, which triggered alerts about service degradation in the Japan region. These alerts were quickly escalated and status pages changed. This meant that there was a possible breach of the service level agreement. The engineering team immediately began investigating the alerts and confirmed the impact through internal testing and customer reports.
Once the scope and severity were understood, the incident was classified as a high-priority event, and a coordinated response was initiated.

Response

When the problem was found, the engineers worked quickly to fix it as soon as possible. The quick fix was to find the likely cause and make a fix to fix the problem right away. The hotfix was first put into a test environment, then to Japan. It worked well to fix design-time services.

During the response, the team provided ongoing status updates and closely monitored the recovery process. When the initial fix did not resolve the issue in all environments, the team quickly iterated on the solution, ultimately restoring full functionality. Recovery was verified through both automated monitoring and manual testing to ensure that affected features were working as expected before closing the incident.

Follow-Up

To prevent similar incidents in the future, we are implementing several improvements. First, we are enhancing our automated test coverage for design-time features to include more advanced workflows. This will help us spot problems earlier in the process. Additionally, we are updating our deployment process to include more rigorous validation steps in test environments that closely mirror production.

We are also reviewing our change management and incident response processes to ensure faster detection and resolution of issues that may affect critical customer workflows. These improvements are being prioritized for completion in the next few weeks. We promise to keep customers updated on our progress.

We're sorry for the trouble this incident caused. We appreciate your patience as we work to make our platform stronger and more reliable. Your feedback is invaluable in helping us improve our services.