On December 4, 2023, Turso identified and fixed an incident affecting 0.07% of its free tier databases. The issue, which began impacting databases on December 1st, involved a data leak where data from one database could be accessed by another, and a subsequent fix led to the potential loss of recent data for affected users. The underlying problem was introduced by a system change on November 20th.

The incident stemmed from a bug during database migration processes that caused some free tier databases to be configured with an empty backup identifier. This meant that instead of pointing to unique S3 backup paths, these affected databases shared a common, null ID backup location. When these databases, which scale to zero after inactivity, needed to be restored due to host unavailability on fly.io, they pulled data from this shared location.

This misconfiguration led to two primary impacts: a data leak, where data from one database became accessible to others sharing the null ID backup, and potential data loss. To resolve the leak, Turso applied a conservative fix that involved recreating affected databases from their December 1st backups, resulting in the discarding of any data written after that date for the impacted users.

The immediate fix involved re-running the migration with correct parameters and restoring affected databases from their December 1st backups. For future prevention, Turso is implementing additional internal checks for backup correctness and data isolation, improving configuration validation, enhancing deployment methods to prevent backup ID issues during migrations, and establishing better security incident notification mechanisms. The company has prioritized these improvements to prevent similar incidents.