Postmortem Index

Explore incident reports from various companies

Basecamp

In November 2018 a database hit the integer limit, leaving the service in read-only mode while the team migrated the affected id column from a 4-byte to an 8-byte integer.

A subsequent follow-up post explains that the affected table held the events log used by every Basecamp 3 page render, so once writes were paused the rest of the app degraded into a read-only mode for users. Restoring write capacity required altering the column on the live primary, which took several hours of downtime because Postgres has to rewrite the table to widen an integer column.