Customers of incident.io began reporting that their per-organization incident identifiers, such as #INC-N, were unexpectedly jumping. For example, one customer observed an ID jump from #INC-7 directly to #INC-39, indicating a consistent skip of 32 numbers. This suggested a systematic issue with how incident IDs were being generated.

Initially, the team considered transaction rollbacks as a potential cause, as PostgreSQL sequences increment even if a transaction is rolled back. However, this was deemed unlikely because it would require 32 failed incident creations for a single organization, and no such issues or customer reports of failed incident creation were observed.

The root cause was identified in PostgreSQL’s sequence behavior. PostgreSQL pre-allocates 32 values (SEQ_LOG_VALS) to optimize Write-Ahead Log (WAL) writes, avoiding logging every nextval call. During a database upgrade, a follower node was promoted to become the new primary. When this promotion occurred, the new primary assumed all pre-allocated values on the previous primary had been used, causing the sequence to effectively “skip” up to 32 values forward.

While no data was lost or misattributed, the unexpected jumps in incident IDs were confusing for customers. To resolve this, incident.io modified their ID generation logic. Instead of relying on nextval from a PostgreSQL sequence, they implemented a trigger that explicitly calculates the next external_id by querying SELECT COALESCE(MAX(external_id), 0) + 1 for the specific organization, ensuring strictly sequential and gap-free IDs.