Postmortem Index

Explore incident reports from various companies

GitHub Actions and Pages impacted by scoped token INT32 overflow

GitHub · GitHub Actions, GitHub Pages, GitHub API, Git commands

On May 16, 2021, GitHub experienced an incident lasting 9 hours and 48 minutes, which significantly impacted GitHub Actions, GitHub Pages, and operations against the GitHub API and low-level Git commands that utilized scoped tokens. Users encountered high failure rates across these services.

The root cause was identified as a foreign key for scoped tokens exceeding the maximum value for an INT32 data type. This integer overflow led to widespread service degradation for features relying on these tokens.

Mitigation involved a long-running schema migration to change the affected foreign key to an INT64 data type, allowing for a larger range of values. Following the migration, invalid token records stored in the cache layer were systematically removed to restore functionality.

Existing alerting and linting mechanisms designed to prevent integer overflows were insufficient in this specific case because the foreign key predated the implementation of these preventative measures. GitHub is now manually auditing all INT32 columns and enhancing automation to prevent similar issues.

During the incident, a single GitHub Action on one repository briefly received unauthorized access grants. These grants were promptly revoked, and GitHub confirmed that no unauthorized access was gained through this vulnerability.

Separately, on May 8, 2021, a 46-minute incident affected the GitHub Container registry service due to failures in an underlying MySQL database. This was resolved by performing a failover to a database replica, and subsequent work is planned to improve resilience against such outages.

Keywords

githubactionspagesapigitscoped tokensint32database