On Tuesday, March 11th, GitHub experienced a distributed denial of service (DDoS) attack, rendering the service largely unreachable for approximately two hours. The incident began at 21:25 UTC when connectivity problems were reported, and GitHub opened an incident on its status site at 21:29 UTC.

The initial attack vector involved several thousand HTTP requests per second from thousands of IP addresses targeting a crafted URL. These requests were sent to the non-SSL HTTP port and then redirected to HTTPS, overwhelming GitHub’s load balancers and application tier. Mitigation efforts to block these requests were deployed by 22:35 UTC, at which point the site appeared to stabilize.

However, a second attack vector emerged, focusing on exhausting SSL processing capacity through a high number of SSL connections on the load balancers. GitHub responded using its mitigation platform, but the countermeasures required significant tuning to minimize false positives, leading to an additional 25 minutes of downtime between 23:05 UTC and 23:30 UTC. By 23:34 UTC, the site was fully operational, though the attack continued without further customer impact.

The incident highlighted two key areas for improvement. Firstly, GitHub’s monitoring was primarily focused on bandwidth, failing to quickly detect attacks characterized by high packets-per-second but not significantly increased bandwidth. Secondly, while the capabilities to mitigate such attacks existed, the specific countermeasures were not pre-configured, leading to valuable time spent on configuration, testing, and tuning during the incident.

In response, GitHub adjusted its monitoring to better detect and alert on indicative traffic pattern changes. Automation was implemented to enable mitigation for specific attack patterns, and Hubot was updated with new templates for rapid response. The company also planned to investigate attack simulation and engage third-party security consultants to proactively plan for novel attack types.