Incident with Packages, Issues, Pull Requests and Webhooks
Incident
Report for GitHub
Resolved
On December 19, 2023, between 20:30 and 22:50 UTC multiple services experienced intermittent errors communicating with our MySQL backends. An average of 0.12% and max of 1.30% of requests hit 5xx errors and timeouts for web and API requests during this period. The majority of impact was mitigated at 22:10, with a lower rate of failures ending by 22:50. This issue was caused by unhandled exceptions during routine maintenance of our internal networking hosts. Hosts that were thought to be no longer serving traffic were removed from service which led to the initial errors. Recovery took longer than expected because of issues in our automated recovery tooling. As a result, manual intervention was performed to remove the unhealthy hosts and requests returned to normal. The bug in the automated recovery tooling has been fixed and we’ve made changes to prevent unintended removal of hosts that are serving traffic.
Update
Webhooks is operating normally.
Update
Issues is operating normally.
Update
Pull Requests is operating normally.
Update
We applied a configuration change to our database, and we are seeing a recovery.
Update
Packages is operating normally.
Update
Webhooks is experiencing degraded performance. We are continuing to investigate.
Update
We are seeing database connection issues among our applications. We are investigating the root cause.
Update
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Update
Issues is experiencing degraded performance. We are continuing to investigate.
Investigating
We are investigating reports of degraded performance for Packages
This incident affected: Webhooks, Issues, Pull Requests, and Packages.
