Incident with Issues, API Requests, Pull Requests, Actions, Pages, Git Operations, Webhooks, Packages and Codespaces
Incident
Report for GitHub
Resolved
On January 9 between 12:45 and 13:56 UTC, services in one of our three sites experienced elevated latency for connections. This led to a sustained period of timed out requests across a number of services, including but not limited to our git backend. An average of 5% and max of 10% of requests failed with a 5xx response or timed out during this period. This was caused by a combination of events that led to connection limits being hit in load balancer proxies in that site. An upgrade of hosts was in flight, which meant a subset of proxy hosts were draining and coming offline as the upgrade rolled through the fleet. A config change event also triggered a connection reset across all services in that site. These events are commonplace, but led to a spike in connection establishment events that led to the online proxy hosts hitting the connection limit. Upon further analysis, that limit was lower than it should have been. We have increased that limit to prevent this from recurring. We have also identified improvements to our monitoring of connection limits and behavior and changes to reduce the risk of proxy host upgrades leading to reduced capacity.
Update
Pull Requests is operating normally.
Update
Packages is operating normally.
Update
Git Operations is operating normally.
Update
Codespaces is operating normally.
Update
Actions is operating normally.
Update
API Requests is operating normally.
Update
Issues is operating normally.
Update
Webhooks is operating normally.
Update
Pages is operating normally.
Update
API Requests is experiencing degraded performance. We are continuing to investigate.
Update
5xx error rates remain elevated but are seeing a downward trend with many services fully recovered. We will continue monitoring the situation and keep users updated on progress toward full recovery.
Update
Actions is experiencing degraded performance. We are continuing to investigate.
Update
We are experiencing an elevated rate of 5xx errors on the order of 1-5% being returned from numerous APIs across the site. The issue has been isolated to one datacenter. We will continue to keep users updated on progress towards mitigation.
Update
Codespaces is experiencing degraded performance. We are continuing to investigate.
Update
Packages is experiencing degraded performance. We are continuing to investigate.
Update
Webhooks is experiencing degraded performance. We are continuing to investigate.
Update
Git Operations is experiencing degraded performance. We are continuing to investigate.
Update
Pages is experiencing degraded performance. We are continuing to investigate.
Update
API Requests is experiencing degraded availability. We are continuing to investigate.
Update
We are an increase in the rate of 5xx errors on the order of 1-3% being returned from numerous APIs across the site. We will continue to keep users updated on progress towards mitigation.
Update
Actions is experiencing degraded availability. We are continuing to investigate.
Update
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Investigating
We are investigating reports of degraded performance for Issues and API Requests
This incident affected: Git Operations, API Requests, Webhooks, Issues, Pull Requests, Actions, Packages, Pages, and Codespaces.
