We are investigating reports of degraded performance.
Incident
Report for GitHub
Resolved
On 2023-11-15, from 09:44 to 10:42 UTC, some GitHub customers experienced increased latency or errors accessing repo data.
High concurrent access to a specific git object exposed a bug that forced a backend service to perform excessive calculations, overloading the service. Access to this repo was paused while load was re-rerouted, mitigating the problem.
The conditions that triggered the expensive operations have been identified and refactored.
High concurrent access to a specific git object exposed a bug that forced a backend service to perform excessive calculations, overloading the service. Access to this repo was paused while load was re-rerouted, mitigating the problem.
The conditions that triggered the expensive operations have been identified and refactored.
Update
Error rates and performance have returned to normal.
Update
We have identified the source of the issue and have removed the additional load from the service. Sporadic delays in pull request experiences and intermittent 500s are still occurring and impacting a very small percentage of traffic. Next update is expected within 30 minutes.
Update
We are seeing connectivity issues between some of our systems and git backend services. This is causing intermittent error responses and delays in pull request experiences for a very small percentage of traffic. We are investigating mitigations and expect to provide another update within 30 minutes.
Investigating
We are currently investigating this issue.
