Incident with Pull Requests
Incident
Report for GitHub
Resolved
On October 9, 2023 at 14:24 UTC, a noticeable delay in commits appearing in pull requests was detected. During the incident, approximately 9% of pull requests (less than the 20% first reported) experienced staleness of up to 7m. The root cause was identified to be an increase in the latency of a downstream dependency causing pull request workers to saturate their available capacity, resulting in delayed updates to PRs - no data was lost during this incident.
We mitigated this by adding additional capacity to the affected worker pool at 15:02 UTC. This allowed our background jobs to catch up with the backlog of updates and provide relief to our customers. Additionally, we have significantly increased the performance of the downstream service to prevent recurrence
We mitigated this by adding additional capacity to the affected worker pool at 15:02 UTC. This allowed our background jobs to catch up with the backlog of updates and provide relief to our customers. Additionally, we have significantly increased the performance of the downstream service to prevent recurrence
Update
We are investigating delays for commits showing up on Pull Requests page loads in the web UI. As a result of this, about 20% of pull requests are currently showing stale data of up-to 7m. We are currently investigating contributing factors right now.
Investigating
We are investigating reports of degraded performance for Pull Requests
This incident affected: Pull Requests.
Subscribe to our developer newsletter
Get tips, technical guides, and best practices. Twice a month. Right in your inbox.
Subscribe