Incident with Issues
Incident
Report for GitHub
Resolved
On July 21st, 2025, between 07:00 UTC and 09:45 UTC the API, Codespaces, Copilot, Issues, Package Registry, Pull Requests and Webhook services were degraded and experienced dropped requests and increased latency. At the peak of this incident (a 2 minute period around 07:00 UTC) error rates peaked at 11% and went down shortly after. Average web request load times rose to 1 second during this same time frame. After this period, traffic gradually recovered but error rate and latency remained slightly elevated until the end of the incident.
This incident was triggered by a kernel bug that caused a crash of some of our load balancers during a scheduled process after a kernel upgrade. In order to mitigate the incident, we halted the roll out of our upgrades, and rolled back the impacted instances. We are working to make sure the kernel version is fully removed from our fleet. As a precaution, we temporarily paused the scheduled process to prevent any unintended use in the affected kernel. We also tuned our alerting so we can more quickly detect and mitigate future instances where we experience a sudden drop in load-balancing capacity.
This incident was triggered by a kernel bug that caused a crash of some of our load balancers during a scheduled process after a kernel upgrade. In order to mitigate the incident, we halted the roll out of our upgrades, and rolled back the impacted instances. We are working to make sure the kernel version is fully removed from our fleet. As a precaution, we temporarily paused the scheduled process to prevent any unintended use in the affected kernel. We also tuned our alerting so we can more quickly detect and mitigate future instances where we experience a sudden drop in load-balancing capacity.
Update
API Requests and Codespaces are operating normally.
Update
Copilot is operating normally.
Update
Webhooks is operating normally.
Update
Mitigations have been applied and we are seeing recovery. We are continuing to closely monitor the situation to ensure complete recovery has been achieved.
Update
Issues is operating normally.
Update
Packages is operating normally.
Update
We are currently implementing mitigations for this issue.
Update
Copilot is experiencing degraded performance. We are continuing to investigate.
Update
We continue to investigate reports of degraded performance and intermittent timeouts across GitHub.com.
Update
Pull Requests is operating normally.
Update
We're continuing to investigate reports of degraded performance and intermitiant timeouts across GitHub.com.
Update
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Update
Packages is experiencing degraded performance. We are continuing to investigate.
Update
API Requests is experiencing degraded performance. We are continuing to investigate.
Update
Codespaces is experiencing degraded performance. We are continuing to investigate.
Investigating
We are investigating reports of degraded performance for Issues and Webhooks
This incident affected: Webhooks, API Requests, Issues, Pull Requests, Packages, Codespaces, and Copilot.
Subscribe to our developer newsletter
Get tips, technical guides, and best practices. Twice a month. Right in your inbox.
Subscribe