We're experiencing errors
Incident
Report for GitHub
Resolved
On May 26, 2025, between 06:20 UTC and 09:45 UTC GitHub experienced broad failures across a variety of services (API, Issues, Git, etc). These were degraded at times, but peaked at 100% failure rates for some operations during this time.
On May 23, a new feature was added to Copilot APIs and monitored during rollout but it was not tested at peak load. At 6:20 UTC on May 26, load increased on the code path in question and started to degrade a Copilot API because the caching for this endpoint and circuit breakers for high load were misconfigured.
In addition, the traffic limiting meant to protect wider swaths of the GitHub API from queuing was not yet covering this endpoint, meaning it was able to overwhelm the capacity to serve traffic and cause request queuing.
We were able to mitigate the incident by turning off the endpoint until the behavior could be reverted.
We are already working on a quality of service strategy for API endpoints like this that will limit the impact of a broad incident and are rolling it out. We are also addressing the specific caching and circuit breaker misconfigurations for this endpoint, which would have reduced the time to mitigate this particular incident and the blast radius.
On May 23, a new feature was added to Copilot APIs and monitored during rollout but it was not tested at peak load. At 6:20 UTC on May 26, load increased on the code path in question and started to degrade a Copilot API because the caching for this endpoint and circuit breakers for high load were misconfigured.
In addition, the traffic limiting meant to protect wider swaths of the GitHub API from queuing was not yet covering this endpoint, meaning it was able to overwhelm the capacity to serve traffic and cause request queuing.
We were able to mitigate the incident by turning off the endpoint until the behavior could be reverted.
We are already working on a quality of service strategy for API endpoints like this that will limit the impact of a broad incident and are rolling it out. We are also addressing the specific caching and circuit breaker misconfigurations for this endpoint, which would have reduced the time to mitigate this particular incident and the blast radius.
Update
We continue to see signs of recovery.
Update
Issues is operating normally.
Update
Git Operations is operating normally.
Update
API Requests is operating normally.
Update
Copilot is operating normally.
Update
Packages is operating normally.
Update
Actions is operating normally.
Update
Packages is experiencing degraded performance. We are continuing to investigate.
Update
Copilot is experiencing degraded performance. We are continuing to investigate.
Update
Actions is experiencing degraded performance. We are continuing to investigate.
Update
We are continuing to investigate degraded performance.
Update
Issues is experiencing degraded performance. We are continuing to investigate.
Investigating
We are investigating reports of degraded performance for API Requests and Git Operations
This incident affected: Git Operations, API Requests, Issues, Actions, Packages, and Copilot.
Subscribe to our developer newsletter
Get tips, technical guides, and best practices. Twice a month. Right in your inbox.
Subscribe