Incident with Git Operations and Actions
Incident
Report for GitHub
Resolved
Between 06:59 UTC to 13:18 UTC GitHub customers experienced intermittent increased latency when using our git archive and raw file downloading capability. This capability is typically used for larger CI/CD systems and bulk download of repository data. This also caused intermittent periods of Actions workflow failures.
During this incident we saw bursts of increased unique traffic on specific endpoints that overloaded our caching capability for this system. In addition, the traffic caused saturation of the service that exposed a bug in downloading archived files at high load which caused some archives to be downloaded with incomplete data. This caused intermittent failure of workflows for GitHub Actions, which primarily downloads actions data via these endpoints.
We mitigated this through improvements to our rate limiting and changes to get to a healthy state across our backend service and cache . We initially relied on automated communication messages during this incident and will be working to ensure future incidents have more relevant status updates with specific customer impact.
During this incident we saw bursts of increased unique traffic on specific endpoints that overloaded our caching capability for this system. In addition, the traffic caused saturation of the service that exposed a bug in downloading archived files at high load which caused some archives to be downloaded with incomplete data. This caused intermittent failure of workflows for GitHub Actions, which primarily downloads actions data via these endpoints.
We mitigated this through improvements to our rate limiting and changes to get to a healthy state across our backend service and cache . We initially relied on automated communication messages during this incident and will be working to ensure future incidents have more relevant status updates with specific customer impact.
Update
Recovery is in progress. We are still taking additional mitigation steps and will status back to green when those are completed and we are confident they are effective.
Update
We have mitigated the impact on download and raw file operations and are seeing recovery on response times but are continuing to monitor.
Update
Operations that download Git archives and raw files are seeing highly elevated response times (up to 4+ seconds) for users. Other Git operations are not impacted. We are working on mitigation.
Update
We are investigating reports of issues with service(s): Git Operations. We will continue to keep users updated on progress towards mitigation.
Update
We are investigating reports of issues with service(s): Git Operations. We will continue to keep users updated on progress towards mitigation.
Update
We are investigating reports of issues with service(s): Git Operations. We will continue to keep users updated on progress towards mitigation.
Update
Actions is operating normally.
Update
We are investigating reports of issues with service(s): Git Operations. We will continue to keep users updated on progress towards mitigation.
Update
We are investigating reports of issues with service(s): Git Operations. We will continue to keep users updated on progress towards mitigation.
Update
We are investigating reports of issues with service(s): Git Operations. We will continue to keep users updated on progress towards mitigation.
Update
We are investigating reports of issues with service(s): Git Operations, Actions. We will continue to keep users updated on progress towards mitigation.
Update
Actions is experiencing degraded performance. We are continuing to investigate.
Update
We are investigating reports of issues with service(s): Git Operations. We will continue to keep users updated on progress towards mitigation.
Update
We are investigating reports of issues with service(s): Git Operations. We will continue to keep users updated on progress towards mitigation.
Investigating
We are investigating reports of degraded performance for Git Operations
This incident affected: Git Operations and Actions.
