Incident with Git Operations, Issues, Pull Requests, Actions, API Requests, Codespaces, Packages, Pages and Webhooks
Incident
Report for GitHub
Resolved
A performance and resilience optimization to the authorization microservice contained a memory leak that was exposed under high traffic. This resulted in a number of pages returning 404’s that should not have. Testing the build in our canary ring did not expose the service to sufficient traffic to discover the leak, allowing it to graduate to production at 6:37 PM UTC. The memory leak under high load caused pods to crash repeatedly starting at 6:42 PM UTC, failing authorization checks. These failures triggered alerts at 6:44 PM UTC. Rolling back the authorization service change was delayed as parts of the deployment infrastructure relied on the authorization service and required manual intervention to complete. Rollback completed at 7:08 PM UTC and all impacted GitHub features recovered after pods came back online. We are evaluating changes to our rollout strategy to better detect this sooner and with less impact, changes to remove the dependency between authorization services and deployment rollback, and incident response improvements to reduce the overall time to recover.
This incident is unrelated to the Slack integration incident. The combined status updates are a limitation of our status reporting tooling and we recognize the confusion this creates. Work to address this was already in progress and will be complete this month.
This incident is unrelated to the Slack integration incident. The combined status updates are a limitation of our status reporting tooling and we recognize the confusion this creates. Work to address this was already in progress and will be complete this month.
Update
Slack notifications have recovered.
Update
Webhooks is operating normally.
Update
Pull Requests is operating normally.
Update
Issues is operating normally.
Update
Git Operations is operating normally.
Update
API Requests is operating normally.
Update
Actions is operating normally.
Update
Pages is operating normally.
Update
Codespaces is operating normally.
Update
Packages is operating normally.
Update
We have completed the rollback and are monitoring recovery.
Update
We’re in the process of rolling back an authorization-related change that is causing 404s and other errors.
Update
Packages is experiencing degraded availability. We are continuing to investigate.
Update
Pages is experiencing degraded availability. We are continuing to investigate.
Update
Actions is experiencing degraded availability. We are continuing to investigate.
Update
Packages is experiencing degraded performance. We are continuing to investigate.
Update
Codespaces is experiencing degraded performance. We are continuing to investigate.
Update
API Requests is experiencing degraded performance. We are continuing to investigate.
Update
Actions is experiencing degraded performance. We are continuing to investigate.
Update
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Update
Issues is experiencing degraded performance. We are continuing to investigate.
Update
Git Operations is experiencing degraded performance. We are continuing to investigate.
Update
The delayed Slack notifications should be fully processed in about 30 minutes.
Update
Delayed Slack notifications are processing and the queue is expected to clear in just over an hour.
Update
Users may see delayed Slack notifications coming through as the queue is processed.
Update
Fix has been deployed, Slack integrations are recovering
Update
We are testing a fix for Slack integrations.
Update
We are aware of issues with Slack integration and are working on resolving the problem.
Investigating
We are currently investigating this issue.
This incident affected: Git Operations, API Requests, Webhooks, Issues, Pull Requests, Actions, Packages, Pages, and Codespaces.
