Incident with Webhooks
Incident
Report for GitHub
Resolved
On January 9, 2025, between 01:26 UTC and 01:56 UTC GitHub experienced widespread disruption to many services, with users receiving 500 responses when trying to access various functionality. This was due to a deployment which introduced a query that saturated a primary database server. On average, the error rate was 6% and peaked at 6.85% of update requests.
We mitigated the incident by identifying the source of the problematic query and rolling back the deployment.
We are investigating methods to detect problematic queries prior to deployment to prevent, and to reduce our time to detection and mitigation of issues like this one in the future.
We mitigated the incident by identifying the source of the problematic query and rolling back the deployment.
We are investigating methods to detect problematic queries prior to deployment to prevent, and to reduce our time to detection and mitigation of issues like this one in the future.
Update
We have identified the root cause and have deployed a fix. Majority of the services have recovered. Actions service is in the process of being recovered.
Update
Copilot is operating normally.
Update
Issues is operating normally.
Update
Pages is operating normally.
Update
Git Operations is operating normally.
Update
Webhooks is operating normally.
Update
Pull Requests is operating normally.
Update
Codespaces is operating normally.
Update
We have identified the root cause and have deployed a fix. Service are recovering.
Update
API Requests is experiencing degraded performance. We are continuing to investigate.
Update
We are continuing the investigation of multiple service issues. We will continue to keep users updated on progress towards mitigation.
Update
Copilot is experiencing degraded performance. We are continuing to investigate.
Update
Codespaces is experiencing degraded performance. We are continuing to investigate.
Update
Codespaces is experiencing degraded availability. We are continuing to investigate.
Update
Git Operations is experiencing degraded availability. We are continuing to investigate.
Update
We are investigating reports of issues with multiple services including authentication, PRs, codespaces, pages, git operation, and apis. We will continue to keep users updated on progress towards mitigation.
Update
Pages is experiencing degraded performance. We are continuing to investigate.
Update
Git Operations is experiencing degraded performance. We are continuing to investigate.
Update
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Update
Issues is experiencing degraded performance. We are continuing to investigate.
Update
Actions is experiencing degraded performance. We are continuing to investigate.
Investigating
We are investigating reports of degraded availability for Webhooks
This incident affected: Git Operations, Webhooks, API Requests, Issues, Pull Requests, Actions, Pages, Codespaces, and Copilot.
Subscribe to our developer newsletter
Get tips, technical guides, and best practices. Twice a month. Right in your inbox.
Subscribe