GitHub header
Incident with Webhooks
Incident Report for GitHub
Resolved
On January 9, 2025, between 01:26 UTC and 01:56 UTC GitHub experienced widespread disruption to many services, with users receiving 500 responses when trying to access various functionality. This was due to a deployment which introduced a query that saturated a primary database server. On average, the error rate was 6% and peaked at 6.85% of update requests.

We mitigated the incident by identifying the source of the problematic query and rolling back the deployment.

We are investigating methods to detect problematic queries prior to deployment to prevent, and to reduce our time to detection and mitigation of issues like this one in the future.
Posted Jan 09, 2025 - 02:27 UTC
Update
We have identified the root cause and have deployed a fix. Majority of the services have recovered. Actions service is in the process of being recovered.
Posted Jan 09, 2025 - 02:19 UTC
Update
Copilot is operating normally.
Posted Jan 09, 2025 - 02:14 UTC
Update
Issues is operating normally.
Posted Jan 09, 2025 - 02:13 UTC
Update
Pages is operating normally.
Posted Jan 09, 2025 - 02:13 UTC
Update
Git Operations is operating normally.
Posted Jan 09, 2025 - 02:12 UTC
Update
Webhooks is operating normally.
Posted Jan 09, 2025 - 02:12 UTC
Update
Pull Requests is operating normally.
Posted Jan 09, 2025 - 02:12 UTC
Update
Codespaces is operating normally.
Posted Jan 09, 2025 - 02:11 UTC
Update
We have identified the root cause and have deployed a fix. Service are recovering.
Posted Jan 09, 2025 - 02:09 UTC
Update
API Requests is experiencing degraded performance. We are continuing to investigate.
Posted Jan 09, 2025 - 02:01 UTC
Update
We are continuing the investigation of multiple service issues. We will continue to keep users updated on progress towards mitigation.
Posted Jan 09, 2025 - 01:59 UTC
Update
Copilot is experiencing degraded performance. We are continuing to investigate.
Posted Jan 09, 2025 - 01:53 UTC
Update
Codespaces is experiencing degraded performance. We are continuing to investigate.
Posted Jan 09, 2025 - 01:51 UTC
Update
Codespaces is experiencing degraded availability. We are continuing to investigate.
Posted Jan 09, 2025 - 01:51 UTC
Update
Git Operations is experiencing degraded availability. We are continuing to investigate.
Posted Jan 09, 2025 - 01:49 UTC
Update
We are investigating reports of issues with multiple services including authentication, PRs, codespaces, pages, git operation, and apis. We will continue to keep users updated on progress towards mitigation.
Posted Jan 09, 2025 - 01:46 UTC
Update
Pages is experiencing degraded performance. We are continuing to investigate.
Posted Jan 09, 2025 - 01:44 UTC
Update
Git Operations is experiencing degraded performance. We are continuing to investigate.
Posted Jan 09, 2025 - 01:43 UTC
Update
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Posted Jan 09, 2025 - 01:42 UTC
Update
Issues is experiencing degraded performance. We are continuing to investigate.
Posted Jan 09, 2025 - 01:41 UTC
Update
Actions is experiencing degraded performance. We are continuing to investigate.
Posted Jan 09, 2025 - 01:37 UTC
Investigating
We are investigating reports of degraded availability for Webhooks
Posted Jan 09, 2025 - 01:36 UTC
This incident affected: Git Operations, Webhooks, API Requests, Issues, Pull Requests, Actions, Pages, Codespaces, and Copilot.