Incident with Issues, Git Operations and API Requests
Incident
Report for GitHub
Resolved
On March 3rd 2025 between 04:07 UTC and 09:36 UTC various GitHub services were degraded with an average error rate of 0.03% and peak error rate of 9%. This issue impacted web requests, API requests, and git operations.
This incident was triggered because a network node in one of GitHub's datacenter sites partially failed, resulting in silent packet drops for traffic served by that site. At 09:22 UTC, we identified the failing network node, and at 09:36 UTC we addressed the issue by removing the faulty network node from production.
In response to this incident, we are improving our monitoring capabilities to identify and respond to similar silent errors more effectively in the future.
This incident was triggered because a network node in one of GitHub's datacenter sites partially failed, resulting in silent packet drops for traffic served by that site. At 09:22 UTC, we identified the failing network node, and at 09:36 UTC we addressed the issue by removing the faulty network node from production.
In response to this incident, we are improving our monitoring capabilities to identify and respond to similar silent errors more effectively in the future.
Update
We have seen recovery across our services and impact is mitigated.
Update
Git Operations is operating normally.
Update
Webhooks is operating normally.
Update
We are investigating intermittent connectivity issues between our backend and databases and will provide further updates as we have them. The current impact is you may see elevated latency while using our services.
Update
We are seeing intermittent timeouts across our various services. We are currently investigating and will provide updates.
Update
Webhooks is experiencing degraded performance. We are continuing to investigate.
Investigating
We are investigating reports of degraded performance for API Requests, Git Operations and Issues
This incident affected: Git Operations, Webhooks, API Requests, and Issues.
Subscribe to our developer newsletter
Get tips, technical guides, and best practices. Twice a month. Right in your inbox.
Subscribe