GitHub header

Incident with Issues, Git Operations and API Requests

Incident Report for GitHub

Resolved

On March 3rd 2025 between 04:07 UTC and 09:36 UTC various GitHub services were degraded with an average error rate of 0.03% and peak error rate of 9%. This issue impacted web requests, API requests, and git operations.

This incident was triggered because a network node in one of GitHub's datacenter sites partially failed, resulting in silent packet drops for traffic served by that site. At 09:22 UTC, we identified the failing network node, and at 09:36 UTC we addressed the issue by removing the faulty network node from production.

In response to this incident, we are improving our monitoring capabilities to identify and respond to similar silent errors more effectively in the future.
Posted Mar 03, 2025 - 05:31 UTC

Update

We have seen recovery across our services and impact is mitigated.
Posted Mar 03, 2025 - 05:30 UTC

Update

Git Operations is operating normally.
Posted Mar 03, 2025 - 05:20 UTC

Update

Webhooks is operating normally.
Posted Mar 03, 2025 - 05:20 UTC

Update

We are investigating intermittent connectivity issues between our backend and databases and will provide further updates as we have them. The current impact is you may see elevated latency while using our services.
Posted Mar 03, 2025 - 04:54 UTC

Update

We are seeing intermittent timeouts across our various services. We are currently investigating and will provide updates.
Posted Mar 03, 2025 - 04:23 UTC

Update

Webhooks is experiencing degraded performance. We are continuing to investigate.
Posted Mar 03, 2025 - 04:21 UTC

Investigating

We are investigating reports of degraded performance for API Requests, Git Operations and Issues
Posted Mar 03, 2025 - 04:20 UTC
This incident affected: Git Operations, Webhooks, API Requests, and Issues.