Incident with Actions, Pages, Pull Requests, Codespaces, API Requests, Issues, Webhooks and Packages
Incident
Report for GitHub
Resolved
From 16:24-16:43 UTC, multiple GitHub services were down or degraded due to an outage in one of our primary databases.
The primary host for a shared datastore for GitHub experienced an underlying file system write error which affected availability for the majority of public-facing GitHub services. SAML login was affected, as was access to Actions, Issues, Pull Requests, Pages, API, Webhooks, Codespaces, and Packages. In this case, our automatic failover was unable to handle the partial file system failure mode. We mitigated by manually performing a forced failover, initiated 17 minutes after our first alert and completed 2 minutes later.
With the incident mitigated, we are working to assess more detailed impact and resilience improvements to each impacted service while also improving the automated failover mechanism to support this scenario.
The primary host for a shared datastore for GitHub experienced an underlying file system write error which affected availability for the majority of public-facing GitHub services. SAML login was affected, as was access to Actions, Issues, Pull Requests, Pages, API, Webhooks, Codespaces, and Packages. In this case, our automatic failover was unable to handle the partial file system failure mode. We mitigated by manually performing a forced failover, initiated 17 minutes after our first alert and completed 2 minutes later.
With the incident mitigated, we are working to assess more detailed impact and resilience improvements to each impacted service while also improving the automated failover mechanism to support this scenario.
Update
Pages is operating normally.
Update
Actions is operating normally.
Update
Issues is operating normally.
Update
We have performed a mitigation affecting write traffic across services and are seeing recovery for affected customers.
Update
Packages is operating normally.
Update
Webhooks is operating normally.
Update
API Requests is operating normally.
Update
Codespaces is operating normally.
Update
Packages is experiencing degraded performance. We are continuing to investigate.
Update
Webhooks is experiencing degraded performance. We are continuing to investigate.
Update
We are investigating an issue that is impacting a small percentage of requests across all services including authentication and are working on mitigating impact.
Update
Issues is experiencing degraded performance. We are continuing to investigate.
Update
API Requests is experiencing degraded performance. We are continuing to investigate.
Update
Codespaces is experiencing degraded availability. We are continuing to investigate.
Update
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Update
Pages is experiencing degraded performance. We are continuing to investigate.
Investigating
We are investigating reports of degraded performance for Actions
This incident affected: API Requests, Webhooks, Issues, Pull Requests, Actions, Packages, Pages, and Codespaces.
