All GitHub services are experiencing significant disruptions
Incident
Report for GitHub
Resolved
On August 14, 2024 between 23:02 UTC and 23:38 UTC, all GitHub services on GitHub.com were inaccessible for all users.
This was due to a configuration change that impacted traffic routing within our database infrastructure, resulting in critical services unexpectedly losing database connectivity. There was no data loss or corruption during this incident.
At 22:59 UTC an erroneous configuration change rolled out to all GitHub.com databases that impacted the ability of the database to respond to health check pings from the routing service. As a result, the routing service could not detect healthy databases to route application traffic to. This led to widespread impact on GitHub.com starting at 23:02 UTC.
We mitigated the incident by reverting the change and confirming restored connectivity to our databases. At 23:38 UTC, traffic resumed and all services recovered to full health. Out of an abundance of caution, we continued to monitor before resolving the incident at 00:30 UTC on August 15th, 2024.
To prevent recurrence we are implementing additional guardrails in our database change management process. We are also prioritizing several repair items such as faster rollback functionality and more resilience to dependency failures.
Given the severity of this incident, follow-up items are the highest priority work for teams at this time.
This was due to a configuration change that impacted traffic routing within our database infrastructure, resulting in critical services unexpectedly losing database connectivity. There was no data loss or corruption during this incident.
At 22:59 UTC an erroneous configuration change rolled out to all GitHub.com databases that impacted the ability of the database to respond to health check pings from the routing service. As a result, the routing service could not detect healthy databases to route application traffic to. This led to widespread impact on GitHub.com starting at 23:02 UTC.
We mitigated the incident by reverting the change and confirming restored connectivity to our databases. At 23:38 UTC, traffic resumed and all services recovered to full health. Out of an abundance of caution, we continued to monitor before resolving the incident at 00:30 UTC on August 15th, 2024.
To prevent recurrence we are implementing additional guardrails in our database change management process. We are also prioritizing several repair items such as faster rollback functionality and more resilience to dependency failures.
Given the severity of this incident, follow-up items are the highest priority work for teams at this time.
Update
We have fully rolled-back the changes to database infrastructure and mitigated the impact. All services are now fully operational.
Update
Git Operations is operating normally.
Update
Copilot is operating normally.
Update
Codespaces, Packages and Pages are operating normally.
Update
Webhooks is operating normally.
Update
Actions is operating normally.
Update
Actions is operating normally.
Update
Actions is experiencing degraded performance. We are continuing to investigate.
Update
Actions is operating normally.
Update
Pull Requests is operating normally.
Update
Issues is operating normally.
Update
Copilot is experiencing degraded performance. We are continuing to investigate.
Update
Git Operations is experiencing degraded performance. We are continuing to investigate.
Update
Codespaces is experiencing degraded performance. We are continuing to investigate.
Update
Webhooks is experiencing degraded performance. We are continuing to investigate.
Update
Actions is experiencing degraded performance. We are continuing to investigate.
Update
Pages is experiencing degraded performance. We are continuing to investigate.
Update
Service health continues to improve, and we are working to stabilize all services. Some services may experience delays in updates and notifications as we work through a backlog of events.
Update
Packages is experiencing degraded performance. We are continuing to investigate.
Update
Issues is experiencing degraded performance. We are continuing to investigate.
Update
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Update
The database infrastructure change is being rolled back. We are seeing improvements in service health and are monitoring for full recovery.
Update
We are experiencing interruptions in multiple public GitHub services. We suspect the impact is due to a database infrastructure related change that we are working on rolling back.
Update
Codespaces is experiencing degraded availability. We are continuing to investigate.
Update
Webhooks is experiencing degraded availability. We are continuing to investigate.
Update
Issues is experiencing degraded availability. We are continuing to investigate.
Update
Git Operations is experiencing degraded availability. We are continuing to investigate.
Update
Packages is experiencing degraded availability. We are continuing to investigate.
Update
We are investigating reports of issues with GitHub.com and GitHub API. We will continue to keep users updated on progress towards mitigation.
Update
Copilot is experiencing degraded availability. We are continuing to investigate.
Update
Pages is experiencing degraded availability. We are continuing to investigate.
Investigating
We are investigating reports of degraded availability for Actions, Pages and Pull Requests
This incident affected: Git Operations, Webhooks, Issues, Pull Requests, Actions, Packages, Pages, Codespaces, and Copilot.
Subscribe to our developer newsletter
Get tips, technical guides, and best practices. Twice a month. Right in your inbox.
Subscribe