Incident with multiple GitHub services
Incident
Report for GitHub
Resolved
On July 17, 2024, between 16:15:31 UTC and 17:06:53 UTC, various GitHub services were degraded including Login, the GraphQL API, Issues, Pages and Packages. On average, the error rate was 0.3% for requests to github.com and the API, and 3.0% of requests for Packages. This incident was triggered by two unrelated events:
- A planned testing event of an internal feature caused heavy loads on our databases, disrupting services across GitHub.
- A network configuration change deployed to support capacity expansion in a GitHub data center.
We partially resolved the incident by aborting the testing event at 16:17 UTC and fully resolved the incident by rolling back the network configuration changes at 16:49 UTC.
We have paused all planned capacity expansion activity within GitHub data centers until we have stabilized the root cause of this incident. In addition, we are reexamining our load testing practices so they can be done in a safer environment and making architectural changes to the feature that caused issues.
- A planned testing event of an internal feature caused heavy loads on our databases, disrupting services across GitHub.
- A network configuration change deployed to support capacity expansion in a GitHub data center.
We partially resolved the incident by aborting the testing event at 16:17 UTC and fully resolved the incident by rolling back the network configuration changes at 16:49 UTC.
We have paused all planned capacity expansion activity within GitHub data centers until we have stabilized the root cause of this incident. In addition, we are reexamining our load testing practices so they can be done in a safer environment and making architectural changes to the feature that caused issues.
Update
Git Operations is operating normally.
Update
Pages is operating normally.
Update
Packages is operating normally.
Update
We're continuing to investigate reports of issues with multiple services. We will continue to keep users updated on progress.
Update
Packages is experiencing degraded performance. We are continuing to investigate.
Update
Issues is operating normally.
Update
Pages is experiencing degraded performance. We are continuing to investigate.
Update
Issues is experiencing degraded performance. We are continuing to investigate.
Update
We are investigating reports of issues with service(s): Git, Actions, Rules and Permissions, SSH authentication, and Authorization.. We will continue to keep users updated on progress towards mitigation.
Investigating
We are investigating reports of degraded performance for Git Operations
This incident affected: Git Operations, Issues, Packages, and Pages.
Subscribe to our developer newsletter
Get tips, technical guides, and best practices. Twice a month. Right in your inbox.
Subscribe