Incident with Issues, API Requests and Git Operations
Incident
Report for GitHub
Resolved
On 2023-09-19 at 20:36 UTC, while migrating the primary datastore for Projects, we caused an outage that led to Projects data becoming unavailable for approximately four hours.
While working to restore Projects data, we also experienced a data replication interruption which affected Git Operations, APIs, and Issues. We first resolved the data replication issue, which returned Git Operations, APIs, and Issues to normal operation. We then restored Projects data to its pre-migration state, and re-inserted data that was added during the period of partial availability. The incident was resolved on 2023-09-20 at 00:06 UTC.
As a result of this incident, we have improved validation of data migrations in test and during rollout. We have also identified improvements to reduce both the time to restore and to fix the replication issue.
While working to restore Projects data, we also experienced a data replication interruption which affected Git Operations, APIs, and Issues. We first resolved the data replication issue, which returned Git Operations, APIs, and Issues to normal operation. We then restored Projects data to its pre-migration state, and re-inserted data that was added during the period of partial availability. The incident was resolved on 2023-09-20 at 00:06 UTC.
As a result of this incident, we have improved validation of data migrations in test and during rollout. We have also identified improvements to reduce both the time to restore and to fix the replication issue.
Update
Issues is operating normally.
Update
We experienced an incident causing missing Project item data. We have restored all data from before the incident, up to 19 Sep 20:36 UTC. Project data modified between 20:36 UTC and 20 Sep 00:06 UTC was partially restored.
We will publish a root cause analysis for this issue by Sep 29.
We will publish a root cause analysis for this issue by Sep 29.
Update
Projects data has been restored to the state prior to the incident (19 Sep 20:57 UTC). We are continuing to restore the last of Project metadata updated since 20:57 UTC.
Update
Projects data has been restored to the state prior to the incident (19 Sep 20:57 UTC). We are continuing to restore the last of Project metadata updated since 20:57 UTC.
Update
While rolling out an update to Projects, Project data was affected and needed to be restored. While restoring data, we experienced a replication interruption which affected Git Operations, Issues and APIs. We restarted replication, and restored Projects data to its state prior to the rollout, and continue working to restore the last of Project metadata updated since 19 Sep 20:57 UTC.
Update
Issues is experiencing degraded performance. We are continuing to investigate.
Update
We've restored access to a majority of Project data. A small number of customers are still affected, and we continue to work on full restoration.
Update
We continue restoring access to Project data. We do not have an estimated time to full restoration at this time, and are working to provide that.
Update
We continue working on restoring access to Projects data, along with an estimated time to full restoration.
Update
API Requests is operating normally.
Update
Git Operations is operating normally.
Update
Git operations, API requests, and Issues are recovering after restoring data replication, and we continue working on restoring access to Projects data.
Update
Git Operations is experiencing degraded performance. We are continuing to investigate.
Update
* We are actively working on restoring access to Projects data.
* Issues and APIs are experiencing degraded performance. We are working on mitigating the issue.
* Issues and APIs are experiencing degraded performance. We are working on mitigating the issue.
Update
API Requests is experiencing degraded performance. We are continuing to investigate.
Update
We have identified the issue and are actively working to restore missing project items
Update
Issues is experiencing degraded availability. We are continuing to investigate.
Update
We are currently investigating an issue with project items not loading
Investigating
We are currently investigating this issue.
This incident affected: Git Operations, API Requests, and Issues.
