Incident with Copilot
Incident
Report for GitHub
Resolved
Copilot completions are currently hosted in 4 regions globally: Central US, France, Switzerland and Japan. Users are typically routed to the nearest geographic region, but may be routed to other regions when the nearest region is unhealthy.
Beginning at 2023-10-25 09:13 UTC, Copilot began experiencing outages of individual regions, lasting 12 minutes per region. These outages were due to the nodes hosting the completion model getting unhealthy due to a recent upgrade. There were intermittent outages in multiple regions with a subset of Copilot users experiencing completion errors. The outages were partial and varied across the different regions.
In order to prevent similar incidents from occurring in the future, we are focusing on improving our global load balancing of completion traffic during regional failures, in addition to determining and preventing the root cause of these outages.
Beginning at 2023-10-25 09:13 UTC, Copilot began experiencing outages of individual regions, lasting 12 minutes per region. These outages were due to the nodes hosting the completion model getting unhealthy due to a recent upgrade. There were intermittent outages in multiple regions with a subset of Copilot users experiencing completion errors. The outages were partial and varied across the different regions.
In order to prevent similar incidents from occurring in the future, we are focusing on improving our global load balancing of completion traffic during regional failures, in addition to determining and preventing the root cause of these outages.
Update
The observed Copilot API error rate is around 5% of the requests. As a result, some of the Copilot code suggestions are skipped or not delivered on time.
Update
We are seeing an impact in the US region as well. We continue the investigation.
Update
Copilot is experiencing intermittent issues in our Japan region. Engineers are currently investigating.
Investigating
We are investigating reports of degraded performance for Copilot
This incident affected: Copilot.
Subscribe to our developer newsletter
Get tips, technical guides, and best practices. Twice a month. Right in your inbox.
Subscribe