Incident with Copilot
Incident
Report for GitHub
Resolved
Copilot completions are currently hosted in 4 regions globally: Central US, France, Switzerland and Japan. Users are typically routed to the nearest geographic region, but may be routed to other regions when the nearest region is unhealthy.
Beginning at 2023-10-25 09:13 UTC, Copilot began experiencing outages of individual regions, lasting 12 minutes per region. These outages were due to the nodes hosting the completion model getting unhealthy due to a recent upgrade. There were intermittent outages in multiple regions with a subset of Copilot users experiencing completion errors. The outages were partial and varied across the different regions.
In order to prevent similar incidents from occurring in the future, we are focusing on improving our global load balancing of completion traffic during regional failures, in addition to determining and preventing the root cause of these outages.
Beginning at 2023-10-25 09:13 UTC, Copilot began experiencing outages of individual regions, lasting 12 minutes per region. These outages were due to the nodes hosting the completion model getting unhealthy due to a recent upgrade. There were intermittent outages in multiple regions with a subset of Copilot users experiencing completion errors. The outages were partial and varied across the different regions.
In order to prevent similar incidents from occurring in the future, we are focusing on improving our global load balancing of completion traffic during regional failures, in addition to determining and preventing the root cause of these outages.
Update
We have applied a fix to help with Copilot performance. Initial signals show good recovery. We will continue to monitor for the time being and resolve when confident the issue has been resolved.
Update
We are investigating degraded performance in Europe for Copilot. We will continue to keep users updated on progress towards mitigation.
Investigating
We are investigating reports of degraded performance for Copilot
This incident affected: Copilot.
Subscribe to our developer newsletter
Get tips, technical guides, and best practices. Twice a month. Right in your inbox.
Subscribe