Incident with Copilot
Incident
Report for GitHub
Resolved
On Monday February 12th, 2024, 03:00 UTC we deployed a code change to a component of Copilot. At 06:00 UTC we observed an increase in timeouts for code completions impacting 55% of Copilot users at peak across Asia and Europe.
At 12:00 UTC we restarted the nodes, and response durations returned to normal operation until 13:00 UTC when response durations degraded again. At 16:15 UTC we made a configuration change to send traffic to regions that were not exhibiting the errors, which resulted in code completions working fully although completing at a higher latency than normal for some users. At 18:00 UTC we reverted the deploy and response durations returned to normal.
We have added better monitoring to components that failed to decrease resolution times to incidents like this in the future.
At 12:00 UTC we restarted the nodes, and response durations returned to normal operation until 13:00 UTC when response durations degraded again. At 16:15 UTC we made a configuration change to send traffic to regions that were not exhibiting the errors, which resulted in code completions working fully although completing at a higher latency than normal for some users. At 18:00 UTC we reverted the deploy and response durations returned to normal.
We have added better monitoring to components that failed to decrease resolution times to incidents like this in the future.
Update
Code completion response times have returned to normal.
Update
We’re still continuing to investigate slower than expected code completions for a subset of users in Europe. Next update to be provided in 30 minutes.
Update
We’re continuing to investigate slower than expected code completions for a subset of users in Europe. Next update to be provided in 30 minutes.
Update
We're continuing to investigate slower than expected code completions for a subset of users in Europe. Next update to be provided in 30 minutes.
Update
Code completions are now working for the impacted users, but completing more slowly than expected. Investigation continues to completely mitigate the issue and restore Copilot code completion functionality to normal.
Update
Following mitigation steps taken, we have reduced the impact to a more narrow subset of users. Investigation continues to completely mitigate the issue and restore Copilot code completion functionality.
Update
We are continuing to investigate the issues with Copilot code completion currently impacting some users in Europe. We will provide further details as we have them.
Update
We have confirmed that this is a reoccurrence of the earlier issue. Impact is currently limited to some European users. The team is working through alternative mitigation strategies to resolve the issue and return normal service.
Update
We are investigating reports that the earlier problem with Copilot code completions is reoccurring.
Investigating
We are investigating reports of degraded performance for Copilot
This incident affected: Copilot.
