Incident with Codespaces
Incident
Report for GitHub
Resolved
From 10:59 UTC to 13:48 UTC, GitHub Codespaces service was down or degraded due to an outage in our authentication service.This issue impacted 67% of users over this time period.
Our service auth layer experienced throttling with our third party dependencies due to higher load. This impacted all user facing scenarios for Codespaces service. Our automated regional failover kicked in, but it failed to mitigate as this issue impacted the service globally. We mitigated manually by reducing load on external dependency.
With the incident mitigated, we are working to assess and implement scaling improvements to make our service more resilient with increasing load.
Our service auth layer experienced throttling with our third party dependencies due to higher load. This impacted all user facing scenarios for Codespaces service. Our automated regional failover kicked in, but it failed to mitigate as this issue impacted the service globally. We mitigated manually by reducing load on external dependency.
With the incident mitigated, we are working to assess and implement scaling improvements to make our service more resilient with increasing load.
Update
Codespaces is experiencing degraded performance. We are continuing to investigate.
Update
We are continuing with efforts to mitigate Codespaces issues and are beginning to see some Codespace creations succeed.
Update
We have identified an issue impacting most Codespaces operations and are working on a mitigation.
Update
Codespaces is experiencing degraded availability. We are continuing to investigate.
Investigating
We are investigating reports of degraded performance for Codespaces
This incident affected: Codespaces.
