Incident with Codespaces and API Requests
Incident
Report for GitHub
Resolved
On March 15, 2024, between 19:42 UTC and 20:24 UTC several services were degraded due to a regression in calling the permissions system.
New GitHub Codespaces could not be created, as were Codespaces sessions that required minting a new auth token.
Actions saw delays and infrastructure failures due to the upstream dependency on fetching tokens for the repository for runs to successfully execute.
GitHub Pages were affected due to the impact on Actions, resulting in 1266 page builds failing, which at the low point represented 33% of page builds failing. This resulted in page edits not being reflected on those impacted sites.
We deployed an application update that included a newer version of our database query builder. The new version uses a newer MySQL syntax for upsert queries that is not supported by the database proxy service we use for some of our production-environment database clusters. This incompatibility impacted the permissions cluster specifically, causing requests that attempted such queries to fail.
We responded by rolling back the deployment, restoring the previous query use, and thus mitigated the incident.
We have identified and corrected a misconfiguration of the permissions cluster in our development and CI environments that will ensure queries utilize the proxy service to prevent future syntax additions causing issues in production.
New GitHub Codespaces could not be created, as were Codespaces sessions that required minting a new auth token.
Actions saw delays and infrastructure failures due to the upstream dependency on fetching tokens for the repository for runs to successfully execute.
GitHub Pages were affected due to the impact on Actions, resulting in 1266 page builds failing, which at the low point represented 33% of page builds failing. This resulted in page edits not being reflected on those impacted sites.
We deployed an application update that included a newer version of our database query builder. The new version uses a newer MySQL syntax for upsert queries that is not supported by the database proxy service we use for some of our production-environment database clusters. This incompatibility impacted the permissions cluster specifically, causing requests that attempted such queries to fail.
We responded by rolling back the deployment, restoring the previous query use, and thus mitigated the incident.
We have identified and corrected a misconfiguration of the permissions cluster in our development and CI environments that will ensure queries utilize the proxy service to prevent future syntax additions causing issues in production.
Update
Codespaces is operating normally.
Update
API Requests is operating normally.
Update
We rolled back the most recent deployment and are seeing improvements across all services, and will continue to monitor for additional impact.
Update
API Requests is experiencing degraded performance. We are continuing to investigate.
Update
Codespaces is experiencing degraded availability. We are continuing to investigate.
Update
API Requests is experiencing degraded availability. We are continuing to investigate.
Update
API Requests is experiencing degraded performance. We are continuing to investigate.
Investigating
We are investigating reports of degraded performance for Codespaces
This incident affected: API Requests and Codespaces.
