GitHub header

Incident with Codespaces

Incident Report for GitHub

Investigating

On 2024-05-28 between 18:30 and 21:30 UTC, the Codespaces service was degraded. Users with dev container configurations that depended on the DockerHub container registry were unable to create new codespaces unless they provided their own DockerHub registry credentials. We observed a rise of at most 6% of creations entering recovery mode due to this incident.

The GitHub Codespaces team worked directly with Docker to mitigate and has engaged with Docker to prevent similar issues in the future.
Posted May 28, 2024 - 21:24 UTC

Resolved

This incident has been resolved.
Posted May 28, 2024 - 21:24 UTC

Update

A fix has been applied and we are seeing some recovery. We will continue to monitor for a bit before marking this issue resolved.
Posted May 28, 2024 - 21:19 UTC

Update

We are still investigating root cause and remediation options. In the meantime, here is a workaround to be able to pull images from DockerHub:

1. Make a free DockerHub account at https://hub.docker.com (or use an existing account if you have one).
2. Create a DockerHub secret/PAT from https://hub.docker.com/settings/security (Read permission should be sufficient).
3. Go to https://github.com/settings/codespaces

Add three Codespace secrets:

- DOCKERHUB_CONTAINER_REGISTRY_PASSWORD (equal to the DockerHub PAT you created)
- DOCKERHUB_CONTAINER_REGISTRY_SERVER (equal to https://index.docker.io/v1/)
- DOCKERHUB_CONTAINER_REGISTRY_USER (equal to your DockerHub username)

4. Make sure these secrets are set as visible to the target repo.
5. Create/rebuild your Codespace

Steps above are distilled from the official docs: https://docs.github.com/en/codespaces/reference/allowing-your-codespace-to-access-a-private-registry#example-secrets
Posted May 28, 2024 - 20:53 UTC

Update

Duplicate update, same as above
Posted May 28, 2024 - 20:53 UTC

Update

Some Codespaces are currently failing to be properly created for images hosted by DockerHub. Other registries should be unaffected. We are investigating root cause and will report back shortly.
Posted May 28, 2024 - 20:23 UTC

Investigating

We are investigating reports of degraded performance for Codespaces
Posted May 28, 2024 - 20:17 UTC
This incident affected: Codespaces.