Incident with Actions, API Requests, Codespaces, Git Operations, Issues, Pages, Pull Requests and Webhooks

Incident Report for GitHub

Resolved

This incident has been resolved.

Posted May 11, 2023 - 19:00 UTC

Update

Reindexing is complete, and the /pulls and /search pages are up to date.

Posted May 11, 2023 - 18:54 UTC

Update

We’ve reindexed a little over half of the pull requests missing from the /pulls and /search pages.

Posted May 11, 2023 - 18:19 UTC

Update

We have reindexed about 20% of the pull requests missing from the /pulls and /search pages.

Posted May 11, 2023 - 17:53 UTC

Update

We’ve identified the full set of impacted pull requests not showing up on the /pulls and /search pages. We’ve kicked off a job to reindex them.

Posted May 11, 2023 - 17:21 UTC

Update

We’ve completed re-running the push jobs. We continue to investigate some pull requests not appearing in the /pulls and /search pages.

Posted May 11, 2023 - 16:55 UTC

Update

We’re continuing to investigate some pull requests not appearing in the /pulls and /search pages. Also, we continue to re-run the missed push jobs and webhooks.

Posted May 11, 2023 - 16:23 UTC

Update

We’re continuing to investigate some pull requests not appearing in the /pushes and /search pages. Also, we continue to re-run the missed webhooks.

Posted May 11, 2023 - 15:55 UTC

Update

We’re continuing to investigate a small number of pull requests not appearing on the /pulls and /search pages.

Posted May 11, 2023 - 15:20 UTC

Update

Pages is operating normally.

Posted May 11, 2023 - 14:46 UTC

Update

Actions, API Requests, Codespaces, Git Operations, Issues and Webhooks are operating normally.

Posted May 11, 2023 - 14:40 UTC

Update

Git operations are restored and operating normally. We are working on backfilling delayed pull request push updates from 13:30 to 13:55 UTC. Some webhooks sent during that window were not delivered.

Posted May 11, 2023 - 14:38 UTC

Update

We see recovery across most services and requests. We continue to recover from backlogs and are observing for additional impact. We’ll post another update within 30 minutes.

Posted May 11, 2023 - 14:19 UTC

Update

We have restored git operations and are seeing error rates return to normal. We are continuing to monitor for impact.

Posted May 11, 2023 - 14:02 UTC

Update

We are investigating issues with git operations leading to degraded experiences for many services. We will post an update within 30 minutes.

Posted May 11, 2023 - 13:46 UTC

Update

API Requests is experiencing degraded performance. We are still investigating and will provide an update when we have one.

Posted May 11, 2023 - 13:41 UTC

Update

Codespaces is experiencing degraded performance. We are continuing to investigate.

Posted May 11, 2023 - 13:40 UTC

Update

Actions, Git Operations, Pages and Webhooks are experiencing degraded performance. We are continuing to investigate.

Posted May 11, 2023 - 13:35 UTC

Investigating

We are investigating reports of degraded performance for Issues and Pull Requests.

Posted May 11, 2023 - 13:33 UTC

This incident affected: Git Operations, Webhooks, API Requests, Issues, Pull Requests, Actions, Pages, and Codespaces.