GitHub header

Incident with Pages, Actions, Codespaces, API Requests, Issues and Pull Requests

Incident Report for GitHub

Resolved

On April 5, 2024, between 8:11 and 8:58 UTC a number of GitHub services were degraded, returning error responses. Web request error rate peaked at 6%, API request error rate peaked at 10%. Actions had 103,660 workflow runs fail to start.

A database load balancer change caused connection failures in one of our three data centers to various critical database clusters. The incident was mitigated once that change was rolled back.

We have updated our deployment pipeline to better detect this problem in earlier stages of rollout to reduce impact to end users.
Posted Apr 05, 2024 - 09:18 UTC

Update

Pull Requests is operating normally.
Posted Apr 05, 2024 - 09:17 UTC

Update

Issues is operating normally.
Posted Apr 05, 2024 - 09:17 UTC

Update

API Requests is operating normally.
Posted Apr 05, 2024 - 09:17 UTC

Update

Codespaces is operating normally.
Posted Apr 05, 2024 - 09:17 UTC

Update

Actions is operating normally.
Posted Apr 05, 2024 - 09:17 UTC

Update

Pages is operating normally.
Posted Apr 05, 2024 - 09:17 UTC

Update

Actions is experiencing degraded performance. We are continuing to investigate.
Posted Apr 05, 2024 - 09:17 UTC

Update

We've reverted a change we believe caused this, are seeing initial indications of reduced errors, and are monitoring for full recovery
Posted Apr 05, 2024 - 09:00 UTC

Update

Pages is experiencing degraded performance. We are continuing to investigate.
Posted Apr 05, 2024 - 08:59 UTC

Update

We're seeing connection failures to some databases in two of three sites and are investigating.
Posted Apr 05, 2024 - 08:51 UTC

Update

Pull Requests is experiencing degraded performance. We are continuing to investigate.
Posted Apr 05, 2024 - 08:50 UTC

Update

Issues is experiencing degraded performance. We are continuing to investigate.
Posted Apr 05, 2024 - 08:49 UTC

Update

API Requests is experiencing degraded performance. We are continuing to investigate.
Posted Apr 05, 2024 - 08:49 UTC

Update

Codespaces is experiencing degraded performance. We are continuing to investigate.
Posted Apr 05, 2024 - 08:49 UTC

Investigating

We are investigating reports of degraded availability for Actions
Posted Apr 05, 2024 - 08:33 UTC
This incident affected: API Requests, Issues, Pull Requests, Actions, Pages, and Codespaces.