Incident with Webhooks, Actions, Pull Requests and Issues

Incident Report for GitHub

Resolved

On February 26, 2024, between 18:34 UTC and 19:37 UTC our background job service was degraded and caused job start delays up to 15 minutes. Users experienced delays in Webhooks, Actions, and some UI updates (e.g. a delay in UI updates on pull requests). This was due to capacity problems with our job queueing service, and a failure of our automated failover system.

We mitigated the incident by manually failing over to our secondary cluster. No data was lost - recovery began at 18:55 UTC, when the backlog of enqueued jobs began to process.

We are actively working to repair our failover automation and expand the capacity of our background job queuing service to prevent issues like this in the future.

Posted Feb 26, 2024 - 19:37 UTC

Update

Actions and Pull Requests are operating normally.

Posted Feb 26, 2024 - 19:37 UTC

Update

Webhooks and Issues are operating normally.

Posted Feb 26, 2024 - 19:37 UTC

Update

Issues is experiencing degraded performance. We are continuing to investigate.

Posted Feb 26, 2024 - 19:05 UTC

Update

Pull Requests is experiencing degraded performance. We are continuing to investigate.

Posted Feb 26, 2024 - 18:57 UTC

Update

We have deployed a fix for issues affecting Webhooks, Actions, and some other services. We are beginning to see recovery and will continue to monitor and fix as needed.

Posted Feb 26, 2024 - 18:55 UTC

Update

Webhooks is experiencing degraded performance. We are continuing to investigate.

Posted Feb 26, 2024 - 18:55 UTC

Update

Actions is experiencing degraded performance. We are continuing to investigate.

Posted Feb 26, 2024 - 18:48 UTC

Investigating

We are investigating reports of degraded performance for Webhooks

Posted Feb 26, 2024 - 18:47 UTC

This incident affected: Webhooks, Issues, Pull Requests, and Actions.