Incident with Webhooks, Actions, Pull Requests and Issues
Incident
Report for GitHub
Resolved
On February 26, 2024, between 18:34 UTC and 19:37 UTC our background job service was degraded and caused job start delays up to 15 minutes. Users experienced delays in Webhooks, Actions, and some UI updates (e.g. a delay in UI updates on pull requests). This was due to capacity problems with our job queueing service, and a failure of our automated failover system.
We mitigated the incident by manually failing over to our secondary cluster. No data was lost - recovery began at 18:55 UTC, when the backlog of enqueued jobs began to process.
We are actively working to repair our failover automation and expand the capacity of our background job queuing service to prevent issues like this in the future.
We mitigated the incident by manually failing over to our secondary cluster. No data was lost - recovery began at 18:55 UTC, when the backlog of enqueued jobs began to process.
We are actively working to repair our failover automation and expand the capacity of our background job queuing service to prevent issues like this in the future.
Update
Actions and Pull Requests are operating normally.
Update
Webhooks and Issues are operating normally.
Update
Issues is experiencing degraded performance. We are continuing to investigate.
Update
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Update
We have deployed a fix for issues affecting Webhooks, Actions, and some other services. We are beginning to see recovery and will continue to monitor and fix as needed.
Update
Webhooks is experiencing degraded performance. We are continuing to investigate.
Update
Actions is experiencing degraded performance. We are continuing to investigate.
Investigating
We are investigating reports of degraded performance for Webhooks
This incident affected: Webhooks, Issues, Pull Requests, and Actions.
