GitHub header

Incident with Pull Requests, Issues, Webhooks and Actions

Incident Report for GitHub

Resolved

This incident has been resolved.
On November 27, 2023 at 18:46 UTC, we attempted to rotate our OpenID Connect (OIDC) authentication flow certificates. Due to an error in the certificate formatting, we uploaded an invalid certificate configuration that was not observed in our pre-production testing. Our background job servers were unable to start because a valid configuration is required at worker start up. As a result, users experienced delays in Pull Requests, Webhooks, Issues, Actions and Projects. Rollback of the change was slowed by the invalid certificate as our deployment system relied on the same certificate. Rollback was completed at 20:35 UTC. Most services recovered by 20:44 UTC.



Delayed updates to Issues and Pull Requests were applied normally once the changes were rolled back. After the change was rolled back, a large queue of Actions-related jobs built up which included Pull Request, Pull Request review and Pull Request review comment events. About 2.3% of Actions jobs failed during the duration of the incident. Job queue times returned to normal once all remaining jobs were processed.



We are working to improve our certificate testing and rotation process to reduce the risk of customer-impacting errors.
Posted Nov 27, 2023 - 21:11 UTC

Update

Webhooks is operating normally.
Posted Nov 27, 2023 - 20:44 UTC

Update

Issues is operating normally.
Posted Nov 27, 2023 - 20:44 UTC

Update

Pull Requests is operating normally.
Posted Nov 27, 2023 - 20:43 UTC

Update

Actions customers are experiencing workflow start delays as part of the ongoing PRs incident. We are seeing previously delayed runs kick off and will continue to monitor.
Posted Nov 27, 2023 - 20:39 UTC

Update

Actions is experiencing degraded performance. We are continuing to investigate.
Posted Nov 27, 2023 - 20:30 UTC

Update

Customers are also experiencing delays in webhook delivery and issue updates. We are seeing recovery and are continuing to monitor.
Posted Nov 27, 2023 - 20:22 UTC

Update

Webhooks is experiencing degraded performance. We are continuing to investigate.
Posted Nov 27, 2023 - 20:16 UTC

Update

Issues is experiencing degraded performance. We are continuing to investigate.
Posted Nov 27, 2023 - 20:16 UTC

Update

Customers are seeing delays in pushed commits appearing on pull requests. We are currently investigating.
Posted Nov 27, 2023 - 19:46 UTC

Investigating

We are investigating reports of degraded performance for Pull Requests
Posted Nov 27, 2023 - 19:43 UTC
This incident affected: Webhooks, Issues, Pull Requests, and Actions.