Disruption with some GitHub services

Incident Report for GitHub

Resolved

On November 19, 2024, between 10:56:00 UTC and 12:03:00 UTC the notifications service was degraded and stopped sending notifications. On average, notifications delivery was delayed about 1 hour. This was due to a database host coming out of a regular maintenance process in read only-mode.
We mitigated the incident by making the host writable again. After that the notifications delivery recovered and any delivery job that had failed during the incident was successfully retried.
We are working to improve our observability across database clusters to reduce our time to detection and mitigation of issues like this one in the future.

Posted Nov 19, 2024 - 12:03 UTC

Update

We have resolved the issue but are waiting for queues to catch up.

Posted Nov 19, 2024 - 11:56 UTC

Update

We're investigating an issue with two-factor authentication in the GitHub mobile app

Posted Nov 19, 2024 - 11:38 UTC

Investigating

We are currently investigating this issue.

Posted Nov 19, 2024 - 11:36 UTC