GitHub header

Incident with Webhooks

Incident Report for GitHub

Resolved

On August 12, 2024 from 13:39 to 14:28 UTC some users experienced an elevated rate of errors of up to 0.45% from the GitHub API. Less than 5% of webhooks interactions failed and less than 0.5% of Actions runs were delayed.

This impact was caused by internal networking instances being insufficiently scaled.

We mitigated the incident by provisioning additional instances. We are working to enhance the sizing strategy for the relevant infrastructure to prevent similar issues and to also improve monitoring and processes to reduce our time to detection and mitigation of issues like this one in the future.
Posted Aug 12, 2024 - 14:41 UTC

Update

Webhooks is experiencing degraded performance. We are continuing to investigate.
Posted Aug 12, 2024 - 14:41 UTC

Update

Webhooks is operating normally.
Posted Aug 12, 2024 - 14:41 UTC

Update

Some customers may see failures calling the webhooks API or running queries for enterprise or organization audit logs. We have started mitigating the issue and are watching recovery. We will provide an update within the next 30 minutes.
Posted Aug 12, 2024 - 14:32 UTC

Investigating

We are investigating reports of degraded performance for Webhooks
Posted Aug 12, 2024 - 14:09 UTC
This incident affected: Webhooks.