Incident with Actions and Webhooks
Incident
Report for GitHub
Resolved
This incident has been resolved. An interaction between two feature flag rollouts caused us to suppress delivery of push webhooks between 16:23 and 17:11 UTC in a manner that evaded our existing observability. This affected 71,000 repositories whose users will have noted missing webhook deliveries and/or experienced Actions jobs with push triggers failing to start. During this period, around 25% of new Actions jobs were impacted. The issues were resolved by disabling one of the feature flags in question.
After weighing various options for retroactively dispatching old push webhooks, we concluded that the risk of delivering stale data to customers with these redeliveries outweighed the possible benefit.
As follow up to this incident, we are working to improve monitoring of webhook throughput and to document a policy around webhook redelivery timelines.
After weighing various options for retroactively dispatching old push webhooks, we concluded that the risk of delivering stale data to customers with these redeliveries outweighed the possible benefit.
As follow up to this incident, we are working to improve monitoring of webhook throughput and to document a policy around webhook redelivery timelines.
Update
Actions is operating normally.
Update
Customers saw pull requests push event deliveries for Actions and Webhooks fail between 16:23 and 17:12 (UTC). We fixed the issue, and we are working to re-process push events for the affected time period.
Investigating
We are investigating reports of degraded performance for Actions and Webhooks
This incident affected: Webhooks and Actions.
