On February 9, 2024 between 10:34 UTC and 11:24 UTC, the Webhooks service was degraded and 63% of webhooks were delayed by up to 16 minutes with an average delay of 5 minutes. No webhook deliveries were lost. This was due to an issue with an overloaded backend data store that was unable to process network requests fast enough.
We mitigated the incident by manually failing over traffic to healthy hosts.
We are expanding the capacity of the backing store as well as making the Webhooks service more resilient to this kind of issue.
Posted Feb 09, 2024 - 11:28 UTC
Update
Webhooks is operating normally.
Posted Feb 09, 2024 - 11:25 UTC
Update
We are investigating latency in processing webhooks. Customers may see a delay of around 5 minutes at this time. We will continue to keep users updated on progress towards mitigation.
Posted Feb 09, 2024 - 11:11 UTC
Investigating
We are investigating reports of degraded performance for Webhooks
