On February 25th, 2025, between 14:25 UTC and 16:44 UTC email and web notifications experienced delivery delays. At the peak of the incident the delay resulted in ~10% of all notifications taking over 10 minutes to be delivered, with the remaining ~90% being delivered within 5-10 minutes. This was due to insufficient capacity in worker pools as a result of increased load during peak hours.
We also encountered delivery delays for a small number of webhooks, with delays of up-to 2.5 minutes to be delivered.
We mitigated the incident by scaling out the service to meet the demand.
The increase in capacity gives us extra headroom, and we are working to improve our capacity planning to prevent issues like this occurring in the future.
Posted Feb 25, 2025 - 16:50 UTC
Update
Web and email notifications are caught up, resolving the incident.
Posted Feb 25, 2025 - 16:49 UTC
Update
We're continuing to investigate delayed web and email notifications.
Posted Feb 25, 2025 - 16:16 UTC
Update
We're continuing to investigate delayed web and email notifications.
Posted Feb 25, 2025 - 15:43 UTC
Update
We're investigating delays in web and email notifications impacting all customers.