On August 28, 2024, from 21:40 to 23:43 UTC, up to 25% of unauthenticated dotcom traffic in SE Asia (representing <1% of global traffic) encountered HTTP 500 errors. We observed elevated error rates at one of our global points of presence, where geo-DNS health checks were failing. We identified unhealthy cloud hardware in the region, indicated by abnormal CPU utilization patterns. As a result, we drained the site at 23:26 UTC, which promptly restored normal traffic operations.
Posted Aug 28, 2024 - 23:43 UTC
Update
Our mitigation has taken effect and impact is resolved.
Posted Aug 28, 2024 - 23:43 UTC
Update
Our mitigation is in place and we are seeing a reduction in overall impact. We are continuing to monitor until we are confident that the issue has been resolved.
Posted Aug 28, 2024 - 23:38 UTC
Update
We have identified a potential mitigation and are currently testing.
Posted Aug 28, 2024 - 23:33 UTC
Update
We are experiencing reduced overall impact, but continue to see a continued small impact to unauthenticated requests. We are continuing to investigate.
Posted Aug 28, 2024 - 23:14 UTC
Update
We are continuing to see customer impact and are still investigating.
Posted Aug 28, 2024 - 22:53 UTC
Update
We are continuing to investigate.
Posted Aug 28, 2024 - 22:37 UTC
Update
We are seeing cases of user impact in some locations are continuing to investigate.
Posted Aug 28, 2024 - 22:19 UTC
Update
We are seeing elevated traffic in some of our regions that we are in process of investigating.