On February 27, 2025, between 11:30 UTC and 12:22 UTC, Actions experienced degraded performance, leading to delays in workflow runs. On average, 5% of Actions workflow runs were delayed by 31 minutes. The delays were caused by updates in a dependent service that led to failures in Redis connectivity in one region. We mitigated the incident by failing over the impacted service and re-routing the service’s traffic out of that region. We are working to improve monitoring and processes of failover to reduce our time to detection and mitigation of issues like this one in the future.
Posted Feb 27, 2025 - 12:22 UTC
Update
The team is confident that recovery is complete. Thank you for your patience as this issue was investigated.
Posted Feb 27, 2025 - 12:22 UTC
Update
Our mitigations have rolled out successfully and have seen recovery for all Actions run starts back within expected range. Users should see Actions runs working normally.
We will keep this incident open for a short time while we continue to validate these results.
Posted Feb 27, 2025 - 12:16 UTC
Update
We have identified the cause of the delays to starting Action runs.
Our team is working to roll out mitigations and we hope to see recovery as these take effect in our systems over the next 10-20 minutes.
Further updates as we have more information.
Posted Feb 27, 2025 - 12:01 UTC
Update
We are seeing an increase in run start delays since 1104 UTC. This is impacting ~3% of Action runs at this time.
The team is working to understand the causes of this and to mitigate impact. We will continue to update as we have more information.
Posted Feb 27, 2025 - 11:39 UTC
Update
Actions is experiencing degraded performance. We are continuing to investigate.