GitHub header
Issues with VNet Injected Larger Hosted Runners in East US 2
Incident Report for GitHub
Resolved
On January 9, 2025, larger hosted runners configured with Azure private networking in East US 2 were degraded, causing delayed job starts for ~2,300 jobs between 16:00 and 20:00 UTC. There was also an earlier period of impact from 2025-01-08 22:00 UTC to 2025-01-09 4:10 UTC with 488 jobs impacted. The cause of both these delays was an incident in East US 2 impacting provisioning and network connectivity of Azure resources. More details on that incident are visible at https://azure.status.microsoft/en-us/status/history (Tracking ID: PLP3-1W8). Because these runners are reliant on private networking with networks in the East US 2 region, there were no immediate mitigations available other than restoring network connectivity. Going forward, we will continue evaluating options to provide better resilience to 3rd party regional outages that affect private networking customers.
Posted Jan 09, 2025 - 20:00 UTC
Update
The impact to Large Runners has been mitigated. The third party incident has not been fully mitigated but is being actively monitored at https://azure.status.microsoft/en-us/status in case of reoccurrence.
Posted Jan 09, 2025 - 20:00 UTC
Update
We are continuing to see improvements while still monitoring updates from the third party at https://azure.status.microsoft/en-us/status
Posted Jan 09, 2025 - 19:27 UTC
Update
We are still monitoring the third party networking updates via https://azure.status.microsoft/en-us/status. Multiple workstreams are in progress by the third party to mitigate the impact.
Posted Jan 09, 2025 - 18:53 UTC
Update
We are still monitoring the third party networking updates via https://azure.status.microsoft/en-us/status. Multiple workstreams are in progress by the third party to mitigate the impact.
Posted Jan 09, 2025 - 18:18 UTC
Update
The underlying third party networking issues have been identified and are being work on. Ongoing updates can be found at https://azure.status.microsoft/en-us/status
Posted Jan 09, 2025 - 17:43 UTC
Investigating
We are currently investigating this issue.
Posted Jan 09, 2025 - 17:12 UTC