On March 7, 2025, from 09:30 UTC to 11:07 UTC, we experienced a networking event that disrupted connectivity to our search infrastructure, impacting about 25% of search queries and indexing attempts. Searches for PRs, Issues, Actions workflow runs, Packages, Releases, and other products were impacted, resulting in failed requests or stale data. The connectivity issue self-resolved after 90 minutes. The backlog of indexing jobs was fully processed and saw recovery soon after, and queries to all indexes also saw an immediate return to normal throughput.
We are working with our cloud provider to identify the root cause and are researching additional layers of redundancy to reduce customer impact in the future for issues like this one. We are also exploring mitigation strategies for faster resolution.
Posted Mar 07, 2025 - 11:24 UTC
Update
We continue investigating degraded experience with searching for issues, pull, requests and actions workflow runs.
Posted Mar 07, 2025 - 10:54 UTC
Update
Actions is experiencing degraded performance. We are continuing to investigate.
Posted Mar 07, 2025 - 10:27 UTC
Update
Searches for issues and pull-requests may be slower than normal and timeout for some users
Posted Mar 07, 2025 - 10:12 UTC
Update
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Posted Mar 07, 2025 - 10:06 UTC
Update
Issues is experiencing degraded performance. We are continuing to investigate.
Posted Mar 07, 2025 - 10:05 UTC
Investigating
We are currently investigating this issue.
Posted Mar 07, 2025 - 10:03 UTC
This incident affected: Issues, Pull Requests, and Actions.