Resolved -
Between 2026-02-23 19:10 and 2026-02-24 00:46 UTC, all lexical code search queries in GitHub.com and the code search API were significantly slowed, and during this incident, between 5 and 10% of search queries timed out. This was caused by a single customer who had created a network of hundreds of orchestrated accounts which searched with a uniquely expensive search query. This search query concentrated load on a single hot shard within the search index, slowing down all queries. After we identified the source of the load and stopped the traffic, latency returned to normal.
To avoid this situation occurring again in the future, we are making a number of improvements to our systems, including: improved rate limiting that accounts for highly skewed load on hot shards, improved system resilience for when a small number of shards time out, improved tooling to recognize abusive actors, and capabilities that will allow us to shed load on a single shard in emergencies.
Feb 24, 00:46 UTC
Update -
We have identified a cause for the latency and timeouts and have implemented a fix. We are observing initial recovery now.
Feb 24, 00:38 UTC
Update -
Customers using code search continue to see increased latency and timeout errors. We are working to mitigate issues on the affected shard.
Feb 23, 23:10 UTC
Update -
Elevated latency and timeouts for code search is isolated to a single shard experiencing elevated CPU. We are taking steps to isolate and mitigate the affected shard.
Feb 23, 22:22 UTC
Update -
Elevated latency and timeouts for code search is isolated to a single shard experiencing elevated CPU. We are continuing to investigate the cause and steps to mitigate.
Feb 23, 21:18 UTC
Update -
We are continuing to investigate elevated latency and timeouts for code search.
Feb 23, 20:33 UTC
Investigating -
We are investigating reports of impacted performance for some GitHub services.
Feb 23, 19:59 UTC