On July 02, 2024, between 18:21 UTC and 19:24 UTC the code search service was degraded and returned elevated 500 HTTP status responses. On average, the error rate was 38% of code search requests. This was due to a bad deployment causing some user's rate limit calculations to error while processing code search requests. This impacted approximately 2,000 users. We mitigated the incident by rolling back the bad deployment along with resetting rate limits for all users. We have identified and implemented updates in the testing of rate limit calculations to prevent this problem from happening again, and clarified deployment processes for verification before a full production rollout to minimize impact in the future.
Posted Jul 02, 2024 - 19:24 UTC
Update
API Requests is operating normally.
Posted Jul 02, 2024 - 19:24 UTC
Update
The fix has been rolled out and our telemetry indicates that the errors with code search have resolved.
Posted Jul 02, 2024 - 19:22 UTC
Update
An issue with faulty data in an in-memory storage causes around one third of code search requests to fail. The team has identified the issue and is working on rolling out a fix.
Posted Jul 02, 2024 - 19:18 UTC
Update
API Requests is experiencing degraded performance. We are continuing to investigate.