Incident with Git Operations, API Requests and Issues

Incident Report for GitHub

Resolved

On April 28th, 2025, between 4AM and 11AM UTC, ~0.5% of customers experienced HTTP 500 or 429 responses for raw file access (via the GitHub website and APIs). Additionally, ~0.5% of customers may have seen slow pull request page loads and increased timeouts in the GraphQL API.

The incident was caused by queueing in serving systems due to a change in traffic patterns, specifically scraping activity targeting our API. We have adjusted limits and added flow control to systems in response to the changing traffic patterns to improve our ability to prevent future large queueing issues. We’ve additionally updated rate limiting unauthenticated requests to reduce overall load, more details are here:
https://github.blog/changelog/2025-05-08-updated-rate-limits-for-unauthenticated-requests/

Posted Apr 28, 2025 - 11:09 UTC

Update

We are seeing signs of recovery and continue to monitor latency.

Posted Apr 28, 2025 - 10:35 UTC

Update

We continue to investigate impact to Issues and Pull Requests. Customers may see some timeouts as we work towards mitigation.

Posted Apr 28, 2025 - 09:52 UTC

Update

We are continuing to investigate impact to Issues and Pull Requests. We will provide more updates as we have them.

Posted Apr 28, 2025 - 08:58 UTC

Update

Users may see timeouts when viewing Pull Requests. We are still investigating the issues related to Issues and Pull Requests and will provide further updates as soon as we can

Posted Apr 28, 2025 - 08:23 UTC

Update

Pull Requests is experiencing degraded performance. We are continuing to investigate.

Posted Apr 28, 2025 - 08:21 UTC

Update

Issues API is currently seeing elevated latency. We are investigating the issue and will provide further updates as soon as we have them.

Posted Apr 28, 2025 - 08:05 UTC

Investigating

We are investigating reports of degraded performance for API Requests, Git Operations and Issues

Posted Apr 28, 2025 - 08:03 UTC

This incident affected: Git Operations, API Requests, Issues, and Pull Requests.