GitHub header
All Systems Operational

About This Site

For the status of GitHub Enterprise Cloud - EU, please visit: eu.githubstatus.com

Git Operations ? Operational
Webhooks ? Operational
Visit www.githubstatus.com for more information Operational
API Requests ? Operational
Issues ? Operational
Pull Requests ? Operational
Actions ? Operational
Packages ? Operational
Pages ? Operational
Codespaces ? Operational
Copilot Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Dec 6, 2024

No incidents reported today.

Dec 5, 2024

No incidents reported.

Dec 4, 2024
Resolved - This incident has been resolved.
Dec 4, 19:27 UTC
Update - Pull Requests is operating normally.
Dec 4, 19:26 UTC
Update - Pull Requests is experiencing degraded performance. We are continuing to investigate.
Dec 4, 19:21 UTC
Update - Issues is operating normally.
Dec 4, 19:20 UTC
Update - API Requests is operating normally.
Dec 4, 19:18 UTC
Update - Webhooks is operating normally.
Dec 4, 19:17 UTC
Update - We have identified the cause of timeouts impacting users across multiple services. This change was rolled back and we are seeing recovery. We will continue to monitor for complete recovery.
Dec 4, 19:11 UTC
Update - Issues is experiencing degraded performance. We are continuing to investigate.
Dec 4, 19:07 UTC
Update - API Requests is experiencing degraded performance. We are continuing to investigate.
Dec 4, 19:05 UTC
Update - Webhooks is experiencing degraded performance. We are continuing to investigate.
Dec 4, 19:05 UTC
Investigating - We are currently investigating this issue.
Dec 4, 18:58 UTC
Dec 3, 2024
Resolved - On December 3rd, between 23:29 and 23:43 UTC, Pull Requests experienced a brief outage and teams have confirmed the issue to be resolved. Due to brevity of incident it was not publicly statused at the time however an RCA will be conducted and shared in due course.
Dec 3, 23:30 UTC
Resolved - On December 3, 2024, between 19:35 UTC and 20:05 UTC API requests, Actions, Pull Requests and Issues were degraded. Web and API requests for Pull Requests experienced a 3.5% error rate and Issues had a 1.2% error rate. The highest impact was for users who experienced errors while creating and commenting on Pull Requests and Issues. Actions had a 3.3% error rate in jobs and delays on some updates during this time.

This was due to an erroneous database credential change impacting write access to Issues and Pull Requests data. We mitigated the incident by reverting the credential change at 19:52 UTC. We continued to monitor service recovery before resolving the incident at 20:05 UTC.

There are a few improvements we are making in response to this. We are investing in safe guards to the change management process in order to prevent erroneous database credential changes. Additionally, the initial rollback attempt was unsuccessful which led to a longer time to mitigate. We were able to revert through an alternative method and are updating our playbooks to document this mitigation strategy.

Dec 3, 20:05 UTC
Update - Pull Requests is operating normally.
Dec 3, 20:05 UTC
Update - Actions is operating normally.
Dec 3, 20:04 UTC
Update - API Requests is operating normally.
Dec 3, 20:02 UTC
Update - We have taken mitigating actions and are starting to see recovery but are continuing to monitor and ensure full recovery. Some users may still see errors.
Dec 3, 19:59 UTC
Update - Some users will experience problems with certain features of pull requests, actions, issues and other areas. We are aware of the issue, know the cause, and are working on a mitigation.
Dec 3, 19:54 UTC
Investigating - We are investigating reports of degraded performance for API Requests, Actions and Pull Requests
Dec 3, 19:48 UTC
Resolved - This incident has been resolved.
Dec 3, 04:39 UTC
Update - We saw a recurrence of the large hosted runner incident (https://www.githubstatus.com/incidents/qq1m7mqcl6zk) from 12/1/2024. We've applied the same mitigation and see improvements. We will continue to work on a long term solution.
Dec 3, 04:38 UTC
Update - We are investigating reports of degraded performance for Hosted Runners
Dec 3, 04:16 UTC
Investigating - We are currently investigating this issue.
Dec 3, 04:11 UTC
Dec 2, 2024
Resolved - Between Dec 1 12:20 UTC and Dec 2 1:05 UTC, availability of large hosted runners for Actions was degraded due to failures in background VM provisioning jobs. Users would see workflows queued waiting for a runner. On average, 8% of all workflows requiring large runners over the incident time were affected, peaking at 37.5% of requests. There were also lower levels of intermittent queuing on Dec 1 beginning around 3:00 UTC. Standard and Mac runners were not affected.

The job failures were caused by timeouts to a dependent service in the VM provisioning flow and gaps in the jobs’ resilience to those timeouts. The incident was mitigated by circumventing the dependency as it was not in the critical path of VM provisioning.

There are a few immediate improvements we are making in response to this. We are addressing the causes of the failed calls to improve the availability of calls to that backend service. Even with that impact, the critical flow of large VM provisioning should not have been impacted, so we are improving the client behavior to fail fast and circuit break non-critical calls. Finally the alerting for this service was not adequate in this particular scenario to ensure fast response by our team. We are improving our automated detection from this to reduce our time to detection and mitigation of issues like this one in the future.

Dec 2, 01:05 UTC
Update - We've applied a mitigation to fix the issues with large runner jobs processing. We are seeing improvements in telemetry and are monitoring for full recovery.
Dec 2, 00:57 UTC
Update - We continue to investigate large hosted runners not picking up jobs.
Dec 2, 00:14 UTC
Update - We continue to investigate issues with large runners.
Dec 1, 23:43 UTC
Update - We're seeing issues related to large runners not picking up jobs and are investigating.
Dec 1, 23:24 UTC
Investigating - We are currently investigating this issue.
Dec 1, 23:18 UTC
Dec 1, 2024
Nov 30, 2024

No incidents reported.

Nov 29, 2024

No incidents reported.

Nov 28, 2024
Resolved - This incident has been resolved.
Nov 28, 07:01 UTC
Update - We identified the issue and applied a mitigation, resulting in the cessation of timeouts. While we are considering this incident resolved for now, we are continuing to investigate the root cause and plan to implement a permanent fix. Updates will follow as we progress.
Nov 28, 07:00 UTC
Update - We are investigating issues with timeouts in some requests in Codespaces. We will update you on mitigation progress.
Nov 28, 06:36 UTC
Investigating - We are investigating reports of degraded performance for Codespaces
Nov 28, 06:27 UTC
Resolved - This incident has been resolved.
Nov 28, 05:11 UTC
Update - We identified and addressed failures in two proxy servers and applied mitigation. Since then, timeouts have ceased, and we are considering the incident resolved. We will continue to monitor the situation closely and provide updates if any changes occur.
Nov 28, 05:10 UTC
Update - We are investigating some network proxy issues that may be contributing to the timeouts in a small percentage of requests in Codespaces. We will continue to investigate.
Nov 28, 04:34 UTC
Update - We are continuing to investigate issues with timeouts in a small percentage of requests in Codespaces. We will update you on mitigation progress.
Nov 28, 04:03 UTC
Update - We are investigating issues with timeouts in some requests in Codespaces. Some users may not be able to connect to their Codespaces at this time. We will update you on mitigation progress.
Nov 28, 03:32 UTC
Investigating - We are investigating reports of degraded performance for Codespaces
Nov 28, 03:29 UTC
Nov 27, 2024

No incidents reported.

Nov 26, 2024

No incidents reported.

Nov 25, 2024
Resolved - Between 13:30 and 15:00 UTC, repository searches were timing out for most users. The ongoing efforts from the similar incident last week helped uncover the main contributing factors. We have deployed short-term mitigations and identified longer term work to proactively identify and limit resource-intensive searches.
Nov 25, 15:25 UTC
Update - Search is now operating normally. We are declaring this issue resolved.
Nov 25, 15:24 UTC
Update - We are now observing signs of complete recovery in search. We will continue to monitor and assess.
Nov 25, 15:10 UTC
Update - We're observing signs of recovery in search, we will continue to monitor.

Next update within 15 minutes.

Nov 25, 14:40 UTC
Update - We are seeing failures in repos, users, discussions, and wikis search. Customers may see failing searches and searching by topic may fail to load.

Code and issues search continue to be available.

The team is investigating, next update in 30m.

Nov 25, 14:10 UTC
Update - We are seeing failures in search. Customers may see failing searches and searching by topic may fail to load.
Nov 25, 13:58 UTC
Investigating - We are currently investigating this issue.
Nov 25, 13:57 UTC
Resolved - On November 25th, 2024 between 10:38 UTC and 12:00 UTC the Claude model for GitHub Copilot Chat experienced degraded performance. During the impact, all requests to Claude would result in an immediate error to the user. This was due to upstream errors with one of our infrastructure providers, which have since been mitigated.

We are working with our infrastructure providers to reduce time to detection and implement additional failover options to mitigate issues like this one in the future.

Nov 25, 12:17 UTC
Update - This incident has been mitigated; we are now seeing requests succeeding to the Claude 3.5 Sonnet model in Copilot.
Nov 25, 12:17 UTC
Update - The team is continuing to investigate errors using the Claude 3.5 Sonnet v2 model and has engaged our partners.

All requests to this model are failing, but other Copilot models are functional and can be used as an alternative.

Nov 25, 11:44 UTC
Update - Users cannot use the Claude 3.5 Sonnet model in GitHub Copilot currently, in both VS Code and GitHub.com chat. The team is investigating.
Nov 25, 11:00 UTC
Investigating - We are currently investigating this issue.
Nov 25, 10:51 UTC
Nov 24, 2024

No incidents reported.

Nov 23, 2024

No incidents reported.

Nov 22, 2024

No incidents reported.