Get email notifications whenever GitHub creates, updates or resolves an incident.
Get text message notifications whenever GitHub creates or resolves an incident.
Get incident updates and maintenance status messages in Slack.
Subscribe via Slack
By subscribing you acknowledge our Privacy Policy. In addition, you agree to the Atlassian Cloud Terms of Service and acknowledge Atlassian's Privacy Policy.
Get webhook notifications whenever GitHub creates an incident, updates an incident, resolves an incident or changes a component status.
Visit our support site.
All Systems Operational
About This Site
For the status of GitHub Enterprise Cloud - EU, please visit: eu.githubstatus.com
Git Operations
?
Operational
Webhooks
?
Operational
Visit www.githubstatus.com for more information
Operational
API Requests
?
Operational
Issues
?
Operational
Pull Requests
?
Operational
Actions
?
Operational
Packages
?
Operational
Pages
?
Operational
Codespaces
?
Operational
Copilot
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Dec 11, 2024
No incidents reported today.
Dec 10, 2024
No incidents reported.
Dec 9, 2024
No incidents reported.
Dec 8, 2024
No incidents reported.
Dec 7, 2024
No incidents reported.
Dec 6, 2024
Resolved -
Upon further investigation, the degradation in migrations in the EU was caused by an internal configuration issue, which was promptly identified and resolved. No customer migrations were impacted during this time and the issue only affected GitHub Enterprise Cloud - EU and had no impact on Github.com. The service is now fully operational. We are following up by improving our processes for these internal configuration changes to prevent a recurrence, and to have incidents that affect GitHub Enterprise Cloud - EU be reported on https://eu.githubstatus.com/.
Dec 6, 17:17 UTC
Dec 6, 17:17 UTC
Update -
Migrations are failing for a subset of users in the EU region with data residency. We believe we have resolved the issue and are monitoring for resolution.
Dec 6, 17:17 UTC
Dec 6, 17:17 UTC
Investigating -
We are currently investigating this issue.
Dec 6, 16:58 UTC
Dec 6, 16:58 UTC
Dec 5, 2024
No incidents reported.
Dec 4, 2024
Resolved -
This incident has been resolved.
Dec 4, 19:27 UTC
Dec 4, 19:27 UTC
Update -
Pull Requests is operating normally.
Dec 4, 19:26 UTC
Dec 4, 19:26 UTC
Update -
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Dec 4, 19:21 UTC
Dec 4, 19:21 UTC
Update -
Issues is operating normally.
Dec 4, 19:20 UTC
Dec 4, 19:20 UTC
Update -
API Requests is operating normally.
Dec 4, 19:18 UTC
Dec 4, 19:18 UTC
Update -
Webhooks is operating normally.
Dec 4, 19:17 UTC
Dec 4, 19:17 UTC
Update -
We have identified the cause of timeouts impacting users across multiple services. This change was rolled back and we are seeing recovery. We will continue to monitor for complete recovery.
Dec 4, 19:11 UTC
Dec 4, 19:11 UTC
Update -
Issues is experiencing degraded performance. We are continuing to investigate.
Dec 4, 19:07 UTC
Dec 4, 19:07 UTC
Update -
API Requests is experiencing degraded performance. We are continuing to investigate.
Dec 4, 19:05 UTC
Dec 4, 19:05 UTC
Update -
Webhooks is experiencing degraded performance. We are continuing to investigate.
Dec 4, 19:05 UTC
Dec 4, 19:05 UTC
Investigating -
We are currently investigating this issue.
Dec 4, 18:58 UTC
Dec 4, 18:58 UTC
Dec 3, 2024
Resolved -
On December 3rd, between 23:29 and 23:43 UTC, Pull Requests experienced a brief outage and teams have confirmed the issue to be resolved. Due to brevity of incident it was not publicly statused at the time however an RCA will be conducted and shared in due course.
Dec 3, 23:30 UTC
Dec 3, 23:30 UTC
Resolved -
On December 3, 2024, between 19:35 UTC and 20:05 UTC API requests, Actions, Pull Requests and Issues were degraded. Web and API requests for Pull Requests experienced a 3.5% error rate and Issues had a 1.2% error rate. The highest impact was for users who experienced errors while creating and commenting on Pull Requests and Issues. Actions had a 3.3% error rate in jobs and delays on some updates during this time.
This was due to an erroneous database credential change impacting write access to Issues and Pull Requests data. We mitigated the incident by reverting the credential change at 19:52 UTC. We continued to monitor service recovery before resolving the incident at 20:05 UTC.
There are a few improvements we are making in response to this. We are investing in safe guards to the change management process in order to prevent erroneous database credential changes. Additionally, the initial rollback attempt was unsuccessful which led to a longer time to mitigate. We were able to revert through an alternative method and are updating our playbooks to document this mitigation strategy.
Dec 3, 20:05 UTC
This was due to an erroneous database credential change impacting write access to Issues and Pull Requests data. We mitigated the incident by reverting the credential change at 19:52 UTC. We continued to monitor service recovery before resolving the incident at 20:05 UTC.
There are a few improvements we are making in response to this. We are investing in safe guards to the change management process in order to prevent erroneous database credential changes. Additionally, the initial rollback attempt was unsuccessful which led to a longer time to mitigate. We were able to revert through an alternative method and are updating our playbooks to document this mitigation strategy.
Dec 3, 20:05 UTC
Update -
Pull Requests is operating normally.
Dec 3, 20:05 UTC
Dec 3, 20:05 UTC
Update -
Actions is operating normally.
Dec 3, 20:04 UTC
Dec 3, 20:04 UTC
Update -
API Requests is operating normally.
Dec 3, 20:02 UTC
Dec 3, 20:02 UTC
Update -
We have taken mitigating actions and are starting to see recovery but are continuing to monitor and ensure full recovery. Some users may still see errors.
Dec 3, 19:59 UTC
Dec 3, 19:59 UTC
Update -
Some users will experience problems with certain features of pull requests, actions, issues and other areas. We are aware of the issue, know the cause, and are working on a mitigation.
Dec 3, 19:54 UTC
Dec 3, 19:54 UTC
Investigating -
We are investigating reports of degraded performance for API Requests, Actions and Pull Requests
Dec 3, 19:48 UTC
Dec 3, 19:48 UTC
Resolved -
Between Dec 3 03:35 UTC and 04:35 UTC, availability of large hosted runners for Actions was degraded due to failures in background VM provisioning jobs. This was a shorter recurrence of the issue that occurred the previous day. Users would see workflows queued waiting for a large runner. On average, 13.5% of all workflows requiring large runners over the incident time were affected, peaking at 46% of requests. Standard and Mac runners were not affected.
Following the Dec 1 incident, we had disabled non-critical paths in the provisioning job and believed that would eliminate any impact while we understood and addressed the timeouts. Unfortunately, the timeouts were a symptom of broader job health issues, so those changes did not prevent this second occurrence the following day. We now understand that other jobs on these agents had issues that resulted in them hanging and consuming available job agent capacity. The reduced capacity led to saturation of the remaining agents and significant performance degradation in the running jobs.
In addition to the immediate improvements shared in the previous incident summary, we immediately initiated regular recycles of all agents in this area while we continue to address the issues in both the jobs themselves and the resiliency of the agents. We also continue to improve our detection to ensure we are automatically detecting these delays.
Dec 3, 04:39 UTC
Following the Dec 1 incident, we had disabled non-critical paths in the provisioning job and believed that would eliminate any impact while we understood and addressed the timeouts. Unfortunately, the timeouts were a symptom of broader job health issues, so those changes did not prevent this second occurrence the following day. We now understand that other jobs on these agents had issues that resulted in them hanging and consuming available job agent capacity. The reduced capacity led to saturation of the remaining agents and significant performance degradation in the running jobs.
In addition to the immediate improvements shared in the previous incident summary, we immediately initiated regular recycles of all agents in this area while we continue to address the issues in both the jobs themselves and the resiliency of the agents. We also continue to improve our detection to ensure we are automatically detecting these delays.
Dec 3, 04:39 UTC
Update -
We saw a recurrence of the large hosted runner incident (https://www.githubstatus.com/incidents/qq1m7mqcl6zk) from 12/1/2024. We've applied the same mitigation and see improvements. We will continue to work on a long term solution.
Dec 3, 04:38 UTC
Dec 3, 04:38 UTC
Update -
We are investigating reports of degraded performance for Hosted Runners
Dec 3, 04:16 UTC
Dec 3, 04:16 UTC
Investigating -
We are currently investigating this issue.
Dec 3, 04:11 UTC
Dec 3, 04:11 UTC
Dec 2, 2024
Resolved -
Between Dec 1 12:20 UTC and Dec 2 1:05 UTC, availability of large hosted runners for Actions was degraded due to failures in background VM provisioning jobs. Users would see workflows queued waiting for a runner. On average, 8% of all workflows requiring large runners over the incident time were affected, peaking at 37.5% of requests. There were also lower levels of intermittent queuing on Dec 1 beginning around 3:00 UTC. Standard and Mac runners were not affected.
The job failures were caused by timeouts to a dependent service in the VM provisioning flow and gaps in the jobs’ resilience to those timeouts. The incident was mitigated by circumventing the dependency as it was not in the critical path of VM provisioning.
There are a few immediate improvements we are making in response to this. We are addressing the causes of the failed calls to improve the availability of calls to that backend service. Even with that impact, the critical flow of large VM provisioning should not have been impacted, so we are improving the client behavior to fail fast and circuit break non-critical calls. Finally the alerting for this service was not adequate in this particular scenario to ensure fast response by our team. We are improving our automated detection from this to reduce our time to detection and mitigation of issues like this one in the future.
Dec 2, 01:05 UTC
The job failures were caused by timeouts to a dependent service in the VM provisioning flow and gaps in the jobs’ resilience to those timeouts. The incident was mitigated by circumventing the dependency as it was not in the critical path of VM provisioning.
There are a few immediate improvements we are making in response to this. We are addressing the causes of the failed calls to improve the availability of calls to that backend service. Even with that impact, the critical flow of large VM provisioning should not have been impacted, so we are improving the client behavior to fail fast and circuit break non-critical calls. Finally the alerting for this service was not adequate in this particular scenario to ensure fast response by our team. We are improving our automated detection from this to reduce our time to detection and mitigation of issues like this one in the future.
Dec 2, 01:05 UTC
Update -
We've applied a mitigation to fix the issues with large runner jobs processing. We are seeing improvements in telemetry and are monitoring for full recovery.
Dec 2, 00:57 UTC
Dec 2, 00:57 UTC
Update -
We continue to investigate large hosted runners not picking up jobs.
Dec 2, 00:14 UTC
Dec 2, 00:14 UTC
Update -
We continue to investigate issues with large runners.
Dec 1, 23:43 UTC
Dec 1, 23:43 UTC
Update -
We're seeing issues related to large runners not picking up jobs and are investigating.
Dec 1, 23:24 UTC
Dec 1, 23:24 UTC
Investigating -
We are currently investigating this issue.
Dec 1, 23:18 UTC
Dec 1, 23:18 UTC
Dec 1, 2024
Nov 30, 2024
No incidents reported.
Nov 29, 2024
No incidents reported.
Nov 28, 2024
Resolved -
This incident has been resolved.
Nov 28, 07:01 UTC
Nov 28, 07:01 UTC
Update -
We identified the issue and applied a mitigation, resulting in the cessation of timeouts. While we are considering this incident resolved for now, we are continuing to investigate the root cause and plan to implement a permanent fix. Updates will follow as we progress.
Nov 28, 07:00 UTC
Nov 28, 07:00 UTC
Update -
We are investigating issues with timeouts in some requests in Codespaces. We will update you on mitigation progress.
Nov 28, 06:36 UTC
Nov 28, 06:36 UTC
Investigating -
We are investigating reports of degraded performance for Codespaces
Nov 28, 06:27 UTC
Nov 28, 06:27 UTC
Resolved -
This incident has been resolved.
Nov 28, 05:11 UTC
Nov 28, 05:11 UTC
Update -
We identified and addressed failures in two proxy servers and applied mitigation. Since then, timeouts have ceased, and we are considering the incident resolved. We will continue to monitor the situation closely and provide updates if any changes occur.
Nov 28, 05:10 UTC
Nov 28, 05:10 UTC
Update -
We are investigating some network proxy issues that may be contributing to the timeouts in a small percentage of requests in Codespaces. We will continue to investigate.
Nov 28, 04:34 UTC
Nov 28, 04:34 UTC
Update -
We are continuing to investigate issues with timeouts in a small percentage of requests in Codespaces. We will update you on mitigation progress.
Nov 28, 04:03 UTC
Nov 28, 04:03 UTC
Update -
We are investigating issues with timeouts in some requests in Codespaces. Some users may not be able to connect to their Codespaces at this time. We will update you on mitigation progress.
Nov 28, 03:32 UTC
Nov 28, 03:32 UTC
Investigating -
We are investigating reports of degraded performance for Codespaces
Nov 28, 03:29 UTC
Nov 28, 03:29 UTC
Nov 27, 2024
No incidents reported.
Subscribe to our developer newsletter
Get tips, technical guides, and best practices. Twice a month. Right in your inbox.
Subscribe