On February 25, 2025, between 00:17 UTC and 01:08 UTC, GitHub Packages experienced a service degradation, leading to failures uploading and downloading packages, along with increased latency for all requests to GitHub Packages registry. At peak impact, about 14% of uploads and downloads failed, and all Packages requests were delayed by an average of 7 seconds. The incident was caused by the rollout of a database configuration change that resulted in a degradation in database performance. We mitigated the incident by rolling back the contributing change and failing over the database. In response to this incident, we are tuning database configurations and resolving a source of deadlocks. We are also redistributing certain workloads to read replicas to reduce latency and enhance overall database performance.
Posted Feb 25, 2025 - 01:08 UTC
Update
We have confirmed recovery for the majority of our systems. Some systems may still experience higher than normal latency as they catch up.
Posted Feb 25, 2025 - 01:08 UTC
Update
We have identified the issue impacting packages and have rolled out a fix. We are seeing signs of recovery and continue to monitor the situation.
Posted Feb 25, 2025 - 00:41 UTC
Investigating
We are investigating reports of degraded performance for Packages
