Incident with Pull Requests
Incident
Report for GitHub
Resolved
From March 12, 2024 23:39 UTC to March 13, 2024 1:58 UTC, some Pull Requests updates were delayed and did not reflect the latest code that had been pushed. On average, 20% of Pull Requests page loads were out of sync and up to 30% of Pull Requests were impacted at peak. An internal component of our job queueing system was incorrectly handling invalid messages, resulting in stalled processing.
We mitigated the incident by shipping a fix to handle the edge case gracefully and allow processing to continue.
Once the fix was deployed at 1:47 UTC, our systems fully caught up with pending background jobs at 1:58 UTC.
We’re working to improve resiliency to invalid messages in our system to prevent future delays for these pull request updates. We are also reviewing our monitoring and observability to identify and remediate these types of failure cases faster.
We mitigated the incident by shipping a fix to handle the edge case gracefully and allow processing to continue.
Once the fix was deployed at 1:47 UTC, our systems fully caught up with pending background jobs at 1:58 UTC.
We’re working to improve resiliency to invalid messages in our system to prevent future delays for these pull request updates. We are also reviewing our monitoring and observability to identify and remediate these types of failure cases faster.
Update
Pull Requests is operating normally.
Update
We believe we've found a mitigation and are currently monitoring systems for recovery.
Update
We're continuing to investigate delays in PR updates. Next update in 30 minutes.
Update
We're continuing to investigate an elevated number of pull requests that are out of sync on page load.
Update
We're continuing to investigate an elevated number of pull requests that are out of sync on page load.
Update
We're seeing an elevated number of pull requests that are out of sync on page load.
Investigating
We are investigating reports of degraded performance for Pull Requests
This incident affected: Pull Requests.
