GitHub header

Disruption with some GitHub services

Incident Report for GitHub

Resolved

On August 27, 2025 between 20:35 and 21:17 UTC, Copilot, Web and REST API traffic experienced degraded performance. Copilot saw an average of 36% of requests fail with a peak failure rate of 77%. Approximately 2% of all non-Copilot Web and REST API traffic requests failed.

This incident occurred after we initiated a production database migration to drop a column from a table backing copilot functionality. While the column was no longer in direct use, our ORM continued to reference the dropped column. This led to a large number of 5xx responses and was similar to the incident on August 5th. At 21:15 UTC, we applied a fix to the production schema and by 21:17 UTC, all services had fully recovered.

While repairs were in progress to avoid this situation, they were not completed quickly enough to prevent a second incident. We have now implemented a temporary block for all drop column operations as an immediate solution while we add more safeguards to prevent similar issues from occurring in the future. We are also implementing graceful degradation so that Copilot issues will not impact other features of our product.
Posted Aug 27, 2025 - 21:27 UTC

Update

API Requests and Issues are operating normally.
Posted Aug 27, 2025 - 21:27 UTC

Update

We've discovered the cause of the service disruption and applied a mitigation.
Posted Aug 27, 2025 - 21:25 UTC

Update

We are continuing to investigate this issue.
Posted Aug 27, 2025 - 21:13 UTC

Update

API Requests is experiencing degraded performance. We are continuing to investigate.
Posted Aug 27, 2025 - 20:58 UTC

Update

The team is aware of the root cause of this issue and is working to mitigate the issue quickly.
Posted Aug 27, 2025 - 20:55 UTC

Update

Issues is experiencing degraded performance. We are continuing to investigate.
Posted Aug 27, 2025 - 20:50 UTC

Update

API Requests is experiencing degraded availability. We are continuing to investigate.
Posted Aug 27, 2025 - 20:48 UTC

Investigating

We are currently investigating this issue.
Posted Aug 27, 2025 - 20:41 UTC
This incident affected: API Requests and Issues.