We are investigating reports of degraded performance.
Incident
Report for GitHub
Resolved
This incident was the result of an infrastructure change that was made to our load balancers to prepare us for IPv6 enablement of GitHub.com. This change was deployed to a subset of our global edge sites.
The change had the unintended consequence of causing IPv4 addresses to start being passed as an IPv4-mapped IPv6-compatible address to our IP Allow List functionality.
For example 10.1.2.3 became ::ffff:10.1.2.3. While our IP Allow List functionality was developed with IPv6 in mind, it wasn't developed to handle these mapped addresses, and hence started blocking requests as it deemed these to be not in the defined list of allowed addresses. Request error rates peaked at 0.23% of all requests.
We have so far identified three remediation items here:
- Update the IP Allow List functionality to handle IPv4-mapped addresses.
- Audit the rest of our stack to confirm there are no further places this IPv4-mapped IPv6 addresses flaw exists.
- Improve our testing and monitoring processes to better catch these issues in the future.
The change had the unintended consequence of causing IPv4 addresses to start being passed as an IPv4-mapped IPv6-compatible address to our IP Allow List functionality.
For example 10.1.2.3 became ::ffff:10.1.2.3. While our IP Allow List functionality was developed with IPv6 in mind, it wasn't developed to handle these mapped addresses, and hence started blocking requests as it deemed these to be not in the defined list of allowed addresses. Request error rates peaked at 0.23% of all requests.
We have so far identified three remediation items here:
- Update the IP Allow List functionality to handle IPv4-mapped addresses.
- Audit the rest of our stack to confirm there are no further places this IPv4-mapped IPv6 addresses flaw exists.
- Improve our testing and monitoring processes to better catch these issues in the future.
Update
We have resolved the issue and confirmed all regions are now operating as expected.
Update
The fix for ip allow lists is currently rolling out; and we are awaiting confirmation from specific geographic regions.
Update
We are rolling out a fix to resolve the issues with IP allow lists. This should be resolved shortly.
Update
Some customers are experiencing issues with IP allow lists.
Investigating
We are currently investigating this issue.
