We are investigating reports of degraded performance.
Incident
Report for GitHub
Resolved
From 10:30 to 16:15 UTC on January 3rd, our `/settings/emails` page experienced downtime due to an outage in our contact permissions system. During this time customers were unable to update their email subscription preferences.
This system plays a crucial role in managing email subscriptions and the contactability of GitHub customers by our marketing and sales teams. Unfortunately, the outage prevented the generation of links for managing email subscriptions, resulting in the `/settings/emails` page timing out. Consequently, customers were unable to update their email subscription preferences during this period.
During the incident we implemented graceful degradation on the `/settings/email` page to enable certain functionality, albeit with slower response times. By approximately 16:15 UTC on January 3rd, our contact permissions system was fully operational again, and customers could manage their email subscription preferences as usual.
With the issue now resolved, our team is enhancing the `/settings/emails` page resilience to minimize the impact of system outages in the future. This includes better request timeout handling and improved metrics and monitoring in this area of the application.
This system plays a crucial role in managing email subscriptions and the contactability of GitHub customers by our marketing and sales teams. Unfortunately, the outage prevented the generation of links for managing email subscriptions, resulting in the `/settings/emails` page timing out. Consequently, customers were unable to update their email subscription preferences during this period.
During the incident we implemented graceful degradation on the `/settings/email` page to enable certain functionality, albeit with slower response times. By approximately 16:15 UTC on January 3rd, our contact permissions system was fully operational again, and customers could manage their email subscription preferences as usual.
With the issue now resolved, our team is enhancing the `/settings/emails` page resilience to minimize the impact of system outages in the future. This includes better request timeout handling and improved metrics and monitoring in this area of the application.
Update
The /settings/email page remains slow, but we are continuing to see improvements.
Update
We have resolved the timeouts on our end, but customers will still see slow load times on the /settings/email page.
Update
We have identified a potential mitigation for the issue and are rolling it out. Additionally, we are still working with our external provider for a mitigation from them.
Update
We are continuing to investigate reports of issues with the /settings/emails page. We are working with the provider to get a mitigation and are exploring options to mitigate on our end.
Update
We're seeing issues related to a provider and are engaging them to investigate the issue.
Update
We are continuing to investigate reports of issues with the /settings/emails page. We will continue to keep users updated on progress towards mitigation.
Update
We're seeing problems loading the /settings/emails page.
Investigating
We are currently investigating this issue.
