Incident on 2018-10-21 23:09:19 UTC
Incident Report for GitHub
Resolved
  • 2018-10-21 23:09:19 - We are investigating reports of elevated error rates.
  • 2018-10-21 23:13:31 - We are investigating reports of service unavailability.
  • 2018-10-21 23:43:55 - We're investigating problems accessing GitHub.com.
  • 2018-10-22 00:05:37 - We're failing over a data storage system in order to restore access to GitHub.com.
  • 2018-10-22 00:23:54 - We're continuing to work on migrating a data storage system in order to restore access to GitHub.com.
  • 2018-10-22 00:43:12 - We continue to work on migrating a data storage system in order to restore access to GitHub.com.
  • 2018-10-22 01:02:49 - We continue to migrate a data storage system in order to restore full access to GitHub.com.
  • 2018-10-22 01:22:22 - We continue to work to migrate a data storage system in order to restore access to GitHub.com.
  • 2018-10-22 01:41:19 - We are continuing to work to migrate a data storage system in order to restore access to GitHub.com.
  • 2018-10-22 02:01:43 - We continue work to repair a data storage system for GitHub.com. You may see inconsistent results during this process.
  • 2018-10-22 02:23:48 - We are continuing to repair a data storage system for GitHub.com. You may see inconsistent results during this process.
  • 2018-10-22 02:42:28 - We continue to repair a data storage system for GitHub.com. You may see inconsistent results during this process.
  • 2018-10-22 03:00:09 - We are continuing to repair a data storage system for GitHub.com. You may see inconsistent results during this process.
  • 2018-10-22 03:28:17 - We continue working to repair a data storage system for GitHub.com. You may see inconsistent results during this process.
  • 2018-10-22 04:12:55 - We are continuing work to repair a data storage system for GitHub.com. You may see inconsistent results during this process.
  • 2018-10-22 05:05:37 - We continue working to repair a data storage system for GitHub.com. You may see inconsistent results during this process.
  • 2018-10-22 06:18:18 - We continue work to repair a data storage system for GitHub.com. You may see inconsistent results during this process.
  • 2018-10-22 06:51:50 - We are currently in the later stages of a restore operation, with the aim of serving fully consistent data within the next 2 hours.
  • 2018-10-22 07:35:38 - Our restore operations are proceeding as expected, on track for serving fully consistent data within the next 1.5 hours.
  • 2018-10-22 08:19:00 - Our restore operations are still on track to serve consistent data within the hour. We've posted an update at https://blog.github.com/2018-10-21-october21-incident-report/.
  • 2018-10-22 09:47:03 - We continue to monitor restores which are taking longer than anticipated. We estimate they will be caught up in an hour and a half.
  • 2018-10-22 11:56:52 - The majority of restore processes have completed. We anticipate all data stores will be fully consistent within the next hour.
  • 2018-10-22 13:18:59 - We are validating the consistency of information across all data stores. Webhooks and Pages builds remain paused.
  • 2018-10-22 15:09:47 - Background jobs remain paused as we near completion of validation of data consistency.
  • 2018-10-22 16:24:01 - We've completed validation of data consistency and have enabled some background jobs. We're continuing to monitor as the system recovers and expect to resume delivering webhooks at 16:45UTC.
  • 2018-10-22 16:45:19 - We have resumed delivery of webhooks and will continue to monitor as we process a delayed backlog of events.
  • 2018-10-22 16:51:27 - We have resumed Pages builds and will continue to monitor as we process a delayed backlog of events.
  • 2018-10-22 17:32:55 - We have temporarily paused delivery of webhooks while we address an issue. We are working to resume delivery as soon as possible.
  • 2018-10-22 18:09:34 - Webhook delivery remains paused while we work through an issue. We are currently deploying a fix, and expect to resume delivery soon.
  • 2018-10-22 18:26:41 - We have resumed webhook delivery and will continue to monitor as we process the backlog of events.
  • 2018-10-22 18:40:20 - We are working through the backlogs of webhook deliveries and Pages builds. We continue to monitor as the site recovers.
  • 2018-10-22 19:32:50 - We continue to work through the backlogs for Pages builds and webhook deliveries, and are monitoring as the site recovers.
  • 2018-10-22 20:03:52 - We continue working through the backlogs for Pages builds and webhook deliveries, and we are monitoring as the site recovers.
  • 2018-10-22 20:37:25 - We continue to work through the backlogs for Pages builds and webhook deliveries. We are monitoring as the site recovers.
  • 2018-10-22 21:11:49 - We continue working through the backlogs for Pages builds and webhook deliveries. We are monitoring as the site recovers.
  • 2018-10-22 21:46:24 - We continue working through the backlogs for Pages builds and webhook deliveries, and are monitoring as the site recovers.
  • 2018-10-22 22:23:07 - Webhook deliveries have caught up, and we continue working through the backlogs for Pages builds as the site recovers.
  • 2018-10-22 23:03:57 - Everything operating normally.
Posted 7 months ago. Oct 21, 2018 - 23:09 UTC
This incident affected: API Requests, Issues, PRs, Dashboard, Projects, and GitHub Pages.