At 5:45a EDT we fully restored the application. Total downtime was approximately 6 hours, which is highly unusual. Since this incident started after hours we spent much longer than we normally would before moving to plan B. We don't anticipate any additional problems in the near term, but we will need to schedule another maintenance window to achieve a full resolution.
Application is confirmed to be operating as expected at this time.
Posted Oct 14, 2016 - 05:45 EDT
Monitoring
At 5:30a EDT Database maintenance has been completed. The application dashboard is now available and all calls are being routed normally. We are continuing to monitor for any issues.
Posted Oct 14, 2016 - 05:32 EDT
Identified
At 10:30p EDT we began investigating elevated error rates in our DNI and keyword tracking systems. We identified a problem with our primary database that requires unscheduled maintenance.
At 11:10p EDT we began this maintenance. During this time the application dashboard will be unavailable. Calls will continue to route via our backup routing system.