Interruption in call routing
Incident Report for CallRail
Postmortem

Between 12:00 pm EDT and 1:00 pm EDT on 22 March 2022, one of our critical upstream providers suffered a service outage. The consequence of this for CallRail customers was that calls could not be routed to their destinations. Calls that went to CallRail voicemail were recorded correctly. Calls that were not routed are visible in the call log as either abandoned or short-duration calls, depending on Call Flow configuration.

While this incident was outside of our direct control, and such incidents with our upstream vendors are rare, we are taking steps to mitigate the risk of similar future failures:

  • We are improving the visibility of our status reporting and expanding it to channels beyond direct subscriptions to our status updates (subscribe to the existing notifications at status.callrail.com)
  • We are implementing emergency options to disable Dynamic Number Insertion in the event of a broad call routing failure, to allow calls to our customers to be routed directly to a customer’s main number. These calls would not be attributed or recorded by CallRail, but would ring through, reducing the impact on our customers' businesses.
Posted Mar 24, 2022 - 15:49 EDT

Resolved
This issue has been resolved, though we continue to pursue details with our upstream provider.
Posted Mar 22, 2022 - 14:30 EDT
Monitoring
Our provider indicates they have implemented and are monitoring a fix. Our tests indicate that calls through our numbers are now being routed correctly and error volume is dropping. We will continue to monitor while the situation resolves.
Posted Mar 22, 2022 - 13:04 EDT
Update
Our upstream provider continues to attempt to address this issue; we are monitoring and will update as soon as we know more.
Posted Mar 22, 2022 - 12:59 EDT
Identified
One of our upstream telephony providers is investigating an outage in their service. We continue to investigate to verify this is the root cause of issues affecting our customers.
Posted Mar 22, 2022 - 12:22 EDT
Investigating
We are experiencing difficulties in Call Routing that we are currently investigating. At this time you may see degraded performance around placing outbound and receiving inbound calls, both through our main CallRail application as well as our Lead Center application
Posted Mar 22, 2022 - 12:20 EDT
This incident affected: Lead Center (Lead Center General) and Call Tracking (Call Routing).