All calls are flowing smoothly at this time, and recording uploads are working in real-time again.
For customers with affected custom recordings, we've sent an email notification with further instructions. You'll still need to follow those directions to restore full functionality for your custom call flows, but in the meantime calls are still being directed to your destination numbers correctly.
We take our responsibility to store your data very seriously, and want to acknowledge the disruption this incident has caused. Our Engineering team is conducting an internal postmortem to determine how to prevent any similar incidents from occurring in the future. When that is completed, it will be available here on our status page; you can subscribe to updates to receive a notification when this occurs.
We believe all recording updates have been processed and new ones are flowing through smoothly at this time.
Posted Jun 28, 2019 - 19:57 EDT
Update
We're hearing a few reports that recordings that have been re-uploaded aren't completely active yet. We're churning through a queue of transcoding jobs to get those files ready for playback via phone, and expect that to be back to normal on that shortly. The files being uploaded are being properly stored, but it may take us a moment to get them ready for playback in the app and via phone.
Posted Jun 28, 2019 - 19:32 EDT
Monitoring
We have implemented a fallback call flow for any affected tracking numbers, which preserves functionality other than any custom recordings for greetings, menus, or voicemail prompts. In some cases, we'll be unable to recover these lost recordings. We are in the process of notifying affected customers via email.
We do believe all calls are reaching destination numbers at this time. (The vast majority of calls were reaching destination numbers all along, but every call should be at this time.) If you are still experiencing issues please let us know right away by filing a ticket at https://support.callrail.com.
Posted Jun 28, 2019 - 16:39 EDT
Update
Due to a configuration error, customer-uploaded recordings that were uploaded or recorded in our system more than 30 days ago are no longer available. This is causing call failures for tracking numbers which make use of customer-uploaded recordings.
Our engineering team is still investigating the root cause of that error, and we won’t be able to share more until we’re able to complete a postmortem. At the moment, we’re first focused on recovering full application functionality.
What immediate workaround is available?
The fastest solution is to disable any affected call flow, and instead configure your tracking number or keyword pool to forward directly to your destination number.
If you have your original recordings available, you can edit your call flows to upload them as a new recording. Alternately, you can change from the “Play Recording” option to “Read Message”.
What is our plan?
Right now, the engineering team is working on two fixes. The first will restore call flow functionality, even if you take no action to your flows. We’ll be able to insert standard messaging into your call flows to present sensible behavior to your callers. We expect to have this completed soon.
We're also working to expose a management page for custom recordings to indicate which are currently missing. Once this is complete, you’ll be able to identify which recordings are missing and replace them there, without having to edit call flows.
We recognize this is a serious impact to your business, and we're working to get things corrected as soon as we possibly can. If you have questions specific to your account, please reach out to our support team. We're answering calls as best we can, but volume is high and we're likely able to serve you better by submitting a ticket through https://support.callrail.com.
Posted Jun 28, 2019 - 13:22 EDT
Update
At this point we have isolated the problem only to call flows and tracking numbers with custom recordings (other than computer-generated text-to-speech). We are working on a temporary fix for remaining call flows and will have an update within the hour.
Posted Jun 28, 2019 - 12:56 EDT
Identified
We've identified the root cause of the issue; we apologize for the impact and are working quickly to resolve it.
Posted Jun 28, 2019 - 11:11 EDT
Update
We are continuing to investigate this issue.
Posted Jun 28, 2019 - 10:47 EDT
Investigating
We are currently investigating call routing issues occurring on a subset of calls.
Posted Jun 28, 2019 - 10:47 EDT
This incident affected: Call Tracking (Call Routing).