Degraded Performance
Incident Report for CallRail
Postmortem

At about 10:40 am on 11/30, CallRail began experiencing an issue that led to customers experiencing latency, transcript and recording delays, and Lead Center connection problems.

We recently completed an infrastructure upgrade that will enable us to simplify our platform and deploy processes as well as improve our ability to add new functionality to these existing services quickly. This is critical for stability and scalability.

As part of this upgrade, we needed to balance the number of resources available and DB connections provisioned to meet the needs to customer and internal infrastructure requests.

Going forward, with the initial resource adjustment complete, our monitoring will allow these adjustments to happen without customer impact.

Posted Dec 02, 2022 - 14:09 EST

Resolved
This incident has been resolved. Some transcripts from the affected time period on 2022-11-30 are still processing, but will fill in as processing completes.
Posted Nov 30, 2022 - 19:23 EST
Monitoring
The issue has been resolved and we are continuing to monitor performance of the application. Call recordings and transcripts are still delayed, but should be available shortly as processing completes.
Posted Nov 30, 2022 - 14:19 EST
Identified
The issue has been identified and a fix is being implemented. Calls are routing, page load time is improving, and Lead Center is operational. Call recordings are still delayed, but will be available shortly after processing completes.
Posted Nov 30, 2022 - 13:10 EST
Update
We are continuing to investigate this issue.
Posted Nov 30, 2022 - 12:08 EST
Investigating
Our monitoring tools are reporting degraded performance. The impact is affecting load time of both Callrail Analytics and Lead Center. Additionally, all recording availability is delayed, but that recordings are still being made and will be available when the issue is resolved. Our team is currently investigating this issue and will post an update as we learn more.
Posted Nov 30, 2022 - 11:59 EST
This incident affected: Call Tracking (CallRail Web Application), Lead Center (Lead Center Web Application), and Call Recording, Call Transcription.