Outbound SMS Failures
Incident Report for CallRail
Postmortem

At 10am Eastern on September 30, the Let's Encrypt DST Root CA X3 certificate expired (see https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/)..) While all CallRail hosts and services had been checked and verified to be using the newer ISRG Root X1 certificate, an underlying 3rd-party component was still using the older root certificate. When the certificate expired, services using that component began to fail intra-service SSL handshakes, resulting in two customer-impacting issues:

  1. The main CallRail application was unable to communicate with the internal service responsible for sending SMS messages. This impacted our ability to send outgoing SMS messages, including Multi-Factor Authentication and Automated Response messages.
  2. Within the Application Dashboard, attempts to Load or Export data failed and returned an error message.

The issue was resolved by updating the 3rd-party component to use the root certificates that have been vetted and approved for use by other CallRail services rather than using its internal SSL certificates. Additionally, all other services using the affected third-party component have been similarly updated.

Posted Oct 04, 2021 - 08:59 EDT

Resolved
After deploying the fix, SMS message delivery is functioning normally.
Posted Sep 30, 2021 - 12:38 EDT
Identified
We have identified an issue with internal certificate handshakes and are deploying a fix to address the problem.
Posted Sep 30, 2021 - 12:05 EDT
Investigating
We are currently investigating an issue with outbound SMS message delivery failures, including Multi-Factor Authentication and Automated Response messages.
Posted Sep 30, 2021 - 10:42 EDT
This incident affected: Application Dashboard and Lead Center.