Application Dashboard and DNI Outage

Incident Report for CallRail

Resolved

Starting at 9:45pm EST, a configuration change to one of our legacy network firewalls resulted in blocking some ingress traffic to our application dashboard and DNI services. The configuration change took approximately 10 minutes to fully propagate. Call routing services were unaffected.

The configuration change was a result of replacing the legacy firewall configuration with a configuration that is standardized across our infrastructure and managed by automation tools.

This configuration change was vetted against our non-production infrastructure, however a slight difference between the environments led to a time period in which traffic was unable to be forwarded between our load balancers and application machines. Once the rest of the changes were applied and propagated no later than 10:20pm EST, the traffic was able to resume normally.

Posted Sep 24, 2019 - 21:45 EDT