Incident Report: DNS Outage (July 17th, 2020)
Anthony Eden
on
On Friday, July 17th, 2020, at approximately 21:11 UTC, DNSimple operations engineers identified a DNS incident affecting multiple DNSimple data centers. The outage was the result of a configuration issue at our DDoS defense provider's network backbone. This resulted in a significant portion of traffic to our authoritative name server network either arriving after substantial delay, or being dropped altogether, for a period of approximately 15 minutes.
For the last six years, we have relied on Cloudflare's DNS Firewall product to defend the DNSimple authoritative DNS network from DDoS attacks. Cloudflare has over 200 Points of Presence around the world, and combined with our six Points of Presence, we have been able to safely operate DNSimple's DNS network, while at the same time, achieving very good response times for DNS authoritative requests. Cloudflare's service has been instrumental in protecting DNSimple's customers from a variety of denial of service attacks.
Due to this outage, we are now investigating additional DDoS defense providers to augment our defense capabilities. Our customers rely on consistent responses from our Authoritative DNS network, and we owe it to all of them to remove potential single points of failure whenever possible, to ensure maximum uptime. Our goal is to add an additional DDoS defense provider before the end of 2020. Regardless, we will continue to use Cloudflare into the future.
As the CEO of DNSimple, it is my job to ensure we deliver the best experience to our customers. This incident demonstrates that we have more work to do. I apologize to every customer impacted by this issue, and assure you we are working hard to address the risk of this single point of failure.
Anthony Eden
I break things so Simone continues to have plenty to do. I occasionally have useful ideas, like building a domain and DNS provider that doesn't suck.
We think domain management should be easy.
That's why we continue building DNSimple.
