The last few days were difficult for anyone who hosts a web site or service that used a version of the OpenSSL library vulnerable to CVE-2014-0160. The Heartbleed SSL vulnerability has lived up to its name. We were definitely ill-prepared for both the effects of the vulnerability on our systems as well as the following flood of SSL reissue requests that we received once we made that functionality available. In addition, our upstream certificate provider and the Certificate Authorities we rely upon, have made reissuing certificates more difficult than we would have expected.
The first step we took was to identify which systems we operate were vulnerable. In the process we found that we were using a version of a Linux distro that would not be supported with a patched OpenSSL package. Thus we had to create our own package and begin deploying it to our test systems. Once that was done we continued the deployment to the remaining vulnerable servers. This all happened Tuesday, April 8th, starting in the late morning and finishing in the early afternoon. Once our systems were updated to the latest OpenSSL we then purchased and installed new SSL certificates.
During this period we were also working on the SSL reissue system. While we were able to release the certificate reissue system on Tuesday afternoon, things have not gone smoothly. To understand why, it helps to understand how the reissue process works right now.
When you submit a request to reissue a certificate purchased with DNSimple, we create a new private key and a new Certificate Signing Request (CSR). The original contact information and approval email are retained. At this point the process for reissuing diverges depending on which certificate type you are reissuing: we currently use RapidSSL for single-host certificates, and Comodo for wildcard certificates.
Both Certificate Authorities require re-authorization of the certificate request, which means you must still have the same email address that was used when the original certificate was issued, and that email address must function correctly. Additionally, both providers will only provide the reissued certificate directly to you via email, they will not send us the certificate.
For RapidSSL, we have an API call we can invoke, through Enom, which will submit a reissue request for the certificate. At this point the reissue is completely out of our control. This means that we are unable to know the status of the process, hence we can't place the reissued certificate in DNSimple once it is reissued.
For Comodo wildcard certificates, we must manually copy your CSR into Enom's support system, along with the original certificate order ID, and then Enom must manually request the reissue through Comodo. This is one of the reasons that the re-issue process is so slow. Like for RapidSSL, we have no visibility on the status of the process and we are not notified when the certificate is reissued.
At this point we have both workflows operating but we still expect some delays to occur, espcially with wildcard certificate reissuing.
You can find out more about the reissue process workflow at http://support.dnsimple.com/articles/reissuing-ssl-certificates/
We're still working with our certificate provider to determine if they will allow us to revoke reissued certificates. We believe that the old certificate for any reissued certificate must be revoked, but ultimately we're dependent on our upstream provider to do so. We will continue trying to convey the importance of revocation for these reissued certificates.
General advice is to consider passwords and API tokens to be leaked, therefore you should change your DNSimple password and API key.
When dealing with this vulnerability, there were several things that we did wrong, or that we overlooked. I'd like to enumerate them for you:
It has been a challenging week, but we're happy to report that we've made progress and have some good ideas on our next steps to make dealing with issues like this more managable for us and you.
I break things so Simone continues to have plenty to do. I occasionally have useful ideas, like building a domain and DNS provider that doesn't suck.
Configure DNSimple as your secondary DNS provider to improve your domain's availability and redundancy with AXFR zone transfers.
Get a free limited-edition t-shirt featuring the characters of howdns.works and howhttps.works with any new yearly subscription to DNSimple.