Heartbleed and Getting New Keys and Certificates

The last few days were difficult for anyone who hosts a web site or service that used a version of the OpenSSL library vulnerable to CVE-2014-0160. The Heartbleed SSL vulnerability has lived up to its name. We were definitely ill-prepared for both the effects of the vulnerability on our systems as well as the following flood of SSL reissue requests that we received once we made that functionality available. In addition, our upstream certificate provider and the Certificate Authorities we rely upon, have made reissuing certificates more difficult than we would have expected.

Our Systems

The first step we took was to identify which systems we operate were vulnerable. In the process we found that we were using a version of a Linux distro that would not be supported with a patched OpenSSL package. Thus we had to create our own package and begin deploying it to our test systems. Once that was done we continued the deployment to the remaining vulnerable servers. This all happened Tuesday, April 8th, starting in the late morning and finishing in the early afternoon. Once our systems were updated to the latest OpenSSL we then purchased and installed new SSL certificates.

During this period we were also working on the SSL reissue system. While we were able to release the certificate reissue system on Tuesday afternoon, things have not gone smoothly. To understand why, it helps to understand how the reissue process works right now.

Reissuing Certificates

When you submit a request to reissue a certificate purchased with DNSimple, we create a new private key and a new Certificate Signing Request (CSR). The original contact information and approval email are retained. At this point the process for reissuing diverges depending on which certificate type you are reissuing: we currently use RapidSSL for single-host certificates, and Comodo for wildcard certificates.

Both Certificate Authorities require re-authorization of the certificate request, which means you must still have the same email address that was used when the original certificate was issued, and that email address must function correctly. Additionally, both providers will only provide the reissued certificate directly to you via email, they will not send us the certificate.

For RapidSSL, we have an API call we can invoke, through Enom, which will submit a reissue request for the certificate. At this point the reissue is completely out of our control. This means that we are unable to know the status of the process, hence we can't place the reissued certificate in DNSimple once it is reissued.

For Comodo wildcard certificates, we must manually copy your CSR into Enom's support system, along with the original certificate order ID, and then Enom must manually request the reissue through Comodo. This is one of the reasons that the re-issue process is so slow. Like for RapidSSL, we have no visibility on the status of the process and we are not notified when the certificate is reissued.

At this point we have both workflows operating but we still expect some delays to occur, espcially with wildcard certificate reissuing.

You can find out more about the reissue process workflow at http://support.dnsimple.com/articles/reissuing-ssl-certificates/

What About Revocation?

We're still working with our certificate provider to determine if they will allow us to revoke reissued certificates. We believe that the old certificate for any reissued certificate must be revoked, but ultimately we're dependent on our upstream provider to do so. We will continue trying to convey the importance of revocation for these reissued certificates.

What About Changing Passwords and API Tokens?

General advice is to consider passwords and API tokens to be leaked, therefore you should change your DNSimple password and API key.

Things We Got Wrong

When dealing with this vulnerability, there were several things that we did wrong, or that we overlooked. I'd like to enumerate them for you:

• We did not fully grasp the effect that this vulnerability would have on either our systems or how it would effect the SSL services we offer to customers.
• We had no self-service functionality for reissuing certificates. The first implementation was created very quickly and we overlooked some important pieces, like providing access to download your new key once the certificate is reissued.
• We did not understand the reissue workflow that our upstream certificate provider would require us to adhere to. In fact, we were negotiating with them about improvements to the workflow even as we launched the UI.
• We had checks in place to stop purchases of multiple certificates with the same name, but this stopped our customers from being able to purchase a replacement certificate if they wanted to go that route.

Things We Got Right

• The DNSimple team worked well together, sharing the responsibities of patching our system, developing the reissue system, all while communicating with you through the blog, Twitter and our status page.
• Even though it was challenging, I believe we made the right decision to support reissuing certificates. Even though there are delays, it is good that customers are able to get an updated certificate without being obliged to purchase a new one.

Things We Are Fixing

• We've implemented as much of the self-service interface as possible for now, barring additional APIs from our upstream provider, or direct integration with one or more Certificate Authorities. There are still some bugs to work out but we'll keep at it.
• We now understand the reissue workflow much better, but we will continue to work with our provider, and find additional channels, to improve that workflow.
• We removed the limitation for purchasing certificates with the same name if the existing certificate is already issued. If you need to get a new certificate ASAP and cannot wait for a reissue, then you can now go this route.
• We're looking into alternative certificate providers for a longer-term solution. Ideally we will find a provider with a solid API for reissue and revocation as well as initial issuing.

Conclusion

It has been a challenging week, but we're happy to report that we've made progress and have some good ideas on our next steps to make dealing with issues like this more managable for us and you.

Share on Twitter and Facebook

Anthony Eden

I break things so Simone continues to have plenty to do. I occasionally have useful ideas, like building a domain and DNS provider that doesn't suck.