Tales of a Chef Workflow: Packaging

Over the past couple of years at DNSimple we've learned many valuable lessons with Chef in terms of cookbook authoring, workflow, and system design. We also give back to the Chef community, by providing a great platform and ready-to-use cookbooks for DNS automation and domain management with Chef.

This post will be part of a multi-part series designed to share some of these lessons and how you can avoid them if you're considering these options yourself. Maybe you are already in this situation and looking for a way out of them. Today we're going to be covering the anti-pattern of using Chef Cookbooks to build sources files into binaries and installing them. While it's not a bad way to start, it is most certainly not sustainable and can be rather unpredictable. Don't believe me? Keep on reading to find out why.

Please note that I am making an assumption here that you've got some basic knowledge of Chef and how resources work.

The naive approach

Years ago when I first started with Chef, one of the very first things I wrote was a recipe to download the source code to nginx, compile it, then install the binary. Seems pretty easy, yeah? Well, it definitely is until you realize you've got to build in guards for each of your resources to prevent things like re-compiling every run, randomly replacing compiled copies that are running, etc.

This kind of approach might look like so:

version = node['python']['version']
install_path = "#{node['python']['prefix_dir']}/lib/python#{version.split(/(^\d+\.\d+)/)[1]}"

remote_file "#{Chef::Config[:file_cache_path]}/Python-#{version}.tar.bz2" do
  source "#{node['python']['url']}/#{version}/Python-#{version}.tar.bz2"
  checksum node['python']['checksum']
  mode '0755'
  not_if { ::File.exist?(install_path) }
end

bash 'build-and-install-python' do
  cwd Chef::Config[:file_cache_path]
  code <<-EOF
    tar -jxvf Python-#{version}.tar.bz2
    (cd Python-#{version} && ./configure #{configure_options})
    (cd Python-#{version} && make && make install)
  EOF
  not_if { ::File.exist?(install_path) }
end

Does this example look oddly familiar to you? Well, that is because it comes from the official Chef documentation for remote_file. The code above has a lot going on. We have to grab the source file if it's not already downloaded, then execute a separate shell to run the compile commands via an ad-hoc script unless it's already been installed by make. If I were a system administrator automating the typical unpack, make, and make install pattern I'd probably reach for the same thing myself.

What could possibly go wrong?

We've employed this pattern ourselves at DNSimple in our own Chef setup, but it has some critical flaws that have bitten us in the past. First, it's difficult to maintain in the long run since every new version change has the chance to alter this sequence. What might be a make and make install today might instead be a configure with special options along with another make, some other special steps, etc. This makes your build process more complex.

Another way this can go wrong is the compile might just randomly not work or introduce a new bug causing the Chef run to fail. If this software (say Python or Ruby in this example) requires this to function, you've just broken a deployment and potentially caused an outage.

Let's not forget the final reason this is not the best solution. Generally speaking compiled binaries are a pretty optimized and straightforward thing. They can take a long time when you compile full language binaries like Python, Ruby, and PHP which ultimately slows down your chef runs any time this process changes. This time sink gets multiplied across your systems. Especially if you operate on limited resources like small Digital Ocean instances or ec2 nodes where you'll be making tea and reading Game of Thrones while your stuff compiles on a single shared CPU.

Our solution

There are a couple ways we have resolved this issue for us. Everything from compiling language binaries to internal services were being run through the above pattern and it was majorly slowing down our Chef runs. Just to deploy our website on a brand new system or even for testing was taking upwards of an hour depending on the resources available. Anthony had it even worse with a limited DSL connection at his home making it entire hours just to test one change in our cookbooks.

First, we started to use trusted public packages for languages such as Ruby. This meant deleting the crazy download, unpack, compile dance in favor of adding a PPA since we deploy over Ubuntu systems. Here is a snippet directly from our cookbook.

apt_repository 'brightbox-ruby-ppa' do
  uri 'ppa:brightbox/ruby-ng'
  distribution node['lsb']['codename']
  components ['main']
  deb_src true
end

package 'ruby2.3' do
  action :upgrade
  notifies :reload, 'ohai[refresh_rubies]', :immediately
  notifies :run, 'execute[reinstall_bundler]', :immediately
end

package 'ruby2.3-dev' do
  action :upgrade
  notifies :reload, 'ohai[refresh_rubies]', :immediately
end

execute 'reinstall_bundler' do
  command '/usr/bin/env gem install bundler'
  action :nothing
end

ohai 'refresh_rubies' do
  plugin 'languages/ruby'
  action :nothing
end

Let me break this code down a bit to walk through it.

apt_repository 'brightbox-ruby-ppa' do
  uri 'ppa:brightbox/ruby-ng'
  distribution node['lsb']['codename']
  components ['main']
  deb_src true
end

The above example adds the Brightbox managed apt packages for Ruby. They open source their build systems and process so we trust them to do the right thing for compiling Ruby. We can just as easily do this part ourselves, but that is for later in this article.

package 'ruby2.3' do
  action :upgrade
  notifies :reload, 'ohai[refresh_rubies]', :immediately
  notifies :run, 'execute[reinstall_bundler]', :immediately
end

package 'ruby2.3-dev' do
  action :upgrade
  notifies :reload, 'ohai[refresh_rubies]', :immediately
end

This above example installs Ruby 2.3 and the development package for us to be able to compile rubygems that need the Ruby header files. Don't worry about those notifications, I'm about to explain those below.

execute 'reinstall_bundler' do
  command '/usr/bin/env gem install bundler'
  action :nothing
end

Above, we are installing bundler since it is not there by default in Ruby (and most developers expect it to be there if you use a Gemfile to manage your Ruby project). This is triggered any time the ruby package upgrades to a new version.

ohai 'refresh_rubies' do
  plugin 'languages/ruby'
  action :nothing
end

This last example above is an Ohai trick where the installation of the new Ruby tells Ohai to reload. We need to tell Chef there is a new Ruby available during the Chef run. Why? Well, if you happen to have Ruby projects that use Chef information to find where your system Ruby is located (and not the one installed via Chef) then you need to do this last step. The consequence is that gem installations will go to the wrong location and you'll be left scratching your head wondering why your new Ruby doesn't have any gems.

Another solution

Outside of the language compilation example, there is another way we solved the compiling problem by effectively making our own native system packages. How did we do that? We used FPM and FPM Cookery to grab the source, compile, and package our various internal services and even other dependencies like nginx. How did we do that? Let me show you!

FPM Cookery is a wrapper around the FPM project which aims to make building packages for various package managers (or even just a plain tarball) easy to do. Below is an example of our own custom nginx build which results in a debian package we can install into our Ubuntu systems easily. With FPM Cookery you can write repeatable recipes that compile into easily distributable system packages.

require 'mkmf'

class Nginx < FPM::Cookery::Recipe
  description 'a high performance web server and a reverse proxy server'

  name     'nginx'
  version  '1.8.1'
  revision 4
  homepage 'http://nginx.org/'
  source   "http://nginx.org/download/nginx-#{version}.tar.gz"
  sha256   '8f4b3c630966c044ec72715754334d1fdf741caa1d5795fb4646c27d09f797b7'

  section 'httpd'

  build_depends 'libpcre3-dev', 'zlib1g-dev', 'libssl-dev'
  depends       'libpcre3', 'zlib1g', 'libssl1.0.0'

  provides  'nginx-full', 'nginx-common'
  replaces  'nginx-full', 'nginx-common'
  conflicts 'nginx-full', 'nginx-common'

  config_files '/etc/nginx/nginx.conf', '/etc/nginx/mime.types'

  def build
    configure \
      '--with-http_stub_status_module',
      '--with-http_ssl_module',
      '--with-http_gzip_static_module',
      '--with-pcre-jit',
      '--with-debug',
      '--with-ipv6',

      prefix: prefix,

      user: 'www-data',
      group: 'www-data',

      pid_path: '/var/run/nginx.pid',
      lock_path: '/var/lock/nginx.lock',
      conf_path: '/etc/nginx/nginx.conf',
      http_log_path: '/var/log/nginx/access.log',
      error_log_path: '/var/log/nginx/error.log',
      http_proxy_temp_path: '/var/lib/nginx/proxy',
      http_fastcgi_temp_path: '/var/lib/nginx/fastcgi',
      http_client_body_temp_path: '/var/lib/nginx/body',
      http_uwsgi_temp_path: '/var/lib/nginx/uwsgi',
      http_scgi_temp_path: '/var/lib/nginx/scgi'

    make
  end

  def install
    # config files
    (etc/'nginx').install Dir['conf/*']

    # server
    sbin.install Dir['objs/nginx']

    # man page
    man8.install Dir['objs/nginx.8']
    gzip_path = find_executable 'gzip'
    safesystem gzip_path, man8/'nginx.8'

    # support dirs
    %w( run lock log/nginx lib/nginx ).map do |dir|
      (var/dir).mkpath
    end
  end
end

I won't go into details about FPM and FPM Cookery, but with those tools we were able to pre-compile internal services and other software in our stack very easily. We built a workflow in Travis-CI to automate creating these packages for us via our github repos. You might be thinking after reading this, "How do you actually use this in Chef?" well, I have an answer for you below.

Package based installs via Chef

To store and retrieve our packages via apt in our Ubuntu systems, we signed up for Packagecloud which is one of many hosting services out there that allow you to store private and public apt, rpm, Ruby, Java, and Python repositories. Getting the builds from Travis-CI to Packagecloud was easy because they already have an integration with them, so once a build succeeds the package is automatically sent over to Packagecloud and made available to our servers.

The last piece of this puzzle is getting Chef to install the package via Packagecloud. They already provide a Chef Resource for setting this up and all you need to do is configure it with your Packagecloud repo key. Below is a live code example from how we install our custom nginx package via Packagecloud.

packagecloud_repo 'dnsimple/packages' do
  type 'deb'
  master_token 'thisiswhereyouputyourtoken'
end

apt_package 'nginx' do
  version node['nginx']['version']
  action :install
end

Under the covers, the packagecloud_repo resource configures a new apt source and provisions the GPG key automatically for you. If you were using an rpm based linux distribution then you could swap the type here and it would configure everything accordingly. After the apt repository is created it is then automatically synchronized which allows for the later apt_package resource to install the nginx package.

Why don't you give this a try?

Take note in that we're not downloading source code, managing caches, or compiling code. From our internal cookbook runs we've saved over an hour in time by switching to this method of package installation so I would highly recommend you give this pattern a try. Faster Chef runs mean better tests, quicker deployments of new and existing systems, and more consistent upgrades. It's certainly helped us and I'd like to thank Packagecloud for running an amazing, well-documented, and easy to use service.

Share on Twitter and Facebook

Aaron Kalin

Software and Server maintainer by day, board and video game geek by night.