Writing open-source Chef cookbooks (part I)

As both Chef and it's community evolves, they develop better techniques and tools for managing infrastructure. Some use patterns fade as others emerge. One of the most common patterns that has been working for years is the open-source community cookbook. This pattern has suffered noticiable changes in the last two years. This is the first of a series of blog posts that try to highlight this evolution and how we have used it for one of the open-source cookbooks that we maintain at DNSimple, the PowerDNS cookbook.

The Old Way Of Creating A Cookbook.

Many open-source Chef cookbooks have this structure:

• One or more recipes offering different basic functionalities for the specific piece of software to configure.
• A huge collection of attribute files that offered "sane defaults" to these recipes.
• And of course, templates, libraries, etc.

The idea behind is to use wrap this cookbook with another one, private to the organizacion, this wrapper cookbook will contain all the customizations in form of override attributes, templates and so on, and will include the required recipes from the open-source cookbook.

A good example of this approach is our own 2.0 version of PowerDNS cookbook, which is designed to be wrapped by including one recipe and setting the attributes in the private cookbok.

The PowerDNS cookbook had limitations. It just offered a few configurations that were better suited to our usage and not much else. The main issue here is the cookbook structure that makes expanding the cookbook to accommodate another Linux distribution, init system or PowerDNS backend are very hard. Why? Because it's difficult to isolate functionality from one recipe to another at the same time and maintain flexibility.

Attribute overrides are a common place people get bitten when working with Chef, especially since you can add overrides in so many levels of Chef. It's easy to get lost and try to find where the attribute was overriden and it's even worse when an attribute is saved to the server and you forget to delete it there even after you've removed it from the cookbook.

Finally, why if a user wanted something that we did not take into account at all?, for example: why if a user wanted to configure two database backends? For this user our cookbook would be completely useless.

A New Pattern Emerges

Last year, this talk from Tim Smith showed a new way to write open-source cookbooks in a composable, reusable way. It highlights all of the problems of the attribute based method and proposes a new pattern based on Chef custom resources instead of attributes. This is also highlighted in the Chef custom resources documentation notes but I found difficult to find more information about it, this the final reason of this series of posts; to expose this information to a broad Chef audience.

The idea behind this pattern is to split the functionality of the monolithic, attribute based open-source cookbook in several custom resources. These resources are equivalent to building blocks for the cookbook end users, to create their wrapper cookbooks. These building blocks are designed to provide maximum flexibility and adaptability to cover the maximum (or all) of the use cases of our users by composing this building blocks in a recipe, usually in a wrapper cookbook. This new design also makes easier to contribute and expand the cookbooks, this part will be explain the second article of the series.

To illustrate it we will use the 3.0 version of the PowerDNS cookbook, which provides several resources to use as those building blocks:

• pdns_authoritative_install, this resource installs the PowerDNS authoritative server from official repositories.
• pdns_authoritative_config, this resource creates a PowerDNS set of configuration options.
• pdns_authoritative_service, this resource defines and manages PowerDNS service.
• pdns_authoritative_backend, this resource installs PowerDNS backends.

https://github.com/dnsimple/chef-pdns/blob/enhancement/authoritative_resource/ Now let's use these building blocks in a wrapper cookbook to create our specific PowerDNS installation. This example is adapted from a recipe obtained in the fixtures of the chef-pdns cookbook. We will take the role now of an operator who wants to deploy a PowerDNS server with PostgreSQL as a data backend using our cookbook. More specifically, we want to create a recipe that will take care of installing a PowerDNS authoritative server, the PostgreSQL pdns backend, define a set of options to configure PowerDNS in order connect to PostgreSQL through the backend and install and configure PostgreSQL and with the pdns database schema and database user as well.

Installing the PowerDNS Package

The first of our custom resources just installs the PowerDNS server, we are naming it: server-01.

pdns_authoritative_install 'server-01' do
  action :install
end

Enabling PowerDNS service

Then we enable the PowerDNS service, we use our custom service resource for that.

pdns_authoritative_service 'server-01' do
  action :enable
end

Configuring PowerDNS

We create a configuration for an authoritative server, using the PostgreSQL backend and with a few parameters that allow the backend to connect to the database.

pdns_authoritative_config 'server-01' do
  action :create
  launch ['gpgsql']
  variables(
    gpgsql_host: '127.0.0.1',
    gpgsql_user: 'pdns',
    gpgsql_port: 5432,
    gpgsql_dbname: 'pdns',
    gpgsql_password: 'wadus'
    )
end

Install the PostgreSQL Backend

Finally we use the last building block which is dedicated to install PowerDNS backends to install our desired backend.

pdns_authoritative_backend 'postgresql' do
  action :install
end

The final parts of our recipe are dedicated to install and configure PostgreSQL and load the PowerDNS database schema.

Install PostgreSQL And Ruby Bindings

First we install PostgreSQL server and the PostgreSQL ruby bindings by including specific recipes in the PostgreSQL cookbook.

include_recipe 'postgresql::server'
include_recipe 'postgresql::ruby'

Database And User

We use the database cookbook to create a database user pdns and a database with the same name.

connection_info = { host: '127.0.0.1', username: 'postgres' }

postgresql_database_user 'pdns' do
  connection connection_info
  superuser true
  password 'wadus'
  action :create
end

postgresql_database 'pdns' do
  connection connection_info
  action :create
end

Load Database Schema

This two resources create the pdns schema on the database

cookbook_file '/var/tmp/schema_postgres.sql' do
  owner 'root'
  group 'root'
  mode 0755
end

execute 'psql -d pdns < /var/tmp/schema_postgres.sql' do
  user 'postgres'
  action :run
  not_if 'psql -t -d pdns -c "select \'public.domains\'::regclass;"', user: 'postgres'
end

Restart PowerDNS

In the last step we restart PowerDNS by using again our custom service resource.

pdns_authoritative_service 'server-01' do
  action :restart
end

Monolythic vs Custom Resources

In this example we have shown how to replace a monolythic recipe (include_recipe::pdns_authoritative) with 4 custom resources. The reason is to give flexibility to the cookbook end users. Each one of them has very different needs and we should try to create cookbooks that allow deploying and configuring stuff in the simplest, cleanest way.

Coming back from a similar question to one highlighted at the beginning of this post: What if an operator does not want to restart the service right after a config change or wants to do it manually? With this pattern it's possible, with the old pattern it required using ugly hacks such us chef rewind/unwind or editing the resources.

Summing it up

I hope that with this article the motivations behind the new pattern and how to use it are better understood. How our end users are better served by building cookbooks with custom resources. In the next post we will explain how to create your own open-source community cookbook following this pattern.

At DNSimple we love to share what we have learnt in our quest for Domain Management automation.

Share on Twitter and Facebook

Jacobo García López de Araujo

Devops, infrastructure, urban cyclist, music nerd.