When we initially launched our API v1, we knew that it was just the first step towards our vision of an extensive domain management automation API to support our platform for DNS and domain automation. API v1 was designed as a bridge between the original API v0 and a new set of redesigned API. With years of API v1 under the belt, we had a clear picture of what worked and what didn't, and we were ready to start working on a new, completely redesigned API.

We decided to not evolve that system, but to start fresh with a new architecture.

API v1 was built with a classic Ruby on Rails app: complex business logic spread across models and controllers, rendering logic delegated to respond_to, and ActiveRecord serializers. This structure was inherited from API v0, where we adopted the standard Rails way of serving different responses from the same controller that was quite common back in 2010.

This structure, however, introduces several maintenance and scalability issues. Here are some problems we found and how we solved them.

Chaotic Code Organization

Without a clear guidance from the framework, and with no explicit team guidelines, the code for complex use cases was randomly scattered across controllers and models. The domain registration system had a different organization from the domain renewal, which had a different system than domain transfers, and so on. As you can imagine, this made maintenance hard in a large codebase like ours.

Here's an example of how the registrations controller used to be back in 2011:

class DomainRegistrationsController < ApplicationController
  helper :domains

  before_filter :require_user
  before_filter :setup_name_and_domain
  before_filter :require_subscription
  before_filter :setup_number_of_years
  before_filter :setup_registration_price

  def setup_name_and_domain
    # ...
  end
  protected :setup_name_and_domain
    
  def setup_number_of_years
    # ...
  end
  protected :setup_number_of_years

  def setup_registration_price
    # ...
  end
  protected :setup_registration_price

  # TODO: refactor this and update - duplicate code
  def create
    # ...
  end

  private
  def process_registration(payment_type)
    # ...
  end

  def register
    if @domain.register(@number_of_years, params[:extended_attribute])
      if @domain.registering?
        DomainNotifier.domain_registering(@domain).deliver
      elsif !@domain.registered?
        DomainNotifier.failed_domain_registration(@domain).deliver 
      end

      # TODO: if the domain is registering, then what happens here?
      if params[:privacy] == '1'
        if @wpps_supported
          @domain.enable_wpps
          DomainNotifier.whois_privacy_protection_purchased(@domain).deliver
        else
          DomainNotifier.whois_privacy_protection_unavailable(@domain).deliver
        end
      end

      # TODO: if the domain is registering and fails then this will result
      # in an offering being used with no domain delivered.
      @offer.use! if @offer

      respond_to do |format|
        format.html { redirect_to domains_path }
        format.json { render :json => @domain.to_json(:except => 'powerdns_domain_id'), :location => domain_path(@domain), :status => :created }
        format.xml { render :xml => @domain.to_xml(:except => 'powerdns_domain_id'), :location => domain_path(@domain), :status => :created }
      end
    else
      @domain.registration_failed!
      respond_to do |format|
        format.html { render :action => 'new' }
        format.json { render :json => {:name => @domain.name, :errors => @domain.errors.full_messages }, :status => 422 }
        format.xml { render :xml => {:name => @domain.name, :errors => @domain.errors.full_messages }, :status => 422 }
      end
    end
  end
  
  # TODO: refactor!
  def setup_contact
    if @domain.registrant == nil
      if params[:contact]
        base_attributes = {:user_id => current_user.id, :state_province_choice => 'S'}
        base_attributes[:email_address] = current_user.email if params[:contact][:email_address].blank?
        contact_attributes = base_attributes.merge(params[:contact])
        @contact = @domain.build_registrant(contact_attributes)
        unless @contact.save
          respond_to do |format|
            format.html { render :action => 'new' }
            format.json { render :json => {:name => @domain.name, :errors => @domain.errors.full_messages + @contact.errors.full_messages }, :status => 422 }
            format.xml { render :xml => {:name => @domain.name, :errors => @domain.errors.full_messages + @contact.errors.full_messages }, :status => 422 }
          end
          return false
        end
      elsif current_user.default_contact
        @domain.registrant = current_user.default_contact
      else
        @domain.errors.add(:base, "A registrant is required")
        respond_to do |format|
          format.html { render :action => 'new' }
          format.json { render :json => {:name => @domain.name, :errors => @domain.errors.full_messages }, :status => 422 }
          format.xml { render :xml => {:name => @domain.name, :errors => @domain.errors.full_messages }, :status => 422 }
        end
        return false
      end
    end
    logger.info "Contact setup complete"
    return true
  end

  # ...

end

Pretty messy, isn't it?

As preparatory work for the new API, we extracted the code from models and controllers into objects we called commands. We picked this name as we originally thought we would implement the command design pattern, but it never happened and the name never changed. Today, we would probably call them interactors or operations.

These objects implement the common logic for each use case in our system. We have one object for the login use case, one for the forgotten password, one for the signup, etc.

This abstraction alone simplifies a lot the maintenance.

All the code is predictably organized.

class DomainRegistrationsController < ApplicationController
  # ...

  def create
    @result = DomainRegisterCommand.execute(command_context,
                                            this_account,
                                            domain_params,
                                            contact_params)
    @domain = @result.domain

    if @result.successful? && @result.processing?
      create_registering
    elsif @result.successful?
      create_successful
    else
      create_failed
    end
  end
end
class DomainRegisterCommand
  include Command::Command

  def execute(account, domain_name, domains_params, contact_params)
    # ...
  end
  
  private
  def register(domain, extended_attributes, registrar_premium_price)
    # ...
  end
end

We didn't removed the complexity of this workflow, we just split it in smaller manageable objects. Now testing a feature is easier as we can focus only on the behavior of a single command object at the time.

In 2014 we also started to experiment our own version of service objects, that we formalized in 2016.

If you want to learn more about the evolution of our Rails architecture you can check out the slides and video of the talk Developing and maintaining a platform with Rails and Hanami that Simone presented at RailsConf 2016.

Tight Coupling

In Rails, when an action uses respond_to, the context for the web UI and the JSON API is the same: they share the same set of instance variables used for rendering.

That means you can't easily change one of these instance variables for a component (eg the UI), without affecting the behavior of the other component (eg API). Consider that a public JSON API is versioned, so you can't change it without breaking the entire ecosystem. We soon reached a state of code rigidity, where it was hard or impossible to modify the code for certain actions.

class RecordsController < ApplicationController
  # ...

  def create
    @result = RecordCreateCommand.execute(command_context, @domain.zone, param_record_type, record_params)
    @record = @result.data

    respond_to do |format|
      if @result.exists?
        format.html { redirect_to domain_records_url(@domain), notice: @result.warning }
        format.json { render json: { message: @result.warning }, status: 400 }
      elsif @result.successful?
        format.html { redirect_to domain_records_url(@domain) }
        format.json { render json: @record, status: 201 }
      else
        prepare_complex_records(@record, param_record_type)

        format.html { render_new }
        format.json { render json: { message: @result.error, errors: @record.errors }, status: 400 }
      end
    end
  end
end

The most obvious solution to reduce complexity was to remove responsibilities from these actions. That's why we decided to implement the new API as a standalone Hanami application mounted inside Rails. A change in the web UI (Rails) isn't reflected in the API (Hanami) and viceversa.

When we'll sunset, API v1 we can simplify that action to:

class RecordsController < ApplicationController
  # ...

  def create
    @result = RecordCreateCommand.execute(command_context, @domain.zone, param_record_type, record_params)
    @record = @result.data

    respond_to do |format|
      if @result.exists?
        redirect_to domain_records_url(@domain), notice: @result.warning
      elsif @result.successful?
        render json: @record, status: 201
      else
        prepare_complex_records(@record, param_record_type)
        render_new
      end
    end
  end
end

With the command objects already in place, it became much simpler to implement the new endpoints of the API. Each new endpoint is implemented with an action. It has the role of accepting the input and invoking the command related to the current use case.

Here's an API v2 action:

module Api::V2
  module Controllers::ZonesRecords
    class Create
      include Hanami::Action

      def call(params)
        @zone = DomainFinder.find!(
          params[:zone_id], authentication_context).zone
        @result = RecordCreateCommand.execute(command_context, @zone,
          params[:type], ZoneRecordParams.new(params, [:regions]))
        @record = @result.data

        if @result.successful? && !@result.exists
          render Serializers::RecordSerializer.new(@record), 201
        elsif @result.exists
          error(400, I18n.t("api.zone_records.already_exists"))
        else
          render Serializers::ErrorSerializer.new(@result.error, @record), 400
        end
      end
    end
  end
end

It's still complex, but it has a high cohesion. The purpose is to not eliminate the complexity of the model domain (cause you can't), but to make it maintanable.

Implicit Serializations

The way ActiveRecord serializes a model into JSON is straightforward: it dumps all the attributes. Again, this is easy, but as before, it comes with a cost: you can't change a database column without breaking the JSON API backwards compatibility.

To solve this problem we had to introduce a new layer of serializers. They have the role of translating ActiveRecord attributes into a stable set of key/value pairs for JSON.

In DNSimple we have the concept of contact, which is a person or a company who registers a domain. For a contact serialization we return the email information in the payload, but the corresponding database column is email_address. The ContactSerializer helps to resolve this mismatch with a stable public name: the email key.

module Api::V2
  module Serializers
    class ContactSerializer < ElementSerializer
      attributes :id, :account_id,
                 :label, :first_name, :last_name,
                 :job_title, :organization_name, :email,
                 :phone, :fax,
                 :address1, :address2, :city,
                 :state_province, :postal_code, :country,
                 :created_at, :updated_at

      def email
        object.email_address
      end
    end
  end
end

Even a value can be subject to accidental changes. At one point, Ruby changed the way that JSON.generate serializes timestamps. Without an explicit control on the timestamps format, a change in Ruby (or another library), can suddenly change the output of the API.

require 'json'

# Ruby 1.9
JSON.generate(time: Time.now)
  # => "{\"time\":\"Tue Jan 17 10:25:37 +0100 2017\"}"

# Ruby 2.0
JSON.generate(time: Time.now)
  # => "{\"time\":\"2017-01-17 10:25:37 +0100\"}"

Conclusion

While the API v1 implementation had a short "time to market", it hid unfortunate surprises down the road. In order to make it easier to evolve your systems, it is good to remember to create team guidelines for code structure, avoid tight coupling with the current framework, and take control of your code via explicitness of intents.

This post only scratches the surface of the code changes behind API v2, and the reasons why we went through this journey. You can read more about the changes in API v2 in the API v2 announcement post. I also encourage you to check out the talk The Great 3 Year API Redesign that Anthony presented at CodeMash 2017, where he went through the story of the DNSimple API v2, explaining some of the decisions we took over the last 3 years that shaped the development of the API as well as challenges we faced.