I wrote some code this week that reinforced the power of protocols as a tool for software design. The term “protocol” can mean many things in the world of software. Let me clarify that I’m using protocol to mean the mechanism used by some languages (Elixir, Clojure, etc) to achieve polymorphism. Used properly, protocols allow you the provide users of your code with a set of standard behavior as well as a clear contract for implementing that behavior on standard or custom types.

In this post I’ll provide an introduction to protocols and then describe several uses of protocols that lead to extensible design. The examples in this post are written in Elixir but should be equally useful in other languages (after all, Elixir credits Clojure as inspiration for its implementation of protocols).

An Introduction to Protocols

Protocols are a mechanism for achieving polymorphism. To say it more plainly, protocols let you call a single function (or set of functions) while allowing the subject of the function to dictate the way in which the function is implemented. I know, I know, that’s still confusing. Let’s be more concrete and also provide an example.

A protocol feels very similar to an interface in languages like Java. It consists of (at least) two pieces. First, there is the definition of the protocol itself. This is essentially a template of functions that must be implemented for any type that the protocol can act on. Suppose we’d like to introduce a protocol that determines if a collection is empty. Our protocol could look something like this:

defprotocol Empty do
  def empty?(collection)
end

Our protocol has a single function called empty?. For us to actually use this protocol, we must provide some implementations. Let’s do so for List and Map.

defimpl Empty, for: List do
  def empty?([]), do: true
  def empty?(_), do: false
end

defimpl Empty, for: Map do
  def empty?(map) do
    case Map.keys(map) do
      [] -> true
      _ -> false
    end
  end
end

With these two implementations in place, we can now test to see if Maps and Lists are empty.

Empty.empty?([1, 2, 3])
# => false
Empty.empty?([])
# => true

Empty.empty?(%{foo: "bar"})
# => false
Empty.empty?(%{})
# => true

This isn’t very exciting, but it gets more interesting when we add implementations for custom structs in our application code. Suppose we’ve implemented a RedBlackTree (because reasons).

defmodule RedBlackTree do
  defstruct [:nodes]

  def size(rb_tree) do
    # an implementation goes here
  end
end

Now, we can implement the Empty protocol for our custom Struct.

defimpl Empty, for: RedBlackTree do
  def empty?(rb_tree) do
    RedBlackTree.size(rb_tree) == 0
  end
end

We can check do see if our RedBlackTree is empty in the same way we check Lists and Maps.

Empty.empty?(%RedBlackTree{...})

So What?

Why does this matter? How can we use it to write better libraries and application code? Story time.

This week, I was working to prep my library Scrivener for the upcoming major release of Ecto. A pull request came in that was unrelated to the work I was doing – someone was interested in extending Scrivener to paginate Lists as well as Ecto queries. My goal with Scrivener has been to keep the library small and focused. This idea had me both excited and concerned. I wanted to provide a library where I could focus on the functionality I needed while allowing individuals in the community to easily extend the library for their own needs. Protocols to the rescue.

Suppose my pagination code in Scrivener originally looked something like this:

defmodule Scrivener do
  @spec paginate(Ecto.Query.t, Config.t) :: Page.t
  def paginate(query, config) do
    %Scrivener.Page{
      entries: find_entries(query, config),
      total_pages: find_total_pages(query, config),
      ...
    }
  end
end

As you can see from the function @spec, this takes an Ecto.Query and a Config and returns a Page. That Page struct contains the page’s entries as well as information about the total number of pages, the current page number, etc. This works. It’s great. But when the new PR came in focused on adding pagination for Lists, I was concerned. Will I need to add some kind of pattern matching around the first argument? Will I be stuck maintaining pagination logic for every type of database and collection under the sun? And then it hit me: protocols. I made a very simple change.

defprotocol Scrivener.Paginater do
  @spec paginate(any, Config.t) :: Page.t
  def paginate(pageable, config)
end

defmodule Scrivener do
  @spec paginate(any, Config.t) :: Page.t
  def paginate(pageable, config) do
    Scrivener.Paginater.paginate(pageable, config)
  end
end

defimpl Scrivener.Paginater, for: Ecto.Query do
  @spec paginate(Ecto.Query.t, Config.t) :: Page.t
  def paginate(query, config) do
    %Scrivener.Page{
      entries: find_entries(query, config),
      total_pages: find_total_pages(query, config),
      ...
    }
  end
end

This single change means my library is now massively easier to extend while giving up none of the existing functionality. The individual who asked about adding List pagination was now free to do the work without needing to change Scrivener itself and could release the new functionality as a companion library. You could imagine the code looking something like this:

defimpl Scrivener.Paginater, for: List do
  @spec paginate([any], Config.t) :: Page.t
  def paginate(list, config) do
    %Scrivener.Page{
      entries: find_entries(list, config),
      total_pages: find_total_pages(list, config),
      ...
    }
  end
end

After including this companion library, users of Scrivener can interact with it via the exact same API, but passing in a list instead of an Ecto query. Very powerful indeed.

A Few Other Examples

Two other great examples of using protocols for extensible APIs are the Poison JSON library and the built-in Enum module.

Poison implements JSON encoding via a protocol called Poison.Encoder. The library ships with implementations for all applicable standard types (Maps, Lists, etc) and allows you to easily implement your own encoders for custom types.

The Enum module in the Elixir standard library is another great example. The functions in the Enum module are implemented in terms of the Enumerable protocol. This means that if you implement the Enumerable protocol for your custom collection, you get all the functionality in Enum for free.

So They’re Just Interfaces?

Protocols are very similar to interfaces with one extremely important distinction. An interface author must rely on the consumer of their code to implement the interface for their domain objects. A protocol author can implement the protocol for you on any existing standard or custom types as well as allowing you to implement the protocol for types that you deem applicable.

This means that I was able to introduce a protocol into the Scrivener codebase without having to ask the Ecto team to implement the protocol for me. Fundamentally, protocols allow safe extension of existing code even if it is not owned by the author of the protocol. Interfaces, on the other hand, force the users of your API to implement functionality directly on their domain objects to achieve polymorphism. It is hard to overestimate the impact of this subtle distinction.

Consider Protocols Judiciously

I’d urge you to consider protocols as mechanism for providing extensibility in your libraries and application code. However, it’s important to not overuse them. The temptation to generalize early with protocols is ever-present and problematic. Elixir itself was a victim of this, initially providing a protocol for “dictionary like objects” (Maps, Lists, Keywords) and eventually removing the protocol entirely and saying “just use Maps”.

Use protocols for extensibility, not for making data operations generic. When working with a concrete type, treat the data as that type.