Pagination with Phoenix & Ecto

I've been working on a web application built in Elixir. I'm using Phoenix as the web framework and Ecto to talk to my database. As the amount of data in the application grew, I needed to paginate some of the views. I wasn't able to find an existing pagination solution for these tools so I ended up building my own. This post will discuss what I built.

Goal

Once we're done, we should be able to paginate any Ecto query using parameters provided by a Phoenix controller action. I'm going to assume you're familiar with building composable Ecto queries. If you aren't, read this post.

Let's assume we have a post model.

defmodule Post do  
  use Ecto.Model

  schema "posts" do
    field :name, :string
    field :content, :string
    field :published, :boolean
  end

  def published(query) do
    query |> where([p], p.published == true)
  end

  def order_by_name(query) do
    from p in query,
    order_by: [asc: p.name]
  end
end  

Let's also assume we have an Ecto repository.

defmodule Repo do  
  use Ecto.Repo, ...
end  

Our API for the pagination should accept an Ecto query and a map of options representing the params from a Phoenix controller. The options may provide the page key (defaulting to 1) and the page_size key (defaulting to 10). When we're done, we should be able to write the following code.

defmodule PostController do  
  use Phoenix.Controller

  def index(conn, params) do
    paginator = Post
    |> Post.published
    |> Post.order_by_name
    |> Paginator.new(params)

    render conn, :index,
      posts: paginator.entries,
      page_number: paginator.page_number,
      page_size: paginator.page_size,
      total_pages: paginator.total_pages
  end
end  

Building the Paginator

Here's a skeleton module to kick things off.

defmodule Paginator do  
  defstruct [:entries, :page_number, :page_size, :total_pages]

  def new(query, params) do
    %Paginator{
      entries: [],
      page_number: 0,
      page_size: 0,
      total_pages: 0
    }
  end
end  

Ok, so we've got some dummy data and a struct that we're going to return. Let's start by calculating the current page of entries. We'll do this by adding a limit and offset to our Ecto query based on the page size and page number.

defmodule Paginator do  
  defstruct [:entries, :page_number, :page_size, :total_pages]

  def new(query, params) do
    page_number = params |> Dict.get("page", 1) |> to_int
    page_size = params |> Dict.get("page_size", 10) |> to_int

    %Paginator{
      entries: entries(query, page_number, page_size),
      page_number: page_number,
      page_size: page_size,
      total_pages: 0
    }
  end

  defp entries(query, page_number, page_size) do
    offset = page_size * (page_number - 1)

    query
    |> limit([_], ^page_size)
    |> offset([_], ^offset)
    |> Repo.all
  end

  defp to_int(i) when is_integer(i), do: i
  defp to_int(s) when is_binary(s) do
    case Integer.parse(s) do
      {i, _} -> i
      :error -> :error
    end
  end
end  

To build the entries, we determing the offset by multiplying the page size by the current page number. We then limit the number of results returned to the page size. Finally, we fetch the results using our Ecto Repo. We also added a helper function to_int so that we can deal with string values provided by the params.

Now, all that's left is calculating the total_pages. Let's do it.

defmodule Paginator do  
  defstruct [:entries, :page_number, :page_size, :total_pages]

  def new(query, params) do
    page_number = params |> Dict.get("page", 1) |> to_int
    page_size = params |> Dict.get("page_size", 10) |> to_int

    %Paginator{
      entries: entries(query, page_number, page_size),
      page_number: page_number,
      page_size: page_size,
      total_pages: total_pages(query, page_size)
    }
  end

  defp ceiling(float) do
    t = trunc(float)

    case float - t do
      neg when neg < 0 ->
        t
      pos when pos > 0 ->
        t + 1
      _ -> t
    end
  end

  defp entries(query, page_number, page_size) do
    offset = page_size * (page_number - 1)

    query
    |> limit([_], ^page_size)
    |> offset([_], ^offset)
    |> Repo.all
  end

  defp to_int(i) when is_integer(i), do: i
  defp to_int(s) when is_binary(s) do
    case Integer.parse(s) do
      {i, _} -> i
      :error -> :error
    end
  end

  defp total_pages(query, page_size) do
    count = query
    |> exclude(:order_by)
    |> exclude(:preload)
    |> exclude(:select)
    |> select([e], count(e.id))
    |> Repo.one

    ceiling(count / page_size)
  end
end  

We need to do a few things to calcuate our total pages. First, we exclude any non-countable components of the query (including order_by, preload, select, etc). We then add a select statement to grab the count. Finally, we divide the count by the page_size and calling ceiling on the result.

And that's it! We can now paginate our Ecto queries.

Wrap Up

I hope this deep-dive was helpful in understanding pagination in Ecto and Phoenix applications. If you're interested in using this in your application, I've released a more robust hex package for Ecto pagination called Scrivener.