It is common for test “classifications” to have a plethora of definitions. Sanity tests are no different. Wikipedia says the following on the subject:

the sanity test […] determines whether it is possible and reasonable to proceed with further testing.

This implies that a sanity test is some sort of “pre-test” to determine if further testing even makes sense.

I, however, think of sanity testing as breaking the fourth wall of the codebase. I believe sanity testing should test assumptions about the codebase itself rather than the behavior of the code.

I’ve found sanity testing to be extremely useful both in saving time by allowing future contributors to avoid common pitfalls as well as codifying codebase conventions. Let’s make the concept of sanity tests more explicit with a few examples from an Elixir codebase I’ve been working on recently.

Note: my tests use the ex_spec library that adds a simple BDD-like syntax to ExUnit.

Freezing Migrations

Many applications use libraries to interact with a database. These tools often provide facilities for codifying your database schema and for modifying it as time passes, a concept generally called “database migrations” or “migrations” for short. In Elixir, the Ecto library provides a migration feature similar to Rails’ ActiveRecord migrations. Because these files live in your codebase and version control system, often several members of the team can be modifying a migration at the same time. That’s fine as long as the migration has not yet been run in your production environment. If it has been run in production, however, we would like to ensure that the migration in our codebase never changes in the future. If we want to make changes to what that migration does we’ll need to write a brand new migration to perform those changes.

We can write a sanity test to enforce this rule.

defmodule MyApp.SanityTest do
  use ExSpec

  @migration_hashes %{
    "00100_release_1_00.exs"  => "de1a663896925fb53abc0341eb5ca414",
    "00200_release_2_00.exs"  => "823cc2738e1e43f2dbfe3ffc4b26fb8f",
    "00300_release_3_00.exs"  => "b417b004600b4dd3518f5708aebf6262",
    "00400_release_4_00.exs"  => "51b1ec3944d4875b217a7167d9cd6571",
    "00500_release_5_00.exs"  => "91e0173099631350ec4c40624b5bc862",
  }

  describe "frozen migrations" do
    it "will not change after being deployed" do
      glob = Path.join([
        __DIR__,
        "..",
        "..",
        "priv",
        "repo",
        "migrations",
        "*"
      ])

      Enum.each(Path.wildcard(glob), fn file_name ->
        file = Path.basename(file_name)
        expected_hash = @migration_hashes[file]

        if expected_hash do
          hash = file_name |> File.read! |> md5

          assert hash == expected_hash
        end
      end)
    end
  end

  defp md5(s) do
    :crypto.hash(:md5, s)
    |> :erlang.binary_to_list
    |> Enum.map(&(:io_lib.format("~2.16.0b", [&1])))
    |> List.flatten
    |> :erlang.list_to_bitstring
   end
end

This test iterates over all migrations within priv/repo/migrations and, if the filename is specified in @migration_hashes, it verifies that the md5 of the file contents is the expected value. When a new release of the project is created, any migrations that are released will be added to the @migration_hashes map. If a team member accidentally modifies a migration that has already been deployed, the test will fail and they’ll be gently reminded to create a new migration.

Valid Test Files

More than once I’ve accidentally created a test file in my Elixir application’s test directory that isn’t named properly. I’ll either forget to end my test filename with _test.exs or I’ll remember the correct filename but accidentally use the .ex extension rather than .exs. These errors are particularly painful because they manifest themselves by simply not running your tests in that file. This is a prime candidate for a sanity test. Let’s write one.

defmodule MyApp.SanityTest do
  use ExSpec

  describe "test files" do
    it "checks that all test files are named properly" do
      exclusions = ~w(
        test_helper.exs
      )

      test_glob = Path.join([__DIR__, "..", "**", "*"])

      bad_files = test_glob
      |> Path.wildcard
      |> Enum.map(&Path.expand/1)
      |> Enum.reject(&File.dir?(&1))
      |> Enum.reject(&String.ends_with?(&1, "_test.exs"))
      |> Enum.reject(fn path ->
        Enum.any?(exclusions, &matches?(path, &1))
      end)

      assert bad_files == []
    end
  end

  defp matches?(string, pattern) when is_binary(pattern) do
    String.ends_with?(string, pattern)
  end

  defp matches?(string, pattern) do
    String.match?(string, pattern)
  end
end

Here we find all of our test files and then exclude any that are properly named (ending in _test.exs). We also exclude any files that are explicitly named in our exclusions list. Our exclusions can either be string literals that (specifying explicitly the end of the excluded file’s name) or regexs that can match any number of files. This allows us to exclude files from our test like test_helper.exs which should not follow the convention of ending in _test.exs. If we accidentally add a test file with a malformed name this sanity will fail, saving us countless hours and lots of frustration.

Use Them!

I hope these two examples convey the usefulness of sanity tests as both tools for saving time as well as conveying patterns and idioms. Next time you think to yourself “I can’t believe I did that again”, consider writing a sanity test to stop yourself from making the same mistake in the future.

Freezing Migrations#

Valid Test Files#

Use Them!#

Freezing Migrations

Valid Test Files

Use Them!