Testing AI Agent Outputs in Elixir with ExUnit

TL;DR: Search “testing LLM outputs Elixir” and every result is a library README or an announce thread — Tribunal, Ash’s evals tooling, a forum post asking how anyone does this. Nobody’s written the practitioner version: how you actually wire an eval framework into an ExUnit suite for a real agent loop, running in CI, without an API bill on every push. Here it is. I stub the LLM call boundary deterministically with Req.Test, add one :telemetry event so I can assert the order tool calls happen in — not just what the final string says — and split deterministic checks from LLM-judge checks so mix test stays free and fast while mix test --only eval catches drift. The one thing that doesn’t change: an agent that survives a deploy still needs tests that survive a model upgrade, and those are two different test suites.

The SERP for this is all library docs

I went looking for “how do I test an AI agent’s behavior in CI” and every first-page result is one of three things: a hex.pm package README, a launch thread on X, or a forum post from someone who clearly hasn’t solved it yet. Tribunal — George Guimarães’s LLM evaluation framework for Elixir, same author behind Arcana — has a genuinely good API. Ash’s evals tooling does too. But both sit at the “here’s the assertion macro” layer. Nobody’s shown the thing in between: a real agent loop, wired into a real ExUnit suite, testing the parts that actually break — tool-call sequencing, not just the words in the final answer.

That gap matters because the failure modes of an agent are structurally different from the failure modes of a function. A pure function that returns the wrong string is a bug you can assert_equal your way to. An agent that calls read_file twice, never calls write_file at all, and then produces a final answer that sounds plausible is a different class of bug entirely — and a naive test suite that only checks the last message will pass it every time.

I’ve written before about the 50-line agent loop itself and about running that loop on Oban so it survives a deploy mid-task. This post is the piece that sits between those two: how you actually test the thing before you trust it in either environment.

Two kinds of correctness, and they cost differently

Before any code, get the taxonomy straight, because it’s the difference between a test suite that runs on every commit for free and one that burns your API budget on every push.

Anthropic’s own engineering team lays out the same split in their guide to evaluating AI agents: code-based graders (fast, cheap, deterministic, but brittle to valid rewordings), model-based graders (an LLM judging another LLM’s output — flexible, but non-deterministic and priced per call), and human graders (the ceiling, and too slow for CI). Their framing is exactly the line I draw in an ExUnit suite:

• Deterministic assertions — did the agent call read_file before write_file? Does the response contain the string “30 days”? Did it stay under the token budget? These are pure functions over a transcript. No API call, no flakiness, run them on every commit.
• LLM-judge assertions — is this answer faithful to the source document? Did it hallucinate a policy that doesn’t exist? These require a second model call to grade the first model’s output, which means real latency and a real bill.

The mistake I see most often is treating both categories the same way — either skipping the judge-based checks entirely because “they’re flaky,” or running everything, judge calls included, on every single push, which is how a team ends up with a five-minute, five-dollar test suite and starts skipping it. Neither is right. The deterministic layer should be exhaustive and free. The judge layer should be small, curated, and gated to when it matters — which is a CI config decision I’ll get to below, not a testing-philosophy compromise.

Two seams the 50-line loop needs before it’s testable

The agent loop from the earlier post is a GenServer: init sends itself a :step message, handle_info(:step, ...) calls the LLM, dispatches a tool if one’s requested, and loops. It’s genuinely the whole pattern in 50 lines — but as written, it’s not testable, because it does two things a test can’t control: it hits the real Anthropic API, and it has no seam for observing what happened, only the final IO.puts.

Fixing that is two small changes, not a rewrite.

Seam one: let the HTTP call be stubbed. Req ships its own test-double module, Req.Test, which registers a named plug you route requests through instead of the network. That’s the deterministic half of the eval story — a scripted, repeatable multi-turn transcript with zero API cost and zero flakiness. The loop’s call_llm/3 needs an opts argument threaded through to Req.post/1:

defp call_llm(messages, tools, opts) do
  body = %{
    model: "claude-sonnet-4-20250514",
    max_tokens: 4096,
    messages: messages
  }

  body =
    if tools == [],
      do: body,
      else: Map.put(body, :tools, tools)

  plug_opt =
    if opts[:plug],
      do: [plug: opts[:plug]],
      else: []

  req_opts =
    [
      url: "https://api.anthropic.com/v1/messages",
      json: body,
      headers: [
        {"x-api-key", api_key()},
        {"anthropic-version", "2023-06-01"}
      ]
    ] ++ plug_opt

  {:ok, %{status: 200, body: resp}} =
    Req.post(req_opts)

  parse_response(resp)
end

opts flows down from run/3 through init/1 into state, and handle_info(:step, state) passes state.opts at the call site. In production, opts[:plug] is nil and plug_opt is [] — the request goes to the real API exactly as before. In a test, you pass plug: {Req.Test, Agent.LLM} and every call is intercepted, in order, by whatever you’ve scripted with Req.Test.expect/2 and Req.Test.expect/3.

Seam two: emit an event when a tool fires. The existing dispatch_tool/2 already prints a line for every call — IO.puts("🔧 #{name}(...)") — which tells you it knows the moment a tool executes. The fix is to stop routing that knowledge through stdout and start routing it through :telemetry, which is the standard way to observe what’s happening inside an OTP process without coupling the process to whoever’s watching:

defp dispatch_tool(name, input) do
  :telemetry.execute(
    [:agent, :tool_call],
    %{},
    %{name: name, input: input}
  )

  case name do
    "read_file" ->
      input["path"]
      |> File.read()
      |> then(fn
        {:ok, c} -> c
        {:error, r} -> "Error: #{r}"
      end)

    "list_files" ->
      (input["path"] || ".")
      |> File.ls!()
      |> Enum.join("\n")

    "write_file" ->
      File.write!(
        input["path"],
        input["content"]
      )

      "ok"

    _ ->
      "Unknown tool: #{name}"
  end
end

Neither change touches the loop’s actual behavior. That’s the point — a codebase you’d trust in production shouldn’t need a special “test mode” branch in its business logic, just an observable seam at the boundary.

Asserting on tool-call sequences, not just the final string

With both seams in place, a small Agent process can collect the ordered list of tool names an attached telemetry handler observes:

defmodule Agent.ToolTrace do
  def start do
    Agent.start_link(
      fn -> [] end,
      name: __MODULE__
    )
  end

  def attach do
    :telemetry.attach(
      "trace-#{inspect(self())}",
      [:agent, :tool_call],
      fn _event, _meas, meta, _cfg ->
        Agent.update(
          __MODULE__,
          &(&1 ++ [meta.name])
        )
      end,
      nil
    )
  end

  def calls, do: Agent.get(__MODULE__, & &1)
end

Then the test itself scripts a three-turn transcript with Req.Test.expect/2 — first turn proposes list_files, second proposes read_file, third returns the final answer — and asserts on the order, not just the last line:

defmodule Agent.LoopSequenceTest do
  use ExUnit.Case, async: true

  setup do
    Agent.ToolTrace.start()
    Agent.ToolTrace.attach()
    :ok
  end

  test "reads before it answers" do
    Req.Test.expect(Agent.LLM, fn conn ->
      Req.Test.json(conn, %{
        "stop_reason" => "tool_use",
        "content" => [
          %{
            "type" => "tool_use",
            "id" => "t1",
            "name" => "list_files",
            "input" => %{}
          }
        ]
      })
    end)

    Req.Test.expect(Agent.LLM, fn conn ->
      Req.Test.json(conn, %{
        "stop_reason" => "tool_use",
        "content" => [
          %{
            "type" => "tool_use",
            "id" => "t2",
            "name" => "read_file",
            "input" => %{"path" => "mix.exs"}
          }
        ]
      })
    end)

    Req.Test.expect(Agent.LLM, fn conn ->
      Req.Test.json(conn, %{
        "stop_reason" => "end_turn",
        "content" => [
          %{
            "type" => "text",
            "text" => "Elixir ~> 1.18"
          }
        ]
      })
    end)

    Agent.Loop.run(
      "What Elixir version is this?",
      [],
      plug: {Req.Test, Agent.LLM}
    )

    Process.sleep(20)

    assert Agent.ToolTrace.calls() ==
             ["list_files", "read_file"]
  end
end

This is the assertion a string-matching test can’t make. An agent that calls list_files three times before giving up and calling read_file still produces a final answer that could satisfy assert_contains ans, "1.18" — and the test would pass while the agent burned three extra API calls and 90 seconds figuring out what a one-shot read_file should have told it immediately. Sequence assertions catch the thing that actually costs money and time in production: the agent doing the right thing the slow, expensive way.

Hallucination and faithfulness checks with Tribunal

Sequencing is deterministic; content grounding usually isn’t, which is where a judge-based library earns its cost. Tribunal ships an ExUnit case template with two tiers of assertion — deterministic string checks that run free, and LLM-as-judge checks that call out to a model:

def deps do
  [
    {:tribunal, "~> 1.3"},
    {:req_llm, "~> 1.2", only: :test},
    {:alike, "~> 0.1", only: :test}
  ]
end

The deterministic tier costs nothing and runs on every commit:

defmodule Agent.AnswerShapeTest do
  use ExUnit.Case
  use Tribunal.EvalCase

  test "answers cite the doc, don't invent" do
    {:ok, ans} =
      Agent.Loop.run(
        "Summarize our refund policy",
        docs_tools()
      )

    assert_contains ans, "return"
    refute_contains ans, "lifetime warranty"
  end
end

The judge tier costs a model call, tags itself :eval, and is where faithfulness and hallucination checks actually live:

defmodule Agent.FaithfulnessEvalTest do
  use ExUnit.Case
  use Tribunal.EvalCase

  @moduletag :eval
  @docs File.read!(
          "test/fixtures/refund_policy.md"
        )

  test "grounded in the doc, not guessed" do
    {:ok, ans} =
      Agent.Loop.run(
        "What's our refund window?",
        docs_tools()
      )

    assert_faithful ans, context: @docs

    refute_hallucination ans, context: @docs
  end
end

assert_faithful/2 and refute_hallucination/2 route to a judge model via the optional req_llm dependency — that’s the API cost, and it’s the reason this test is tagged, not left to run alongside the free suite.

Running evals in CI without burning tokens on every push

That :eval tag is the whole CI story. test/test_helper.exs excludes it by default:

ExUnit.start()
ExUnit.configure(exclude: [:eval])

Every mix test on every push runs the deterministic layer — sequencing, string checks, token-budget and timeout guardrails — for free, in seconds, exactly like any other test suite. The judge-based layer only runs on demand:

mix test --only eval

Wire that into a scheduled or pre-merge CI job instead of every-push, and you’ve reproduced the shape Anthropic recommends in the same evals guide cited above: “automated evals are especially useful pre-launch and in CI/CD, running on each agent change and model upgrade as the first line of defense against quality problems” — pre-launch and per-model-upgrade, not per-commit. Their guidance to start with “20-50 simple tasks drawn from real failures” is the right target for the judge tier too: it doesn’t need to be exhaustive, it needs to cover the ways your agent has actually gone wrong before.

The competitive argument for having this at all is worth stating plainly, because it’s the one that gets this line item approved: per Anthropic, “teams without evals face weeks of testing while competitors with evals can quickly determine the model’s strengths, tune their prompts, and upgrade in days” when a new model version ships. That’s not a testing-hygiene argument, it’s a time-to-ship argument.

Evals aren’t a substitute for judgment — even Anthropic’s own postmortem shows it

One honest caveat, because a tutorial that oversells the tool it’s teaching is worse than no tutorial. Anthropic’s own postmortem on a Claude Code quality regression is candid that a system-prompt change shipped despite “multiple weeks of internal testing and no regressions in the set of evaluations we ran” — the eval suite that existed simply didn’t cover the failure mode that showed up in production. Their fix wasn’t to abandon evals, it was to broaden them: they committed to running “a broad suite of per-model evals for every system prompt change,” plus soak periods and gradual rollouts, specifically because their existing suite was too narrow. The lesson transfers directly: an eval suite is only as good as the failure modes you’ve actually written a test for, and the failure modes you haven’t seen yet are exactly the ones that ship anyway. Keep adding cases from real incidents — which is also why a proper AI agent postmortem should always end with “add this transcript to the eval fixtures,” not just “add a monitor.”

The reliability tie-in: surviving a deploy isn’t the same as surviving a model upgrade

This is the connection that made me want to write this post instead of just linking to Tribunal’s README. Running an agent loop on Oban buys you durability against infrastructure failure — the process can die mid-run and the conversation state in Postgres means it picks back up. That’s a real reliability win, and it’s orthogonal to a completely different kind of failure: the agent’s behavior changing out from under you because the model underneath it changed.

A model upgrade isn’t a version bump you can changelog-skim past. The new model can reason differently, follow the same prompt more or less literally, chain tool calls in a different order, or refuse an input the old model accepted without complaint. None of that shows up as a deploy failure — Oban will happily persist and retry a conversation that’s now subtly wrong. The only thing that catches it is a suite that asserts on behavior, not uptime: the sequencing tests from the section above re-run against the new model and either pass or tell you exactly which step changed, and the faithfulness/hallucination tests re-run and tell you whether the new model’s answers are still grounded in the same documents. An agent that survives a deploy and an agent that survives a model upgrade are protected by two different test suites, and most teams that have built the first one haven’t built the second.

Where this fits in an Elixir AI stack

None of this requires an AI-specific test framework at its foundation — it’s ExUnit, :telemetry, and Req.Test, three things that are either in the standard library or already in your mix.exs if you’re calling an LLM API from Elixir at all. Tribunal adds the judge-based assertions on top, and it’s worth adopting for that layer specifically rather than hand-rolling your own LLM-as-judge harness. This is also, I think, a decent argument for why the BEAM is a good home for an AI startup’s backend in the first place: :telemetry being a first-class, ecosystem-standard observability primitive rather than a bolted-on logging library is exactly the kind of thing that makes testing an agent’s behavior — not just its uptime — a normal part of the existing toolchain instead of a new category of infrastructure you have to buy or build.

If you’re standing up an agent that’s going to run unattended in production — durable runtime, real tool access, a model someone else controls the version of — the test suite is not optional scaffolding you add later. It’s the thing that tells you, in CI, before a customer does, that the last model upgrade quietly changed what your agent does with a customer’s data. If you want a second set of eyes on where that suite should sit relative to the rest of your AI infrastructure, that’s a conversation worth having.