TDD With Claude Code in Elixir: What Holds Up

TL;DR: Skip the red-green-refactor tutorial — that content is saturated and generic. The Elixir-specific question is narrower and more useful: which parts of TDD-with-an-agent actually hold up in an ExUnit/Phoenix codebase, and which parts quietly break. The answer: agents follow a failing test far more reliably than they follow a paragraph of prose instructions, which makes test-enforced discipline — mix precommit, tagged async: true conventions, idempotent Ecto.Multi/Oban contracts — the real moat, not the prompt. Where it falls apart: LiveView test flakiness, fixture conventions Claude can’t infer from a schema alone, and the seductive trap of writing a test that passes for the wrong reason. This is what actually happened writing tests alongside an agent in a live Phoenix repo, not a hypothetical.

Skip the preamble — here’s what’s different in Elixir

Every “TDD with AI” post on the internet is language-agnostic, which means it’s written for Python or TypeScript with the specifics filed off. Elixir has real specifics: OTP’s process model means a badly-scoped test can leak state across processes in ways a synchronous language never would; Ecto’s sandbox mode means database tests can run concurrently by default, which is a gift and a trap; and Phoenix’s LiveView tests exercise a genuinely different execution model (a supervised process exchanging events) than a typical HTTP request/response test. None of that shows up in a generic TDD guide, and all of it changes what “write the test first” means in practice.

The frame I keep coming back to, working in this repo day to day: an agent obeys a test far more reliably than it obeys a sentence in CLAUDE.md. I’ve written about the limits of prose instructions before — what actually earns a line in CLAUDE.md after 50 commits is a short list, because most of what you’d want to tell an agent turns out to compress much better into an executable check than a rule. TDD isn’t a nice-to-have discipline layered on top of agent-driven development. It’s the enforcement mechanism that makes agent-driven development safe to run unattended. Everything below is a specific instance of that claim, tested against this site’s own Elixir/Phoenix codebase.

ExUnit async vs DataCase/ConnCase: the default Claude gets wrong

The single most common ExUnit mistake I see an agent make, unprompted, is marking a new test async: true without checking whether the module actually supports it. ExUnit.Case, async: true is fine for a pure module with no shared state. SublimeCodingWeb.ConnCase and SublimeCoding.DataCase — the two base cases that touch the database or the Ecto sandbox — are a different story, and whether they’re safe to run concurrently depends on the checkout mode the case template sets up, not on the test file itself.

Grep this repo’s own test suite and the split is visible immediately:

test/sublime_coding_web/plugs/security_headers_test.exs:
  use ExUnit.Case, async: true

test/sublime_coding_web/seo_health_test.exs:
  use SublimeCodingWeb.ConnCase
  (no async: true)

test/sublime_coding_web/live/vciso_cost_live_test.exs:
  use SublimeCodingWeb.ConnCase, async: true

Three tests, three different answers to “is this safe concurrently,” and the difference isn’t visible from the test body — it’s a property of what the test touches (plug-level logic with no shared connection vs. content lookups against a compiled resource vs. a LiveView test against the sandbox). An agent asked to “add a test for this” will pattern-match on the nearest example file and copy its async setting whether or not it’s the right call for the new test’s actual dependencies. The fix isn’t a longer explanation in the prompt — it’s letting mix test be the check. If a test that shouldn’t be concurrent gets marked async, you get an intermittent, hard-to-reproduce failure under --seed variation, which is a worse failure mode than an outright wrong test, because it looks like flakiness instead of a mistake. Catch it by running the suite more than once locally before trusting an agent’s addition, not by writing a longer rule about it.

mix precommit is the actual instruction Claude follows

This project’s mix.exs defines a precommit alias:

precommit: [
  "compile --warnings-as-errors",
  "deps.unlock --check-unused",
  "format",
  "test"
]

That’s the single most load-bearing line in the project for agent-driven work, and it has nothing to do with prompting. --warnings-as-errors turns an unused variable, an ambiguous pattern match, or a deprecated function call into a build failure instead of a scrollback warning an agent (or a tired human) skims past. Combined with test, an agent can’t claim a change is “done” without the compiler and the full suite both agreeing — and per this repo’s own working agreement, “verify before claiming done” means literally running mix compile && mix test, not asserting it should work.

The practical shift this produces: instead of writing “make sure you don’t break existing tests” into a prompt (which an agent will nod along with and sometimes still violate), you make mix precommit the actual gate — a CI step, a pre-commit hook, or just the command you run before you’ll look at a diff. Prose is a suggestion; a red exit code is not. This is the same principle this site’s own build enforces on itself in an unrelated domain: the tag taxonomy and SERP <title> length aren’t governed by an editorial style guide anyone has to remember to reread — they’re pinned by test/sublime_coding_web/seo_health_test.exs, which asserts merged-out tag slugs are gone from the active taxonomy and that every post’s rendered <title> fits in 70 characters. An agent drafting a new post doesn’t need to have internalized the taxonomy; it needs the test to fail loudly if it picks a dead tag. Tests are how you hand an agent a constraint it can’t talk itself out of.

Ecto.Multi and Oban jobs are TDD-shaped problems whether you plan it that way or not

This is the part that surprised me least, in retrospect, but that most TDD content entirely misses: Ecto.Multi and Oban jobs are already structured as testable contracts, because both are built around the same idea — a named, composable step that either fully applies or doesn’t, with an explicit success/failure shape you can assert against without touching the database twice by accident.

An Ecto.Multi pipeline is naturally test-first-friendly because each named step is independently assertable:

Multi.new()
|> Multi.insert(:account, changeset)
|> Multi.run(:ledger, fn repo, %{account: a} ->
  ledger_entry(repo, a)
end)
|> Repo.transaction()

You can write the test for the :ledger step’s failure branch before the implementation exists — assert that a failed ledger insert rolls back the account insert too — and that test is exactly as valuable written first as written after, because the behavior it’s pinning (atomicity across two writes) is invisible in a code read and only provable by exercising the rollback path.

Oban jobs raise the same shape one level up, and it’s the exact idempotency argument I’ve made about running Oban as a durable AI agent runtime: a job that retries on a transient failure but isn’t idempotent will double-charge, double-send, or double-write on its second attempt. The test that matters isn’t “does the job succeed” — it’s “does running perform/1 twice with the same args produce the same end state as running it once.” That’s a test you can and should write before the job body, because it’s the contract the retry mechanism depends on, and an agent generating job code from a prompt alone has no way to know retries are even in play unless the test says so explicitly. I’ve seen an agent write a perfectly reasonable-looking Oban worker that increments a counter on every perform/1 call with zero acknowledgment that Oban will call it more than once for the same job under a transient failure. A failing idempotency test catches that in one line; a prose reminder to “handle retries” gets forgotten by the third worker of the session.

Where TDD-with-agents actually breaks down

I don’t want to oversell this. Three places where the practice genuinely struggles, honestly:

LiveView test flakiness. Phoenix.LiveViewTest exercises real process message-passing — render_click, render_change, and friends dispatch through the LiveView process and wait on its reply. That’s mostly reliable, but a test that asserts on DOM state immediately after an action that triggers an async assign (a Task, a PubSub broadcast, a debounced form) can pass locally and flake in CI depending on scheduler timing. An agent writing a new LiveView test from scratch has no signal that a given interaction is async under the hood unless the LiveView module’s own code makes that obvious, and it will happily write an assertion that’s correct 95% of the time and a source of exactly the kind of intermittent CI failure that erodes trust in the whole suite. The fix is procedural, not architectural: mix test --repeat-until-failure locally, or just running new LiveView tests several times before trusting them into a commit — the same “don’t take the first green on faith” discipline you’d want from a human, and one an agent won’t apply to its own work without being told, every time, because it isn’t a rule that generalizes from a test file it can imitate.

Fixture and ExMachina conventions Claude can’t infer. Given a schema, an agent will write a plausible-looking fixture — but “plausible” and “matches this project’s actual factory conventions” (required associations, a specific sequence/2 pattern for uniqueness, which fields get sensible defaults vs. which ones must be supplied by the caller) are different things, and the difference only lives in the existing factory file, not in the schema. If there’s an existing ExMachina factory module, point the agent at it explicitly before asking for a new test — “extend the pattern in test/support/factory.ex” produces something reusable; “write a fixture for this schema” produces a one-off that duplicates logic the next test writer (human or agent) won’t find.

Tests that pass for the wrong reason. This is the general failure mode underneath the two specific ones above, and it’s worth naming on its own: an agent under pressure to make a red test green will sometimes narrow the assertion instead of fixing the implementation — asserting assert result instead of asserting the actual expected value, or stubbing out the exact input a broken function happens to handle. I’ve written more broadly about this pattern and the other bugs coding agents introduce reliably enough to check for by habit. The countermeasure in an Elixir codebase specifically: prefer pattern-matching assertions (assert {:ok, %Account{balance: 100}} = result) over boolean ones, because a pattern match fails loudly and specifically if the shape is wrong, where a bare assert result will happily pass against almost anything truthy.

The actual workflow, stated plainly

Concretely, in this repo: I ask for the failing test first, specifying the exact base case (DataCase vs ConnCase vs plain ExUnit.Case) and whether it needs async: true, rather than leaving that inference to pattern-matching against the nearest file. For anything touching Ecto.Multi or an Oban worker, the idempotency test is written before the implementation, not after, because it’s the one an agent won’t think to add unprompted. And nothing gets called done without mix precommit passing clean — not “the new test passes,” the whole suite, with warnings as errors. None of that is exotic. It’s the same discipline any senior Elixir engineer would apply to a junior’s PR; the only change is that the “junior” writes code at agent speed, which means the enforcement has to run automatically or it silently stops happening by the fourth or fifth file of a session.

The broader point, and it applies past Elixir: this is the same reason technical due diligence on an AI-native codebase increasingly looks for test coverage on the paths that actually carry risk, not a percentage number — a codebase where the enforcement lives in the test suite survives an agent-heavy contributor list; one where it lives in a style guide doesn’t. If you’re deciding whether Elixir is even the right backend for an AI-heavy build in the first place, I’ve made the fuller case for Elixir on an AI startup backend separately — this post assumes you’re already there and wants to get the day-to-day discipline right.

If you’re standing up (or auditing) an Elixir codebase that’s going to take a lot of agent-authored commits, this is exactly the kind of workflow review I do as a fractional CTO engagement — let’s talk about what that looks like.