Oban as a Durable AI Agent Runtime in Elixir

TL;DR: If you’re building an AI agent in Elixir — a ReAct/tool-call loop that thinks, calls a tool, observes, repeats — you do not need a bespoke agent framework or a hand-rolled GenServer to drive it. You need a durable job queue, and you already have the best one in the ecosystem: Oban. It’s Postgres-backed, idempotent by construction, retry-with-backoff out of the box, and its jobs survive a node restart because the state lives in a table, not in process memory. Model each agent step as one Oban job that enqueues the next, and your agent run keeps its place through an LLM 429, a flaky tool call, or a deploy that rolls the node mid-loop. The honest tradeoff: Oban is the wrong tool for sub-second interactive streaming (that’s LiveView’s job) and for genuinely ephemeral single-shot calls (just call the API). It shines precisely when a run is long, multi-step, costs real money per step, and absolutely must not silently die.

The agent that lost forty steps to a deploy

Here’s the scenario that made me write this. You’ve got an agent doing something genuinely useful — say, working through a backlog of support tickets, or refactoring a module across a dozen files, or researching a topic across thirty web fetches. It’s a loop: the model proposes a tool call, your code runs the tool, you feed the result back, the model proposes the next call. Twenty, forty, sixty iterations deep. Each model turn costs money. Each tool call has side effects.

You drive it with the obvious thing: a GenServer holding the conversation history and the loop counter in its state, recursing on handle_info.

Then you ship a deploy.

The release rolls the node. The supervisor shuts down your GenServer. Its state — forty steps of accumulated context, the half-finished plan, the tool results you already paid for — evaporates with the process. There’s no terminate/2 heroics that save you here, because the work isn’t “flush a buffer,” it’s “an entire stateful computation lives only in this process’s heap.” When the node comes back, the agent is just gone. No error. No resume. You find out because a customer asks why their ticket never got answered.

I’ve watched a team burn an afternoon adding “checkpointing” to exactly this design — serializing the GenServer state to a table every N steps, reloading on boot, reconciling partial writes. By the time they finished, they had rebuilt a worse version of a durable job queue. They’d named it AgentRunner. They could have named it Oban.

An agent loop is just a job queue you haven’t named yet

Strip an agent down to its control flow and look at it honestly:

• Think — call the LLM, get back a tool call (or a final answer).
• Act — execute the tool. This can fail, time out, or partially succeed.
• Observe — append the result to the running context.
• Repeat — until the model says “done” or you hit a step budget.

Now look at the failure modes that actually bite in production:

• The LLM returns a 429 and you need to back off and try again — without losing the run.
• A tool call fails transiently (network blip, downstream 503) and should be retried, but a different failure (malformed args, a 400) should not be retried forever.
• A step partially succeeds — the tool wrote a row, then the process died before recording that it did — and a naive retry double-writes or double-charges.
• The node restarts mid-run and the whole thing must resume from where it left off.

Every one of those is a solved problem in a durable job queue. Retry-with-backoff: solved. Distinguishing “retry this” from “give up on this”: solved. Idempotency so a retried unit doesn’t double-act: solved. Surviving a restart because state lives in Postgres, not in a process: that’s the entire point of the thing.

This is the reframe: a step of a ReAct loop maps one-to-one onto an Oban job. Thinking and acting happen inside perform/1. The “repeat” is the job enqueuing its successor. The conversation state — the part that must outlive any single process — lives in a row in your own agent_runs table, keyed by a run ID that every job in the chain carries in its args. The queue is durable because Oban persists every job to Postgres and only marks it completed after perform/1 returns successfully. Crash before that, and the job is still executing or available — it will run again.

You don’t get this for free with a GenServer. You get it for free with Oban because durability is the substrate, not a feature you bolt on.

The code: one agent step as a durable, idempotent Oban job

Here’s a real worker. It performs one step of an agent loop, persists the result, and enqueues the next step. It’s open-source Oban (v2.22 at time of writing) — no Pro features — and the comments call out the durability property at each point. APIs here are checked against the current Oban.Worker docs.

First, the run state. This is the part that must survive a restart, so it lives in a table you own:

defmodule MyApp.Agent.Run do
  use Ecto.Schema

  schema "agent_runs" do
    field :goal, :string
    field :status, Ecto.Enum, values: [:running, :completed, :failed], default: :running
    field :step, :integer, default: 0
    field :max_steps, :integer, default: 40
    # The full ReAct transcript: every think/act/observe turn, appended in order.
    # Because it's a column, a node restart can't lose it.
    field :messages, {:array, :map}, default: []
    field :result, :string

    timestamps(type: :utc_datetime_usec)
  end
end

Now the worker — one step of the loop:

defmodule MyApp.Agent.StepWorker do
  use Oban.Worker,
    queue: :agents,
    # An agent run is long. We want generous headroom for transient LLM/tool
    # failures, but NOT infinite — a poison step must eventually give up.
    max_attempts: 8,
    # Idempotency at the QUEUE level: never let two jobs for the same
    # (run_id, step) exist concurrently. If an enqueue is retried, or two
    # producers race, Oban dedupes instead of double-running the step.
    unique: [
      period: :infinity,
      fields: [:worker, :args],
      keys: [:run_id, :step],
      # Only dedupe against jobs that haven't finished. A genuinely new
      # attempt at the same step after a cancel is still allowed.
      states: [:available, :scheduled, :executing, :retryable]
    ]

  alias MyApp.Agent.{Run, Loop}
  alias MyApp.Repo

  @impl Oban.Worker
  def perform(%Oban.Job{args: %{"run_id" => run_id, "step" => step}}) do
    run = Repo.get!(Run, run_id)

    cond do
      run.status != :running ->
        # Idempotency at the DATA level: if a retry lands after the run already
        # finished (e.g. the job died AFTER committing but BEFORE Oban marked it
        # completed), do nothing. Re-running must be a no-op, not a double-spend.
        {:cancel, :run_already_finished}

      run.step >= run.max_steps ->
        finish(run, :failed, "step budget exhausted")

      true ->
        run_step(run, step)
    end
  end

  defp run_step(run, step) do
    # THINK: one LLM call. May raise on a 429 — see classify_error/1 below,
    # which decides retry-with-backoff vs. give-up. If this raises, perform/1
    # never returns :ok, so Oban leaves the job retryable and the run resumes.
    case Loop.next_action(run) do
      {:final, answer} ->
        finish(run, :completed, answer)

      {:tool_call, call} ->
        # ACT: run the tool. Wrap side effects so a retried step can detect
        # "I already did this" — see the idempotency note in execute_tool/2.
        observation = Loop.execute_tool(run, call)

        # OBSERVE + DURABLY ADVANCE: persist the new transcript and bump the
        # step counter in ONE transaction with enqueuing the next job. If the
        # node dies an instant later, either the whole advance committed (next
        # job is queued) or none of it did (this job is still retryable). There
        # is no torn state where we advanced but forgot to schedule the next step.
        advance(run, step, call, observation)
    end
  end

  defp advance(run, step, call, observation) do
    next_step = step + 1

    new_messages =
      run.messages ++
        [%{"role" => "assistant", "tool_call" => call},
         %{"role" => "tool", "result" => observation}]

    changeset = Ecto.Changeset.change(run, messages: new_messages, step: next_step)

    # Multi ties the state write and the next-step enqueue into one DB
    # transaction. Oban.insert/4 inside a Multi is first-class — the inserted
    # job only becomes visible if the whole transaction commits.
    Ecto.Multi.new()
    |> Ecto.Multi.update(:run, changeset)
    |> Oban.insert(:next, __MODULE__.new(%{"run_id" => run.id, "step" => next_step}))
    |> Repo.transaction()
    |> case do
      {:ok, _} -> :ok
      {:error, _op, reason, _changes} -> {:error, reason}
    end
  end

  defp finish(run, status, text) do
    # Terminal write: flip status and record the result/error text. Idempotent
    # because the status guard in perform/1 makes a re-run of a finished run a no-op.
    run |> Ecto.Changeset.change(%{status: status, result: text}) |> Repo.update()
    :ok
  end
end

Kick off a run by inserting the first step:

{:ok, run} =
  %MyApp.Agent.Run{goal: "Triage and reply to ticket #4821"}
  |> Ecto.Changeset.change()
  |> MyApp.Repo.insert()

# Step 0. The `unique` opts mean enqueuing this twice is harmless — the second
# insert is deduped, not run.
%{"run_id" => run.id, "step" => 0}
|> MyApp.Agent.StepWorker.new()
|> Oban.insert()

The shape to notice: each step enqueues exactly the next step, transactionally, alongside the state write. That’s the hand-rolled durable loop. There’s no driver process holding the run together. The Postgres row is the run; the chain of jobs is just how it walks forward. Roll a deploy at step 23 and the in-flight job goes back to available/retryable; when the node returns, Oban picks it up, re-reads the row, and continues. Nothing was lost because nothing important ever lived in a process.

Handling the failures that are specific to LLMs

A generic job queue handles “the tool 503’d.” Agents add three failure modes that need deliberate handling.

1. The 429, and backoff that respects Retry-After. When the LLM rate-limits you, the right move is to back off and try again without consuming a real failure. Oban gives you two levers. For a clean retry that doesn’t burn an attempt the same way an error does, return {:snooze, seconds} — it reschedules the job for later. (Note the documented quirk: snoozing increments max_attempts so total retries are preserved, which slightly skews the default backoff curve.) For genuine errors that should count, return {:error, reason} and let backoff/1 space out the retries:

# In MyApp.Agent.StepWorker

@impl Oban.Worker
# backoff/1 must return non_neg_integer() seconds. The default is exponential
# (min 15s + jitter); we override to add a floor and cap. When a tool or model
# hands us an explicit Retry-After, honor it instead of guessing.
def backoff(%Oban.Job{attempt: attempt, meta: %{"retry_after" => secs}})
    when is_integer(secs),
    do: secs

def backoff(%Oban.Job{attempt: attempt}) do
  base = trunc(:math.pow(2, attempt)) * 5
  min(base, 300) + :rand.uniform(10)
end

And inside perform/1, classify the failure so a retryable 429 backs off while an un-retryable 400 dies cleanly instead of churning through all eight attempts:

defp classify_and_raise({:rate_limited, retry_after}) do
  # Tell Oban to wait this long, then re-raise so the job goes retryable.
  # (Stash retry_after via a meta update, or snooze — pick one; don't do both.)
  {:snooze, retry_after}
end

defp classify_and_raise({:http_error, status}) when status in [400, 422],
  # A malformed request will fail identically every time. Don't retry it 8x and
  # don't poison the run — cancel this step and let the run record the failure.
  do: {:cancel, {:permanent_tool_error, status}}

defp classify_and_raise({:http_error, status}) when status in 500..599,
  # Transient. Return an error so backoff/1 spaces out the retry.
  do: {:error, {:transient_tool_error, status}}

{:cancel, reason} is the one to internalize: it stops the job and stops retrying. That’s exactly what you want for a poison step — a step that will fail identically on every attempt. Burning all eight max_attempts on a guaranteed-400 is just slow money-burning. (:discard does the same thing but is deprecated in favor of {:cancel, reason}.)

2. Partial tool output and double-execution. This is the subtle one. Say the tool charges a payment API or POSTs to a downstream system. The job runs the tool successfully, the side effect happens — and then the node dies before perform/1 returns and before the transaction commits. Oban, correctly, will retry the job, because from its perspective the step never completed. Now you’re about to run the tool a second time.

The queue can’t solve this for you, because the side effect is outside Postgres. You make the tool idempotent:

defp execute_tool(run, %{"name" => "charge", "args" => args} = call) do
  # Derive a stable idempotency key from the run + step. A retry produces the
  # SAME key, so the payment API (or your own ledger) dedupes the second call.
  # This is what makes "retried step doesn't double-charge" actually true.
  idem_key = "run:#{run.id}:step:#{run.step}:charge"
  PaymentAPI.charge(args, idempotency_key: idem_key)
end

Most serious APIs accept an idempotency key. For your own writes, enforce it with a unique constraint keyed on {run_id, step} and treat the constraint violation as “already done.” The queue’s unique option stops duplicate jobs; the idempotency key stops duplicate side effects from one job’s retries. You need both, and they operate at different layers.

3. The poison run, not just the poison step. max_attempts bounds a single step. Bound the whole run too — that’s the run.step >= run.max_steps guard in perform/1. An agent stuck in a tool-call loop will happily enqueue itself forever; the step budget in your own table is the circuit breaker. Cheap, and it lives in the same durable place as everything else.

If your runs are complex enough to need fan-out, branches, or a real dependency graph (step C waits on A and B), that’s where Oban Pro’s Workflow earns its license — it models jobs as a DAG and supports appending jobs to a running workflow when you don’t know all the steps up front. That’s a paid tier feature; the self-enqueuing chain above is plain open-source Oban and covers the linear ReAct loop, which is the common case.

When NOT to reach for Oban here

This pattern is not a hammer. Three cases where it’s the wrong call:

Sub-second interactive streaming. If a human is watching tokens stream into a chat UI and expects a reply in two seconds, Oban is the wrong layer — full stop. Queue latency, even when small, is latency you’re adding to a tight interactive loop, and you lose token streaming entirely. That’s LiveView’s job: stream the model output straight to the client over the socket, keep the in-flight conversation in the LiveView process, and persist when the turn completes. (A dedicated post on the LiveView-streaming side of this is coming soon.) Oban is for the durable background agent, not the live foreground chat.

Genuinely ephemeral single-shot calls. If the whole “agent” is one prompt, one response, no tools, no loop — just call the API in a Task and move on. Wrapping a single stateless call in a durable job is ceremony with no payoff. The durability machinery earns its keep only when there’s multi-step state worth protecting.

An in-memory, long-lived conversation that must stay hot. If you genuinely need a conversation held live in memory across many turns with sub-step-latency access to working state — a GenServer (ideally under a DynamicSupervisor, keyed in a Registry) is the right model, and that’s a legitimate use of a process: it models real concurrent, stateful, isolated work. The honest line is the one I drew at the top: reach for the GenServer when state must stay hot in memory; reach for Oban when state must stay durable on disk and survive the process dying. Most batch/background agents want the latter. For more on when a process is and isn’t the answer, see the BEAM-for-agents piece and the concurrency-model deep dive.

Verdict

The rule I’d give a team: if losing the run is a real cost — money already spent, side effects already committed, a customer waiting — make each step an Oban job. You’ll get durability, retries, idempotency, backoff, and cross-node distribution from a library your Elixir app probably already runs, instead of reinventing a worse queue and calling it an agent framework. Save the GenServer for the genuinely live, in-memory case, and reach for Oban Pro only when a linear chain isn’t enough and you need real DAGs.

A companion post on RAG retrieval as a set of cached, idempotent Oban jobs is coming soon. The durability argument there is the same — it just happens one layer down, at the tool.

Building an Elixir agent system and trying to decide where the durability boundary goes? That’s exactly the kind of call I help teams make as a fractional CTO. Get in touch.

Primary sources: Oban docs · Oban.Worker · Oban on GitHub · Oban Pro Workflow.