I Read Oban's Source Code So You Don't Have To

TL;DR: Oban’s README tells you it’s “reliable” and “backed by Postgres.” It doesn’t tell you that job fetching is one UPDATE ... FOR UPDATE SKIP LOCKED wrapped in a CTE to stop the query planner from optimizing it into something slower, that leader election is a plain INSERT ... ON CONFLICT upsert against a single-row table, or that the same mechanism that makes it reliable — high-frequency UPDATEs on one table — is exactly what fills that table with dead tuples faster than autovacuum can clean them. I read lib/oban/engines/basic.ex, lib/oban/notifiers/postgres.ex, lib/oban/peers/database.ex, and lib/oban/stager.ex in the oban-bg/oban repo at tag v2.23.1 (the current stable release, tagged August 2, 2026) to write this. Every mechanism below is a function I opened, not a claim I remembered from a blog post.

Scope note

I’ve written before about running an AI agent loop on Oban — the case for using it as an agent runtime instead of hand-rolling a GenServer. This post is the other half: why the mechanics hold up under that workload, and where they don’t. It assumes you’re comfortable with the concurrency model underneath it — processes, supervision, and a single Postgres primary doing more work than people expect. If you’re deciding whether to use Oban, read that one first. If you’re already running it and want to know what’s actually happening under Oban.insert/1, read this one.

How does Oban fetch jobs without workers fighting over them?

Oban fetches a batch of jobs with a single UPDATE statement that uses Postgres’s FOR UPDATE SKIP LOCKED row lock — any row another transaction already has locked is silently skipped rather than blocked on, so N workers polling the same queue never queue behind each other waiting for a lock.

The actual query lives in Oban.Engines.Basic.fetch_jobs/3. It’s built with Ecto, and the shape matters more than the syntax:

WITH subset AS (
  SELECT id FROM oban_jobs
  WHERE state = 'available'
    AND queue = $1
  ORDER BY priority, scheduled_at, id
  LIMIT $2
  FOR UPDATE SKIP LOCKED
)
UPDATE oban_jobs
SET state = 'executing',
    attempted_at = now(),
    attempt = attempt + 1
FROM subset
WHERE oban_jobs.id = subset.id
  AND oban_jobs.attempt < oban_jobs.max_attempts
RETURNING oban_jobs.*;

Two things in the real source are easy to miss reading the docs instead of the code. First, the subset CTE isn’t decorative — the comment directly above it in basic.ex explains why it exists: “the Postgres planner may choose to generate a plan that executes a nested loop over the LIMITing subquery, causing more UPDATEs than LIMIT,” so the CTE acts as an “optimization fence” forcing Postgres not to flatten the query. Without it, a clever planner could re-run the inner scan per outer row and update more jobs than the LIMIT requested — silently over-claiming work. Second, the whole fetch runs inside Repo.transaction/2, and the claim (the UPDATE) and the read (RETURNING) happen atomically — a worker either gets a job it now owns, with attempt already incremented, or it gets nothing.

The practical consequence: two nodes running 20 workers each against the same queue never contend for the same row. Each fetch takes what’s free, skips what isn’t, and the lock is released the instant the transaction commits. That’s the whole trick — it’s a boring UPDATE, not a distributed consensus protocol, and that’s exactly why it scales as well as it does under a single Postgres primary.

When does Oban still poll?

Oban still polls once a second by default, through a GenServer called the Stager, to move scheduled and retryable jobs into available state — LISTEN/NOTIFY tells workers a new job exists, but Postgres has no equivalent push mechanism for “this row’s scheduled time has now arrived.”

Oban.Stager (lib/oban/stager.ex) runs Engine.stage_jobs/3 on a timer (interval: :timer.seconds(1) in the struct default) inside handle_info(:stage, state). stage_jobs/3 in basic.ex is another UPDATE, this one moving rows whose scheduled_at <= now() from scheduled/retryable into available. Only then does it call notify_queues/1, which fires Notifier.notify(conf, :insert, payload) — the actual LISTEN/NOTIFY push that wakes idle worker processes.

So the two mechanisms are layered, not redundant: staging is time-based polling because Postgres has no NOTIFY for “a timestamp condition became true,” and notification is push-based because once a row is staged, telling every listening producer is cheap and instant via pg_notify. Oban.Notifiers.Postgres (lib/oban/notifiers/postgres.ex) implements this over a dedicated Postgrex.SimpleConnection that issues raw LISTEN "channel"; statements and relays incoming notifications to registered listener PIDs — one persistent connection per node, not one per queue. Worth noting from the moduledoc directly: this notifier doesn’t work behind a transaction-mode PgBouncer, because LISTEN needs a session-scoped connection; Oban ships Oban.Notifiers.PG (distributed Erlang :pg, no database round-trip at all) as the alternative for that topology.

The one-second stage interval is also why a scheduled job in Oban is never exactly on time — it fires within roughly a second of its scheduled_at, which is fine for retry backoff and cron-adjacent work, and the wrong tool if you need sub-second scheduling precision.

How does a cluster of nodes agree on one leader?

Oban elects a leader with a plain INSERT ... ON CONFLICT upsert against a single-row-per-cluster oban_peers table — whichever node’s insert lands first holds a lease with an expiry, and every other node’s conflicting insert just fails to claim it, so there’s no separate consensus algorithm to reason about.

This matters because several plugins — the cron scheduler, the pruner, the stager’s own “am I the one that runs the global stage” decision — must run exactly once across a cluster, not once per node. Oban.Peers.Database (lib/oban/peers/database.ex) runs its own election on a timer (default 30 seconds, halved when the node currently holds leadership, so a leader re-asserts twice as often as challengers probe). The core of it is regular_upsert/2:

defp regular_upsert(
       %{node: node, expires_at: expires_at}
       = peer_data,
       state
     ) do
  on_conflict =
    "oban_peers"
    |> where([p], p.node == ^node)
    |> update([p],
         set: [expires_at: ^expires_at]
       )

  repo_opts = [
    conflict_target: :name,
    on_conflict: on_conflict
  ]

  case Repo.insert_all(
         state.conf, "oban_peers",
         [peer_data], repo_opts
       ) do
    {0, nil} -> false
    {_, nil} -> true
  end
end

conflict_target: :name means the table has a unique constraint on the cluster name — one row per Oban instance, cluster-wide. If the insert actually adds a row, {1, nil} comes back and that node becomes leader. If a row already exists and belongs to a different node, the on_conflict update doesn’t fire either — look closely at its where([p], p.node == ^node), where node is destructured from the inserting node’s own peer_data. A challenger’s update matches no row, Postgres reports zero rows affected, the case falls to {0, nil} -> false, and that node isn’t leader. Crucially, the other node’s row is never touched; if it were, a challenger would be extending the sitting leader’s lease for it.

That same WHERE clause is what makes renewal work. When the current leader re-upserts, the clause matches its own row, expires_at slides forward, {1, nil} comes back, and it keeps leadership. No heartbeat protocol, no Raft — one unique index and one WHERE clause doing the work of both.

Leadership has a lease, not a permanent hold: delete_expired_peers/1 runs before every election and removes any peer row past its expires_at, so a leader that crashes without a clean shutdown loses leadership within one interval window, not indefinitely. On a graceful shutdown, terminate/2 deletes the leader’s own row and calls Notifier.notify(conf, :leader, %{down: ...}) so the next election happens immediately instead of waiting out the timer.

What each mechanism costs you

None of this is free. Every mechanism above leans on a specific Postgres feature, and every one of those features has a bill that shows up somewhere else.

The cost nobody puts in the getting-started guide

That last row is the one that actually bites in production, and it’s the one an agent workload makes worse, not better. Every job in Oban’s oban_jobs table goes through several UPDATEs in its lifetime — claimed (fetch), completed or errored, and eventually deleted by the pruner — and Postgres’s MVCC model doesn’t overwrite a row in place. Each UPDATE writes a new tuple version and marks the old one dead; VACUUM is the only thing that reclaims that space. Oban’s own scaling guide says it plainly: “the MVCC system only flags rows for deletion later. Then, those rows are deleted when the auto-vacuum runs” — and warns that “the default auto vacuum settings are conservative and may fall behind on active tables.”

This is exactly the profile an AI agent workload produces: short jobs (one LLM call, one tool call), enqueued in bursts, each one going through claim → complete in seconds, at a volume that can run into the tens of thousands per hour on a busy agent fleet. That’s high UPDATE churn on one table, which is precisely what PlanetScale’s writeup on keeping a Postgres queue healthy and Richard Yen’s piece on the consequences of using Postgres as a job queue both flag as the failure mode nobody budgets for: dead tuples accumulating faster than autovacuum can clean them, tables ballooning to tens of gigabytes when the live data is a few megabytes, and — the sharper point — SKIP LOCKED doesn’t mean rows are invisible to a scan. Postgres still has to find each dead or locked row and check its status before moving on, so a bloated table makes the fetch itself slower, not just VACUUM.

If you’re running an agent fleet that enqueues a job per LLM turn, tune this before it becomes an incident — not after the table is into the tens of gigabytes the Postgres writeups keep describing. It’s also past what your application monitoring is watching for, which is the same blind spot instrumenting LLM calls with telemetry exists to close.

Oban’s own guide gives the concrete counter-move: tune autovacuum on oban_jobs specifically, more aggressively than Postgres’s cluster-wide defaults —

ALTER TABLE oban_jobs SET (
  autovacuum_vacuum_scale_factor = 0,
  autovacuum_vacuum_threshold = 100
);

— which tells autovacuum to trigger after roughly 100 dead tuples regardless of table size, instead of the default scale-factor-based threshold that waits for a percentage of the table to go dead first (irrelevant on a small table, dangerously slow on a large one). The Oban.Plugins.Pruner plugin, which deletes completed/cancelled/discarded jobs after a configurable max_age (default 60 seconds), is the other half — it keeps the table itself small so each VACUUM pass has less to do. And at genuinely extreme volume, Oban’s docs point at table partitioning as the real fix, because “dropping tables entirely is instantaneous and leaves zero bloat” — a mechanism only available in Oban Pro, not the open-source core I read for this post.

The takeaway from reading it instead of trusting the README

Oban’s reliability isn’t magic — it’s a small set of ordinary Postgres primitives (row locks, LISTEN/NOTIFY, a unique-constraint upsert, MVCC) composed carefully, with the comments in basic.ex showing real awareness of how the query planner could betray you. That composition is also why the failure modes are ordinary Postgres failure modes: vacuum pressure, connection pooler incompatibility, a hot single-row table under extreme leader-election load. None of that shows up if you only read the README. It shows up the first time your agent workload runs hot enough to out-churn autovacuum, which is a Tuesday, not an edge case.

Reading it end to end is also a reminder of what source code is for. The reliability model came out of following intent across files and building a picture the README never states — a human habit, and one worth keeping as the notation around our code drifts toward readers who aren’t people.

If you’re deciding whether Oban’s operational model fits your team’s AI infrastructure — or you’ve already got one of these tables quietly growing past what your monitoring is watching for — that’s a conversation worth having before the growth curve gets ahead of you. I help AI startups get this kind of infrastructure review done as part of fractional CTO engagements.