Metering LLM Usage Per Customer in Phoenix

TL;DR: Wiring ReqLLM’s telemetry events into LiveDashboard and PromEx answers “what is my LLM spend doing right now” — an ops question. This post answers a different one: “what do I invoice tenant X for last month, and can I defend that number in a support ticket.” That’s a billing-grade problem, not a dashboard problem, and it has a different centerpiece: an append-only Postgres ledger, integer math for every dollar (never :float), tenant attribution baked into the call site instead of bolted on after, and idempotent writes so a retried job can’t double-bill a customer. Vendors like Lago, OpenMeter, and Stripe’s Billing Meters exist for exactly this — they’re the right off-ramp once rating, tax, and dunning outgrow a table you own, not a reason to avoid building the ledger in the first place.

Telemetry tells you what happened; a ledger tells you what to bill

The telemetry post covers ReqLLM’s [:req_llm, :token_usage] event in detail — I won’t re-explain the event shape here beyond the one thing that matters for what follows: its measurements nest token counts under a :tokens key, with cost fields like total_cost sitting flat at the top level. That event is genuinely enough to build a live cost dashboard, and if that’s all you need, stop there.

It is not enough to send an invoice. A dashboard tolerates an undercount — nobody’s paged when yesterday’s total is off by three cents. An invoice line item is a claim you’re making to a paying customer, and the bar for “an LLM call happened, I saw the number go by in LiveDashboard” is nowhere near the bar for “I can show you the row, the tenant it belongs to, and the exact cents.” That’s the gap this post closes: the same telemetry event, but written into a structure built to survive an audit instead of a glance.

This is also the argument I made in why telemetry has to exist before you reach for a better model taken one step further — once you can see usage, the very next question a finance person or a founder asks is “so what does this cost per customer,” and that question doesn’t get answered by a metric, it gets answered by a ledger.

Build vs. buy, up front

Before any schema: don’t build this if you don’t have to. Stripe’s Billing Meters are a reasonable first move if you already run Stripe for payments and your pricing is a single metered dimension — you report usage, Stripe rates and invoices it. OpenMeter and Lago go further: multi-dimensional pricing, credits, entitlements, a customer-facing usage dashboard, and — for Lago specifically — payment-provider independence, since it’s open source and talks to Stripe, Adyen, or GoCardless behind the same API. All three exist because rating, tax, dunning, and self-serve plan changes are a real product surface, not a weekend project, and there is no glory in reinventing Stripe’s invoice engine badly.

What none of them do for you is the part that’s actually specific to your product: deciding what a billable unit is for an LLM-backed feature, and getting the raw usage event out of your application and into a durable record before you hand it to anyone. That’s the build side, and it’s smaller than it looks — a table, a write path, and an aggregation query. It also happens to be a small lift specifically because you’re on this stack: an append-only Postgres table and a job queue you already run aren’t new infrastructure, they’re the same durable-by-default primitives the case for Elixir at an AI startup argues you get close to for free. The decision isn’t “build or buy,” it’s “build the ledger, and decide later whether a vendor should own rating and invoicing on top of it.” I’ve walked through this exact build-vs-buy tradeoff, just for a different function, in the vCISO math post — the shape of the reasoning is identical: pay for the parts that are genuinely someone else’s core competency (tax jurisdictions, dunning logic, PCI scope), build the parts that are actually your product’s domain logic (what counts as usage, who it belongs to). If you’re not sure which side of that line your team is on, that’s the conversation I have as a fractional CTO for AI startups more often than almost any other.

The ledger: append-only, integer cents, never :float

Here’s the Iron Law this post exists to make concrete: money is never a :float. Not “usually,” not “unless it’s small” — never. A float can’t represent most decimal fractions exactly, and LLM pricing makes this worse than typical SaaS billing does, because the per-unit price is fractional by construction: a $3-per-million-token input rate costs $0.000003 per token, and summing thousands of float-rounded per-call costs across a billing period compounds an error that’s invisible on any single row and very visible on an invoice total.

The fix isn’t Decimal here, though Decimal is the right call for currency amounts users type into a form. It’s smaller than that: store cost in integer micro-dollars — one micro-dollar is $0.000001 — and round to cents exactly once, at invoice time, not once per event. Anthropic’s current published rate for Claude Sonnet 4.6 is $3 per million input tokens, $15 per million output tokens, which happens to convert to a clean integer: 3 micro-dollars per input token, 15 micro-dollars per output token. Store the price table in the same unit as the ledger and every multiplication stays an integer, no rounding until the very last step.

The migration:

create table(:usage_events,
             primary_key: false) do
  add :id, :binary_id, primary_key: true
  add :tenant_id, :binary_id, null: false
  add :customer_id, :binary_id, null: false
  add :provider, :string, null: false
  add :model, :string, null: false
  add :input_tokens, :integer, null: false
  add :output_tokens, :integer, null: false
  # Integer micro-dollars. 1 = $0.000001.
  # NEVER :float for this column.
  add :cost_micros, :integer, null: false
  add :request_id, :string, null: false

  timestamps(updated_at: false)
end

create unique_index(
  :usage_events, [:request_id]
)
create index(
  :usage_events, [:tenant_id, :inserted_at]
)

usage_events is append-only on purpose — no update, ever. If a call needs a correction, you insert a compensating row with a negative cost_micros, the same discipline a real accounting ledger uses, instead of mutating history that a past invoice already referenced. The unique_index on request_id is the idempotency guarantee: populate it from the id ReqLLM already stamps on every response, and a retried write can’t create a second billable row for one API call.

The schema, with the Iron Law spelled out where a future reader will actually see it:

defmodule MyApp.Billing.UsageEvent do
  use Ecto.Schema
  import Ecto.Changeset

  @primary_key {:id, :binary_id,
                autogenerate: true}
  schema "usage_events" do
    field :tenant_id, Ecto.UUID
    field :customer_id, Ecto.UUID
    field :provider, :string
    field :model, :string
    field :input_tokens, :integer
    field :output_tokens, :integer
    field :cost_micros, :integer
    field :request_id, :string

    timestamps(updated_at: false)
  end

  @fields ~w(tenant_id customer_id provider
             model input_tokens output_tokens
             cost_micros request_id)a

  def changeset(event, attrs) do
    event
    |> cast(attrs, @fields)
    |> validate_required(@fields)
    |> validate_number(:input_tokens,
         greater_than_or_equal_to: 0)
    |> validate_number(:output_tokens,
         greater_than_or_equal_to: 0)
    |> unique_constraint(:request_id)
  end
end

Attribution at the call site, not retrofitted onto the event

Here’s the failure mode that actually loses money: a call goes out to the provider with no tenant attached, and there is no honest way to reconstruct whose bill it belongs to after the fact. ReqLLM’s [:req_llm, :token_usage] event carries provider and model in its metadata because those are things ReqLLM itself knows. It doesn’t and can’t know your tenant_id — that’s your application’s concept, not the client library’s, and trying to smuggle it in after the fact (global process state, a Logger.metadata read inside a shared :telemetry.attach handler) is fragile exactly where fragility is most expensive: it works in every test and silently drops tenant on some request path you didn’t think to check.

The reliable fix is structural, not clever: wrap every LLM call in a project-owned module whose function signature requires a tenant, and emit your own billing event from inside that wrapper — where tenant_id is already a bound variable, not something you’re trying to recover later.

defmodule MyApp.Billing.LLM do
  @moduledoc """
  The only sanctioned way to call an LLM provider
  in this app. tenant_id is a required argument —
  there is no code path that reaches ReqLLM without
  a tenant already attached.
  """

  @event [:my_app, :usage, :billed]

  # cents-per-million rate table, in micro-dollars
  # per token — see /blog/phoenix-llm-telemetry for
  # the raw ReqLLM event this wraps.
  @rates %{
    "anthropic:claude-sonnet-4-6" =>
      %{input: 3, output: 15}
  }

  def generate(tenant_id, customer_id, model,
               prompt, opts \\ [])
      when is_binary(tenant_id) and
           is_binary(customer_id) do
    {:ok, response} =
      ReqLLM.generate_text(
        model, prompt, opts
      )

    usage = response.usage
    rate = Map.fetch!(@rates, model)

    [provider, _] =
      String.split(model, ":", parts: 2)

    cost_micros =
      usage.input_tokens * rate.input +
        usage.output_tokens * rate.output

    :telemetry.execute(
      @event,
      %{
        input_tokens: usage.input_tokens,
        output_tokens: usage.output_tokens,
        cost_micros: cost_micros
      },
      %{
        tenant_id: tenant_id,
        customer_id: customer_id,
        provider: provider,
        model: model,
        request_id: response.id
      }
    )

    {:ok, response}
  end
end

Two things worth noticing. First, this computes cost_micros itself from token counts and a price table it owns, rather than trusting ReqLLM’s total_cost field verbatim — that keeps the number you bill on inside your own integer math the whole way through, instead of inheriting whatever float precision the provider’s SDK used upstream. Second, this is exactly the pattern the Iron Law “wrap third-party library APIs behind a project-owned module” is for: ReqLLM.generate_text/3 never appears anywhere else in the codebase, which means “no tenant, no call” isn’t a convention anyone can forget — it’s the only function that exists.

A handler attached once, at boot, turns that event into a ledger row:

:telemetry.attach(
  "usage-ledger",
  [:my_app, :usage, :billed],
  &MyApp.Billing.Ledger.record/4,
  nil
)

defmodule MyApp.Billing.Ledger do
  alias MyApp.Repo
  alias MyApp.Billing.UsageEvent

  def record(_event, measurements,
             metadata, _config) do
    attrs = Map.merge(
      metadata,
      measurements
    )

    %UsageEvent{}
    |> UsageEvent.changeset(attrs)
    |> Repo.insert(
      on_conflict: :nothing,
      conflict_target: :request_id
    )
  end
end

on_conflict: :nothing paired with the request_id unique index means the same event firing twice — a retried job, a duplicate :telemetry.execute from a bug — writes the row once. That’s the idempotency guarantee at the data layer; it’s the same discipline the Oban post argues for at the job layer, applied here to a plain insert instead of a queued job.

Aggregation: turning rows into an invoice line

Once events are landing, the invoice query is a straight aggregate — this isn’t the has_many-vs-belongs_to decision the Ecto patterns post covers, it’s the simpler cousin: GROUP BY and sum/1, with the rounding done exactly once, after the sum:

def invoice_lines(tenant_id, period) do
  {start_at, end_at} = period

  from(e in UsageEvent,
    where: e.tenant_id == ^tenant_id,
    where: e.inserted_at >= ^start_at,
    where: e.inserted_at < ^end_at,
    group_by: e.customer_id,
    select: %{
      customer_id: e.customer_id,
      total_micros: sum(e.cost_micros),
      input_tokens: sum(e.input_tokens),
      output_tokens: sum(e.output_tokens)
    }
  )
  |> Repo.all()
  |> Enum.map(&to_cents/1)
end

defp to_cents(%{total_micros: micros} = row) do
  # Exactly one rounding step, at the end, not
  # once per event.
  cents = round(micros / 10_000)
  Map.put(row, :total_cents, cents)
end

Worth running that arithmetic once, by hand, so the “round once” claim isn’t just asserted. Three calls for one tenant on Claude Sonnet 4.6 at $3/$15 per million tokens: 15,000 input + 3,200 output tokens on the first call is 15_000 * 3 + 3_200 * 15 = 93,000 micro-dollars. A second, larger call at 120,000 input + 40,000 output tokens is 120_000 * 3 + 40_000 * 15 = 960,000 micro-dollars. A third, tiny call at 500 input + 50 output tokens is 500 * 3 + 50 * 15 = 2,250 micro-dollars. Summed: 93,000 + 960,000 + 2,250 = 1,055,250 micro-dollars, which is 1,055,250 / 10,000 = 105.525 cents — rounds to 106 cents, $1.06 on the invoice. Every intermediate value in that chain is an integer; the only place a fraction ever appears is the final division, and it appears exactly once.

Overage handling: soft-cap warnings and hard-cap enforcement as Oban jobs

Aggregation on demand is fine for generating a monthly invoice. It’s too slow to be the thing that stops a runaway agent loop from burning a tenant’s month in an afternoon — for that you want a check that runs close to real time, and Oban is the same durable-job substrate the agent-runtime post argues for, applied to spend instead of agent steps. After every ledger write, enqueue a cap check — uniqued per tenant so a burst of usage events doesn’t enqueue a burst of redundant checks:

defmodule MyApp.Billing.CapCheckWorker do
  use Oban.Worker,
    queue: :billing,
    unique: [
      period: 60,
      fields: [:worker, :args],
      keys: [:tenant_id],
      states: [
        :available, :scheduled, :executing
      ]
    ]

  alias MyApp.Billing.{Ledger, Tenants}

  @impl Oban.Worker
  def perform(%Oban.Job{
        args: %{"tenant_id" => tenant_id}
      }) do
    tenant = Tenants.get!(tenant_id)
    spent = Ledger.period_total_micros(tenant_id)

    cond do
      spent >= tenant.hard_cap_micros ->
        Tenants.suspend(tenant, spent)

      spent >= tenant.soft_cap_micros and
          not tenant.soft_cap_warned ->
        Tenants.mark_warned(tenant)
        MyApp.Billing.Notifier.soft_cap(tenant)

      true ->
        :ok
    end
  end
end

The unique block does real work here, not just tidiness: period: 60 collapses every cap check enqueued for one tenant within a rolling minute into a single job, which matters because a busy tenant can generate dozens of usage events a minute and you don’t need — or want — a cap check to run once per event. Tenants.suspend/2 and Tenants.mark_warned/1 need to be idempotent in their own right too, the same way the Oban post’s payment example needed an idempotency key at the tool layer: a job that reruns because Oban retried it after a timeout must not re-send the warning email or re-flip an already-suspended tenant into some worse state. A boolean flag checked before the side effect — not tenant.soft_cap_warned above — is enough; you don’t need anything fancier than the guard clause already sitting in perform/1.

Enqueue it from the same place the ledger write happens:

def record(_event, measurements,
           metadata, _config) do
  attrs = Map.merge(metadata, measurements)

  with {:ok, event} <-
         %UsageEvent{}
         |> UsageEvent.changeset(attrs)
         |> Repo.insert(
              on_conflict: :nothing,
              conflict_target: :request_id
            ) do
    %{tenant_id: metadata.tenant_id}
    |> MyApp.Billing.CapCheckWorker.new()
    |> Oban.insert()

    {:ok, event}
  end
end

When to hand off to a vendor

The ledger above is genuinely small — one table, one wrapper module, one aggregation query, one Oban worker. It’ll carry a startup a long way: correct invoice totals, tenant attribution that can’t silently drop, caps that actually stop a runaway bill. What it doesn’t do, and what you shouldn’t build yourself past a certain point, is the part that’s a real second product: tax calculation across jurisdictions, dunning and retry logic on failed payments, a self-serve customer-facing usage dashboard, multi-currency, proration on mid-cycle plan changes, revenue recognition reporting for your finance team. That’s the moment to point your invoice_lines/2 query at Stripe’s usage-based billing, OpenMeter, or Lago instead of writing an invoicing engine from scratch — you keep the ledger (it’s still the source of truth and the thing an auditor asks for), and let the vendor own rating and collection on top of it.

The honest signal that it’s time: when the billing problem starts eating more engineering time per week than the product problem. If you’re not sure where you are on that curve, that build-vs-buy read is exactly the kind of call I help AI startups make as a fractional CTO — happy to look at what you’ve got before you sink a quarter into either direction.