Instrumenting LLM Calls in Phoenix with Telemetry

TL;DR: ReqLLM — the Req-based LLM client for Elixir — already emits :telemetry events with token counts, calculated cost, and request duration on every call, no wrapper code required. The gap most Phoenix teams have isn’t visibility, it’s five lines of Telemetry.Metrics connecting an event that already exists to a dashboard that already ships with Phoenix. This post is the practitioner path: what ReqLLM emits and where I verified it, how to wire it into LiveDashboard for local dev, how to ship the same numbers to Prometheus with a PromEx plugin for production, the minimal hand-rolled version if you’re on a raw HTTP client instead, and the two numbers actually worth paging someone over.

The “what if I don’t want to buy Helicone” question

I wrote the case for treating LLM telemetry as a first-class layer, not an afterthought — the argument that most “the model is bad” complaints are actually “we can’t see what the model is doing” problems, and that a telemetry layer usually fixes more than a model upgrade does. That post is deliberately stack-agnostic and points at hosted options like Helicone and Langfuse, because most teams asking “how do I see what my LLM calls are doing” are on Python and a hosted proxy is the fastest path.

If you’re running Phoenix, you already have two other options that a Python shop doesn’t: a BEAM-native dashboard that ships in every phx.new app, and a Prometheus/Grafana story that’s one dependency away. This post is the “okay, but I don’t want to route every LLM call through a third-party proxy” answer for that stack. It assumes you’ve already read (or don’t need) the argument for why telemetry matters, and just want the wiring.

One more adjacent post worth knowing about first: testing AI agent outputs with ExUnit covers :telemetry.attach/:telemetry.execute as primitives, using a test-time event that asserts on tool-call order. I won’t re-explain what those functions do here — if you haven’t used :telemetry before, read that post’s first section, then come back. This is the production sibling of that same event system: instead of asserting on an event in a test, you’re aggregating it into cost-per-day and p95 latency.

What ReqLLM already gives you for free

If your Phoenix app talks to an LLM through ReqLLM — the Req/Finch-based client from the Jido ecosystem, covering Anthropic, OpenAI, Google, Bedrock, and a dozen other providers behind one interface — you don’t need to write a :telemetry.span around your LLM calls. ReqLLM already does it. I verified the exact event shape against the library’s telemetry guide and the ReqLLM.Telemetry source rather than trust the README summary, because getting an event name or a measurement key wrong here means a wired-up dashboard that silently reads zeros.

ReqLLM emits two families of events:

The request span — [:req_llm, :request, :start] / :stop / :exception. This is what the docs now recommend for new integrations, because the :stop event carries both duration and the full request context in one place. Measurements on :stop are %{duration: native_time, system_time: ...}. duration is in native monotonic units, the same convention Phoenix’s own endpoint/request telemetry uses, so you convert it with unit: {:native, :millisecond} wherever you consume it rather than doing the math by hand. Metadata on :stop includes request_id, operation, mode, provider, model (an LLMDB.Model struct, not a string), http_status, finish_reason, and the field that matters for cost: usage, a map with token counts and cost fields.

The compat event — [:req_llm, :token_usage]. A narrower event fired specifically for cost/token aggregation, and the one most tutorials will show you first because it’s simpler. Its measurements map nests the raw usage numbers under a tokens key (%{tokens: %{input_tokens: _, output_tokens: _, total_tokens: _}, cost: _, total_cost: _, ...}) rather than flattening them — worth an IO.inspect in iex against your installed version before you write metric definitions against it, because compat-layer field shapes are exactly the kind of thing that shifts between minor releases without a changelog headline.

Either event gets you cost and tokens without touching your LLM call sites. That’s the actual finding here: the boilerplate you’d expect to write — wrap every provider call in :telemetry.span, thread through model name and token counts by hand — doesn’t need writing if ReqLLM is already your client.

Wiring it into LiveDashboard in dev

Phoenix’s Telemetry.Metrics module is the layer both LiveDashboard and PromEx consume — you define metrics once, against event names, and both reporters subscribe to the same list. Every phx.new --live app already has a YourAppWeb.Telemetry module with a metrics/0 function full of Phoenix and Ecto metrics. Add ReqLLM’s events to that same list:

# lib/my_app_web/telemetry.ex
defp req_llm_metrics do
  [
    distribution(
      "req_llm.request.duration",
      event_name: [:req_llm, :request, :stop],
      measurement: :duration,
      unit: {:native, :millisecond},
      tags: [:provider, :model],
      tag_values: fn meta ->
        %{
          provider: meta[:provider],
          model: meta[:model] && meta[:model].id
        }
      end,
      description: "LLM request latency"
    ),
    sum(
      "req_llm.tokens.total",
      event_name: [:req_llm, :token_usage],
      measurement: fn m ->
        get_in(m, [:tokens, :total_tokens]) || 0
      end,
      tags: [:provider],
      tag_values: &%{provider: &1[:provider]},
      description: "Tokens consumed, cumulative"
    ),
    sum(
      "req_llm.cost.total",
      event_name: [:req_llm, :token_usage],
      measurement: fn m -> m[:total_cost] || 0 end,
      tags: [:provider, :model],
      tag_values: fn meta ->
        %{
          provider: meta[:provider],
          model: meta[:model] && meta[:model].id
        }
      end,
      description: "Calculated USD cost, cumulative"
    ),
    counter(
      "req_llm.request.count",
      event_name: [:req_llm, :request, :stop],
      tags: [:provider, :finish_reason]
    )
  ]
end

Then splice req_llm_metrics() into the existing metrics/0 list. That’s the whole integration — no attach calls, no GenServer to hold running totals. Telemetry.Metrics handles the subscription; LiveDashboard’s Metrics tab picks up anything in that list automatically the moment you PORT=4001 mix phx.server and make a call. distribution gives you a live latency histogram per provider/model, sum gives you a running token and cost counter, counter gives you call volume broken out by finish_reason — which is the fastest way to notice a provider silently truncating responses before a user complains.

The tag_values function matters more than it looks: model in ReqLLM’s metadata is an LLMDB.Model struct, and neither LiveDashboard nor Prometheus wants a struct as a label value, so pull .id out before it hits the tag list.

Shipping the same numbers to Prometheus with PromEx

LiveDashboard is a great local and staging tool, but it only shows you the current node’s in-memory state — it doesn’t survive a restart or aggregate across a fleet. For anything you’d actually alert on, ship the same Telemetry.Metrics definitions to Prometheus via PromEx, which is a plugin over the exact same event/metric model — you’re not learning a second API, you’re pointing the same concepts at a different reporter.

A custom PromEx plugin for ReqLLM, following the plugin-writing guide:

defmodule MyApp.PromEx.Plugins.ReqLLM do
  use PromEx.Plugin

  @impl true
  def event_metrics(_opts) do
    Event.build(
      :req_llm_event_metrics,
      [
        distribution(
          [:req_llm, :request, :duration, :milliseconds],
          event_name: [:req_llm, :request, :stop],
          measurement: :duration,
          unit: {:native, :millisecond},
          tags: [:provider],
          tag_values: fn meta ->
            %{provider: meta[:provider]}
          end,
          reporter_options: [
            buckets: [100, 500, 1_000, 3_000, 10_000]
          ]
        ),
        sum(
          [:req_llm, :cost, :usd, :total],
          event_name: [:req_llm, :token_usage],
          measurement: fn m -> m[:total_cost] || 0 end,
          tags: [:provider]
        )
      ]
    )
  end
end

Register it alongside the built-in plugins in your app’s PromEx module:

defmodule MyApp.PromEx do
  use PromEx, otp_app: :my_app

  @impl true
  def plugins do
    [
      PromEx.Plugins.Application,
      PromEx.Plugins.Beam,
      PromEx.Plugins.Phoenix,
      {MyApp.PromEx.Plugins.ReqLLM, []}
    ]
  end
end

From there it’s standard PromEx: mix prom_ex.gen.config if you haven’t set up a datasource yet, and the metrics show up at /metrics for Prometheus to scrape. The buckets on the duration histogram are the one thing worth tuning by hand. LLM calls run in the hundreds-of-milliseconds to tens-of-seconds range, not the single-digit milliseconds Phoenix’s own request histogram defaults assume, so the stock bucket boundaries will bucket almost every LLM call into the same “overflow” bucket if you don’t override them.

If you’re not on ReqLLM: the minimal hand-rolled span

If your app calls a provider’s SDK directly, or does raw Req/Finch requests without ReqLLM, you don’t get the events above for free — but you also don’t need much code to get equivalent ones. :telemetry.span/3 does the start/stop/exception bookkeeping for you:

defmodule MyApp.LLM do
  def complete(prompt, opts \\ []) do
    metadata = %{
      provider: opts[:provider] || :openai,
      model: opts[:model]
    }

    :telemetry.span(
      [:my_app, :llm, :request],
      metadata,
      fn ->
        response = call_provider(prompt, opts)
        usage = extract_usage(response)

        result_meta =
          Map.merge(metadata, %{
            input_tokens: usage.input_tokens,
            output_tokens: usage.output_tokens,
            cost: usage.cost
          })

        {response, result_meta}
      end
    )
  end
end

:telemetry.span/3 fires [:my_app, :llm, :request, :start] before the function runs and ..., :stop] after, with duration computed for you — and ..., :exception] if the function raises, which matters here because a provider timeout is exactly the failure mode you want in the dashboard, not just the successes. The map your function returns becomes the :stop event’s metadata — which is why the code merges the token counts into metadata by hand before returning it; that’s how they make it into the event without a second :telemetry.execute call. Everything downstream — the Telemetry.Metrics definitions, LiveDashboard, PromEx — is identical to the ReqLLM path; only the event name and the code that produces it changes.

The two numbers worth alerting on

Once the metrics exist, resist the urge to build a dashboard with fifteen panels nobody opens. Two numbers actually change what you do:

Cost per day, trending. A sum metric on cost, viewed daily, catches the failure mode that burns money silently: a prompt template that ballooned after an edit, a retry loop with no backoff hammering the most expensive model on every failure, a feature that got popular faster than anyone expected. This is the same instinct behind checking inference unit economics in technical due diligence — the number that’s fine at ten calls a day and ruinous at ten thousand needs a trend line, not a spot check.

p95 latency per provider. Not the average — the average hides the calls that are actually annoying a user right now. A distribution metric with tags on provider (and model, if you route between them) tells you when a specific provider degrades before your support inbox does. If you’re running an agent loop that retries on failure — the kind of durable, idempotent loop Oban makes a good runtime for — a latency spike on one provider is also the leading indicator that your retry budget is about to get expensive, because every retry is another billed call on top of the original.

Everything else — token counts by endpoint, breakdown by finish reason, per-model histograms — is worth having in the dashboard for debugging, but it’s not worth an alert. If you’re already streaming tokens to the client with LiveView’s async APIs, the same PubSub-friendly instinct applies here: don’t build more real-time surface area than someone is actually going to look at.

One more option worth knowing about

If you want tracing and not just metrics — spans that follow a request through an agent’s tool calls, not just a single provider call — agent_obs is a small Elixir library built specifically for this, with trace_agent/3 and trace_llm/3 helpers that emit OpenTelemetry-compatible spans (with an Arize Phoenix integration via OpenInference conventions). It’s a different shape of problem than what this post covers — cost/latency dashboards versus distributed tracing — but if the next question after this post is “now I need to see the whole agent run, not just one call,” that’s where I’d look first.

Where this fits

None of this requires choosing between “buy Helicone” and “build it yourself” as an all-or-nothing decision — plenty of teams run both, a hosted tool for the LLM-specific views (prompt diffing, eval scoring) and Telemetry.Metrics for the operational numbers that live next to the rest of their Phoenix metrics anyway. What it does mean is that if you’re already on the BEAM — and there are real reasons to be, not just aesthetic ones — the telemetry story for LLM calls is closer to “wire up what’s already there” than “adopt a new platform.”

If you’re building out an AI product’s operational layer and want a second set of eyes on what’s actually load-bearing versus what’s dashboard theater, that’s a conversation I have often as a fractional CTO for AI startups — happy to look at what you’ve got.