Throttling LLM Calls in Elixir Before You Hit 429

TL;DR: LLM providers rate-limit you on two dimensions at once — requests per minute (RPM) and tokens per minute (TPM) — and only one of those is something you can count before you act. You know exactly how many requests you’re about to send. You do not know how many tokens a response will cost until it comes back, because output length isn’t fixed. A limiter that only counts requests sails straight past a TPM ceiling and gets 429’d anyway. This post is app-side admission control: size a Hammer bucket against your provider’s real TPM/RPM numbers, reserve an estimate before you call, reconcile against the real usage once it lands, and give each tenant a fair slice of one shared provider budget. It’s the piece that runs before the call goes out — what to do when a call still 429s mid-run is a separate, already-written problem, and I’ll cede it explicitly.

Two scope fences before the code

This post has a narrow job, and it’s worth being precise about what it isn’t, because the adjacent posts already cover the other two pieces.

First: retries and backoff aren’t here. The Oban agent-runtime post covers what happens when a 429 gets through anyway — snoozing the job, respecting Retry-After, distinguishing a transient rate limit from a permanent 400. That’s reactive. This post is proactive: the goal is that the request in that post’s failure scenario never gets sent in the first place, because your own accounting already knew it would blow the budget.

Second: this isn’t the multi-tenant architecture post. The multi-tenant SaaS post has a rate-limit isolation table with one blunt recommendation: if you can afford to give each tenant its own provider project, do that — a noisy tenant then only throttles itself, because Anthropic meters at the organization level with per-workspace overrides, and OpenAI allocates per project — in neither case is the limit attached to the raw key. That post has no Hammer code in it because the architectural fix makes the code moot. This post is for the case where you can’t do that yet — one shared provider key, one shared TPM/RPM budget, multiple tenants or workloads drawing on it — and you need software to enforce fairness that the provider won’t enforce for you.

Rate limits are two-dimensional, and only one dimension is countable in advance

Here’s the asymmetry that makes this harder than a normal API rate limiter. A request counter is trivial: you know before you dial out that this is request number 47 this minute. Providers also cap the token dimension, and Anthropic states plainly that it rate-limits on requests per minute, input tokens per minute, and output tokens per minute simultaneously, per model class — hit any one of the three and you get a 429, regardless of how much headroom you have on the other two.

The RPM ceiling is easy: increment a counter, compare to a limit, done. The TPM ceiling is the one that actually bites in practice, because you don’t know a response’s token cost until the response arrives. Input tokens you can estimate reasonably well from the prompt. Output tokens are the model’s choice — a summarization call might return 40 tokens or 4,000 depending on what’s in the document, and nothing about the request itself tells you which in advance. A limiter that only counts requests will happily let through five calls in one minute, each of which independently looks fine, and collectively blow well past the TPM ceiling. You find out you were over budget the same moment the provider does — as a 429 — which is exactly the failure mode admission control is supposed to prevent.

That’s the spine of everything below: RPM is a counting problem, TPM is an estimation-and-reconciliation problem, and treating them the same way is the bug.

Sizing the bucket against your real numbers

Don’t guess at the ceiling — read it off your own account. Anthropic publishes its standard tier limits, and they’re a useful concrete anchor even if your account sits on a different tier or provider: on the Start tier, Claude Sonnet 5 gets 1,000 RPM, 2,000,000 ITPM, and 400,000 OTPM; Build takes RPM to 5,000 and the token limits to 5,000,000 ITPM / 1,000,000 OTPM — note RPM scales faster than the token dimensions, so a bucket sized by multiplying every number by the same factor will be wrong on the two that matter; Scale doubles all three again (current numbers here — Anthropic also excludes cached input tokens from ITPM on most models, which matters a lot if you’re using prompt caching and should size your bucket off uncached input volume, not raw prompt size).

The mechanical takeaway, independent of provider: your limiter’s limit argument is not a number you pick, it’s a number you read off your provider console and keep in sync with. Anthropic’s own Rate Limits API lets you read the configured limits programmatically instead of hardcoding them — note it authenticates with an Admin API key, not the standard key your app already holds, so this is a separate credential to provision. Worth wiring up if you’re on a tier that changes as your usage history grows, since a stale hardcoded limit either throttles you below what you’re actually allowed or, worse, lets you sail past a limit that got tightened.

Set your app’s ceiling a notch under the provider’s, not equal to it. If the provider says 400,000 OTPM, budget your own bucket at 90% of that — headroom for the estimation error the next section is about to explain you can’t fully eliminate.

Hammer, past the one-line intro

The libraries roundup covers Hammer in one line: a clean rate limiter with pluggable backends, cheap to add before you’re under load. Here’s the part that matters for TPM specifically. Current Hammer (v7.4.0) is a use-based module you define once:

defmodule MyApp.LLMLimiter do
  use Hammer, backend: :ets
end

Started in your supervision tree like anything else:

children = [
  {MyApp.LLMLimiter,
   clean_period: :timer.minutes(1)}
]

The core call is hit/3 — a key, a window size in milliseconds, and a limit:

case MyApp.LLMLimiter.hit(
       "rpm:anthropic",
       :timer.minutes(1),
       1_000
     ) do
  {:allow, _count} -> :ok
  {:deny, _retry_ms} -> {:error, {:rpm_exhausted, ms}}
end

One thing to get right before writing a word more, because it changes what you configure: Hammer’s default algorithm is a fixed window counter, and a fixed window lets a burst cluster at the boundary between two windows, briefly doubling your effective rate. For RPM that’s a minor edge case. For a hard-money TPM ceiling it isn’t.

Hammer ships four other algorithms alongside it, and one of them is the one you actually want here:

defmodule MyApp.LLMLimiter do
  use Hammer,
    backend: :ets,
    algorithm: :token_bucket
end

A token bucket refills continuously instead of resetting at a boundary, which is the same model Anthropic’s own limiter uses. The hit/4 cost argument works identically, so nothing else in this post changes. Everything below is written against :token_bucket; if you leave the default in place, keep a wider safety margin to absorb the boundary burst.

That RPM check above is the easy half. The token half needs hit/4, which takes a custom increment instead of the implicit 1:

MyApp.LLMLimiter.hit(
  "tpm:anthropic",
  :timer.minutes(1),
  400_000,
  estimated_tokens
)

This is what makes Hammer usable for TPM at all: you’re not counting calls, you’re spending down a budget, and the increment argument is where the token count goes in.

Do you get your reserved tokens back if you overestimated?

No — not safely, and the reason why is the actual design decision in this section. The pattern is estimate-then-reconcile: before the call, estimate its token cost and reserve that much against the bucket; after the call, compare the estimate to what the provider actually billed you and settle the difference. Throttle.call/2 below is meant to sit directly in front of whatever makes the actual request — if that’s a ReAct tool-call loop, this wraps the one line in it that dials out to the model, not the loop itself.

defmodule MyApp.LLM.Throttle do
  @moduledoc """
  Admission control in front of every LLM call.
  Reserves an estimate, settles the real usage
  after the response lands.
  """

  alias MyApp.LLMLimiter

  @rpm_limit 1_000
  @tpm_limit 400_000
  @window :timer.minutes(1)

  def call(prompt, opts \\ []) do
    with :ok <- check_rpm(),
         {:ok, estimate} <- reserve_tpm(prompt),
         {:ok, response} <- do_call(prompt, opts) do
      settle_tpm(estimate, response)
      {:ok, response}
    end
  end

  defp check_rpm do
    case LLMLimiter.hit(
           "rpm:anthropic", @window, @rpm_limit
         ) do
      {:allow, _} -> :ok
      {:deny, ms} -> {:error, {:rpm_exhausted, ms}}
    end
  end

  defp reserve_tpm(prompt) do
    estimate = estimate_tokens(prompt)

    case LLMLimiter.hit(
           "tpm:anthropic", @window,
           @tpm_limit, estimate
         ) do
      {:allow, _} -> {:ok, estimate}
      {:deny, ms} -> {:error, {:tpm_exhausted, ms}}
    end
  end

  # Rough: ~4 chars/token for English prose, plus a
  # fixed output allowance. Tune this against your
  # own prompt/response shape — see settle_tpm/2 for
  # why a loose estimate is fine as long as you settle.
  defp estimate_tokens(prompt) do
    input = div(String.length(prompt), 4)
    input + 500
  end

  defp settle_tpm(estimate, response) do
    actual = response.usage.input_tokens +
      response.usage.output_tokens

    shortfall = actual - estimate

    if shortfall > 0 do
      # We under-reserved. Charge the difference now
      # so the NEXT caller sees an accurate budget,
      # even though this call already went through.
      LLMLimiter.hit(
        "tpm:anthropic", @window,
        @tpm_limit, shortfall
      )
    end

    # If shortfall < 0 (we over-reserved), we do
    # NOT try to give tokens back. A fixed-window
    # counter has no safe decrement under concurrent
    # writers — another request could read the freed
    # headroom between your check and your refund,
    # double-spending the same tokens. Overestimating
    # costs you a little slack; that's the trade.
    :ok
  end

  defp do_call(prompt, opts) do
    ReqLLM.generate_text(
      "anthropic:claude-sonnet-5", prompt, opts
    )
  end
end

Three things worth being explicit about. First, a denied hit/4 still spends its increment — Hammer bumps the counter before it compares against the limit, so a rejected reservation pushes the bucket further underwater and only the refill recovers it. That’s fail-safe rather than fail-open, since it over-counts instead of under-counting, but it means a saturated bucket stays saturated a little longer than the arithmetic suggests. It also means an RPM check that passes followed by a TPM check that denies has already burned an RPM slot for a request that never went out; if that ordering bothers you, check the cheaper dimension last. Second, settle_tpm/2 only ever charges more, never less — if you underestimated, the next caller pays for your mistake by finding a tighter bucket, which is exactly the outcome you want (the budget stays honest going forward). Anthropic does the same thing on its side — its docs say ITPM limits are “estimated at the beginning of each request, and the estimate is adjusted during the request to reflect the actual number of input tokens used,” which is estimate-then-reconcile described by the provider you’re reconciling against. If you overestimated, you don’t get the slack back, because there’s no safe way to decrement a shared counter without a race: between your “I over-reserved, refund N” read and write, some other process could have already spent the headroom your refund is about to reopen, and now two callers think they have room for the same tokens. Rounding your estimate a little high and eating the loss is cheaper than building compare-and-swap machinery around a rate limiter. Third, the reservation happens before the call and the settlement happens after — the ordering matters, because reserving low and reconciling never is just a request counter wearing a token costume.

If your traffic is bursty enough that this margin genuinely costs you throughput, that’s the point where Anthropic’s response headers — anthropic-ratelimit-input-tokens-remaining, anthropic-ratelimit-output-tokens-remaining, and their -reset counterparts, documented alongside the rate limit tables — become worth reading back into your own bucket after every real call, so your local estimate self-corrects against the provider’s authoritative count instead of drifting on your own estimate_tokens/1 heuristic alone.

Per-tenant fairness on one shared budget

Everything above treats the whole app as one caller against one bucket. That’s fine for a single-tenant app. It falls apart the moment several tenants share one provider key — which, per the scope fence at the top, is exactly the situation you’re in if you can’t give each tenant its own provider project. The failure is asymmetric: whoever is largest and least latency-sensitive drains the shared ceiling first, and the tenants who notice are the small interactive ones whose requests were never the problem.

The fix is the same Hammer mechanism, keyed one level deeper — a per-tenant bucket nested inside the org-wide one:

def reserve_tpm(tenant_id, prompt) do
  estimate = estimate_tokens(prompt)

  with {:allow, _} <- LLMLimiter.hit(
         "tpm:tenant:#{tenant_id}",
         @window, tenant_tpm_share(tenant_id),
         estimate
       ),
       {:allow, _} <- LLMLimiter.hit(
         "tpm:anthropic", @window,
         @tpm_limit, estimate
       ) do
    {:ok, estimate}
  else
    {:deny, ms} -> {:error, {:tpm_exhausted, ms}}
  end
end

Two checks, both must pass: the tenant’s own slice, and the org-wide ceiling underneath it. tenant_tpm_share/1 is a function you own — a flat per-tenant cap, a plan-tier-based cap, or a proportional share of whatever’s currently free. This is real multi-tenant-SaaS territory: it’s the same fairness problem as noisy-neighbor CPU throttling, just applied to a provider’s TPM ceiling instead of a CPU scheduler, and it’s exactly the kind of shared-resource design question I help AI startups work through as a fractional CTO before it turns into a support queue full of “why did my request fail, I didn’t do anything unusual” tickets.

A per-request check tells you allow or deny for one call; it says nothing about whether you’re trending toward saturation. Once Phoenix’s LLM telemetry wiring is in place, graph bucket occupancy — remaining TPM and RPM headroom over time — beside the usage events it already emits, so a tenant creeping toward their share shows up as a trend line rather than a wave of {:deny, _}.

A shared TPM ceiling behind a multi-deployment proxy

One more shape worth naming: teams running multiple provider deployments behind one internal proxy — several Azure OpenAI deployments, say, each with its own TPM allocation — to get more aggregate throughput than any single deployment’s ceiling allows. The Hammer pattern above extends cleanly: key each deployment’s bucket separately, and route a request to whichever deployment currently has headroom rather than always hitting the first one in the list. That’s a load-balancing problem layered on top of the admission-control problem this post covers, not a replacement for it — each deployment still needs its own hit/4 check sized to its own real limit, for the same estimate-then-reconcile reasons above.

Read this next

If a request does slip through and still comes back a 429, the Oban agent-runtime post covers snoozing, backoff, and idempotent resume — the reactive half this post deliberately left alone. And if you’re deciding whether shared-budget fairness code is even the right call versus giving tenants their own provider projects, the multi-tenant SaaS post is the architectural decision that comes before any of this code gets written.