Phoenix Secrets: runtime.exs, Fly.io, and LLM Keys

TL;DR: config.exs compiles into your release; runtime.exs runs when the release boots. Put a secret in the wrong one and it either bakes an old value into your Docker image forever, or vanishes the moment you deploy. The fix is mechanical once you see it: read every secret in config/runtime.exs with System.fetch_env!/1, not System.get_env/2 with a fallback — a missing LLM key should crash your boot at 2pm during a deploy, not surface as a silent 500 to a customer at 2am. fly secrets set restarts your machines to apply it; --stage lets you land a secret before the code that reads it ships. Dotenvy gives dev the same .env-file convenience without touching System.get_env at all, which is exactly where the existing guides on this stop short. And an Anthropic or OpenAI key is not “just another env var” — it’s a bearer credential against your billing account, and it deserves narrower handling than your database URL.

The trap: config.exs bakes, runtime.exs boots

Every Phoenix app since 1.6 ships two configuration entry points, and the difference between them is the single most common way a new engineer breaks a deploy. config/config.exs (and its environment siblings, dev.exs/prod.exs) runs at compile time, when mix release builds the artifact. Whatever value System.get_env("ANTHROPIC_API_KEY") returns on the build machine, at build time gets frozen into the release’s compiled config. If you build on CI where that variable happens to be unset, you get nil baked in — permanently, until the next rebuild — no matter what you later set in the deployed environment.

config/runtime.exs is different by design: Phoenix executes it after compilation, right before the application starts, on every boot, in every environment including a running release on a production machine. That’s the only place System.get_env reads the actual environment the process is running in. This is why the generated runtime.exs already handles SECRET_KEY_BASE this way — it’s not a style choice, it’s the only correct place to read a value that has to differ between your laptop and the fly machine that’s actually serving traffic.

The trap is subtle because it doesn’t fail loudly. You put an LLM key in config.exs because that’s the file you had open, mix phx.server works fine locally (your shell has the env var, and dev doesn’t build a release), CI passes, the deploy succeeds — and then the first production request that calls the model gets a 401 from a key that was nil at build time. Nothing in that chain complains until a real user hits it.

Fail at boot, not at 2am: fetch_env! over get_env

Once a secret is correctly in runtime.exs, there’s a second decision that matters just as much: what happens when it’s missing.

# config/runtime.exs — wrong instinct: silently proceed
anthropic_key = System.get_env("ANTHROPIC_API_KEY", "")

This “works.” The app boots. Every request to your agent endpoint returns a vague provider error, or worse, silently degrades to a stub response, and you find out from a support ticket instead of a deploy log.

# config/runtime.exs — crash the boot instead
anthropic_key = System.fetch_env!("ANTHROPIC_API_KEY")

config :sublime_coding, :anthropic, api_key: anthropic_key

System.fetch_env!/1 raises ArgumentError immediately if the variable isn’t set, which means the release refuses to start at all. On Fly, a release that crashes on boot fails the deploy’s health check and the rollout stops — you see it in fly logs and fly status before a single customer request lands on the bad machine, because Fly won’t route traffic to a machine that never comes up healthy. That is the failure mode you want: a loud, deploy-time crash you catch in your terminal, not a quiet, request-time failure someone else catches for you.

This generalizes past LLM keys. Any secret without which the app is meaningfully broken — the database URL, SECRET_KEY_BASE, an LLM key that gates a core feature — should use fetch_env!. Reserve System.get_env/2 with a real default for genuinely optional config: an analytics ID, a feature flag, a port number with a sane fallback. This repo’s own runtime.exs draws exactly that line — PORT gets System.get_env("PORT", "4000") because a default port is harmless, while SECRET_KEY_BASE gets a raise with a message telling you how to generate one, because a missing signing key is not something you want silently defaulted in production.

fly secrets set, staged secrets, and what actually restarts

Fly.io’s secrets are the thing runtime.exs reads at boot — they’re injected as process environment variables on the machine, nothing more exotic. The workflow:

fly secrets set ANTHROPIC_API_KEY=sk-ant-... \
                OPENAI_API_KEY=sk-proj-...

This does two things: it writes the secret into the app’s encrypted vault, then immediately restarts every machine in the app to pick it up — which also resets each machine’s ephemeral filesystem, worth knowing if you keep any scratch state outside a volume. That restart-on-set behavior is why you don’t want to fly secrets set casually against a busy production app mid-incident; it’s a real deploy event, not a config tweak.

Two situations where the default behavior is wrong for you:

Setting a secret before the code that reads it ships. If you’re adding a new ANTHROPIC_API_KEY read in runtime.exs in the same release that needs the secret, setting it normally works fine — the restart-and-the-deploy can happen in either order. But if you want to stage the secret ahead of a deploy without triggering an extra restart cycle right now, --stage sets the value in the vault without touching running machines: fly secrets set ANTHROPIC_API_KEY=sk-... --stage. The staged value applies the next time a machine starts or updates — your next fly deploy picks it up naturally, or you can force it early with fly secrets deploy, which redeploys the current release with the staged secrets without rebuilding the image.

Rotating a leaked or expiring key. Same command, no --stage — you want the restart immediately: fly secrets set ANTHROPIC_API_KEY=sk-new-.... The old key stops being read the moment the new machines come up. This is also the moment fetch_env! earns its keep on the other end: if you typo the new key name, the next deploy crashes at boot instead of quietly running every LLM call against a key that no longer exists.

fly secrets list shows names, digests, and set-times — never plaintext values, by design. If you need to confirm what a secret’s value is, you kept it somewhere else (a password manager, not Fly) or you’re regenerating it, not reading it back.

Dev: Dotenvy gives you .env files without touching System.get_env

Everything above covers the release. Locally, you don’t want to export five LLM keys into your shell every session, and you don’t want a .env file that quietly leaks into System.get_env and gets read the same way production secrets are — because then a stray git add .env is a production-shaped incident. This is precisely the gap the existing Phoenix-secrets writeups leave: they cover the compile/runtime split, then wave at “use a .env file in dev” without saying how to keep that convenience from bleeding into your runtime read path.

Dotenvy closes it cleanly because it’s read-only by design — it never calls System.put_env/2, so .env-sourced values are invisible to plain System.get_env or System.fetch_env! calls elsewhere in your app. You read them only through Dotenvy’s own env!/2, which forces you to declare the type and the requirement inline:

# config/runtime.exs
import Config
import Dotenvy

env_dir = System.get_env("RELEASE_ROOT") || Path.expand(".")

source!([
  Path.absname(".env", env_dir),
  Path.absname(".#{config_env()}.env", env_dir),
  System.get_env()
])

config :sublime_coding, :anthropic,
  api_key: env!("ANTHROPIC_API_KEY", :string!)

source!/1 layers files in order — a shared .env, then an env-specific .dev.env or .test.env overriding it, then the real System.get_env() last so production’s actual environment always wins over anything a stray .env file might contain. :string! (the bang type, not the bang function) enforces non-empty, same intent as fetch_env! — missing or blank crashes the boot instead of handing you nil. In prod there’s typically no .env file on disk at all, so source!/1 layers in nothing but System.get_env(), and every value still flows through the exact same env! call — one code path for both environments, which is the actual point of doing it this way instead of branching on config_env() yourself.

.env and .dev.env still belong in .gitignore, same as they always did — Dotenvy changes how the value gets read, not whether the file belongs in git.

LLM keys specifically: narrower rules than a database URL

Everything so far applies to any secret. LLM provider keys earn extra caution for two reasons that don’t apply to, say, your Postgres password.

They’re bearer credentials against a metered bill, not a fixed resource. A leaked database URL gets you rows; a leaked Anthropic or OpenAI key gets an attacker your billing account. When a corporate-tier key leaks, threat actors run high-volume inference workloads against it and drain the account before anyone notices — there’s no row count to bound the damage, only your provider’s rate limit and however long detection takes. GitGuardian’s most recent scan of public GitHub found AI-service secrets growing 81% year over year, faster than almost every other credential category they track, which tells you the scanning bots are tuned for exactly this key shape now — sk-ant- and sk-proj- prefixes are as fingerprintable as AWS’s AKIA.

Never in config.exs, full stop — including for local convenience. The compile-time trap above is annoying when it’s SECRET_KEY_BASE; it’s actively dangerous for an LLM key, because a key baked into a compiled release artifact ships with every copy of that artifact — any Docker image, any build cache, any layer you push to a registry. Read it in runtime.exs — a direct fetch_env! in prod or env! behind Dotenvy in dev.

Scope the key to what actually needs it, and rotate on a schedule you keep. Most providers let you name and scope API keys per project or per use case — use that, so a leaked key from your staging agent isn’t also your production billing key. And rotate proactively rather than reactively; the providers’ own detection catches a lot of leaks fast, OpenAI’s partner program with GitHub revokes recognized key patterns within minutes of detection, but that safety net only covers keys that match a known pattern in a place GitHub scans — it does nothing for a key sitting in your Fly secrets vault that an ex-contractor still has a local copy of.

This post is about the boring, load-bearing layer: getting the key into the process safely. It doesn’t cover what happens after your application holds the key — whether the model itself ever sees it, whether a tool call can leak it back out through a crafted prompt. I’ve written that half separately: secrets management for AI agents covers the broker pattern that keeps a credential scoped to the tool that needs it and out of the model’s context entirely. Different layer, same root discipline — this post gets the key from Fly into your BEAM process correctly; that one governs what your code does with it once it’s there.

The full picture, in order

Put together, the sequence for adding a new LLM key to a Phoenix app on Fly looks like this:

1. Add System.fetch_env!("ANTHROPIC_API_KEY") (or env! via Dotenvy in dev) to config/runtime.exs — never config.exs.
2. fly secrets set ANTHROPIC_API_KEY=sk-ant-... before or alongside the deploy that reads it; use --stage if you want to land the secret ahead of the code without an extra restart.
3. Deploy. If the key is missing or mistyped, the release fails to boot and the rollout stops — check fly logs before assuming it’s a code problem.
4. Scope the key narrowly at the provider, note it in whatever runbook tracks what’s set where, and put its rotation on the same calendar as your other production credentials.

None of this is exotic Elixir. It’s the same discipline that governs SECRET_KEY_BASE in every mix phx.new app, applied on purpose to the credential that’s most likely to be the one you added under deadline pressure without thinking about where it goes. If you’re standing up an AI feature on Phoenix and want a second pair of eyes on the security posture around it — this layer and the ones above it, like the AI-native security stack or the 50-line agent loop this key usually feeds — that’s a conversation worth having before the first deploy, not after the first leak.