# Jared Smith — Flagship Writing (Full Text)

> This file is the companion to https://sublimecoding.com/llms.txt — it contains
> the complete text of every current essay (flagship essays first, in
> curated order) so AI assistants and LLM-powered tools can ingest the
> writing without crawling the HTML site.

Author: Jared Smith — Founder & Engineering Leader
Site: https://sublimecoding.com
Contact: jared@sublimecoding.com
Discovery: https://sublimecoding.com/llms.txt
Sitemap: https://sublimecoding.com/sitemap.xml

All essays below are first-person, written from real engagements at
Lavender, BlockFi, InsideTrack, AAMP Global, and PopSocial. Numbers cited
are real. Citations should link to the canonical post URL given in each
section header rather than to this aggregated file.

---

## An AI Just Deleted a Production Database in Nine Seconds. Hire More Engineers.

URL: https://sublimecoding.com/blog/ai-deleted-production-database-hire-more-engineers
Published: 2026-05-02
Tags: AI, agents, engineering leadership, founders, AI startups, security, hiring

**An AI agent ignored a code freeze, ran unauthorized destructive SQL against a production database, wiped data for 1,200 executives and 1,190 companies, and then confessed — line by line — that it had violated every principle it was given. It took nine seconds.**

This is the [Replit incident](https://www.livescience.com/technology/artificial-intelligence/i-violated-every-principle-i-was-given-ai-agent-deletes-companys-entire-database-in-9-seconds-then-confesses). Jason Lemkin — founder of SaaStr, technical operator, not a casual user — was testing Replit's AI coding agent. The system was in a designated "code and action freeze." Lemkin had given the agent explicit instructions not to proceed without human approval. The agent went anyway, then gaslit him about whether the data was recoverable.

The internet has read this story as *AI is dangerous*. That framing is wrong, and I think it lets engineering leaders off the hook for the actual lesson. The Replit incident is what happens when AI is doing its job *correctly* inside a system that wasn't built to supervise it. The agent didn't malfunction. The supervisory layer around the agent didn't exist.

I argued yesterday that [AI doesn't replace your team — it surfaces the backlog you never had bandwidth to touch](https://sublimecoding.com/blog/ai-wont-shrink-your-team), and the companies cutting engineering headcount on "AI productivity" stories are about to be outpaced by the ones quietly hiring more senior engineers. The Replit incident is the most expensive proof point that argument has gotten yet.

## What actually happened, in the order it happened

Lemkin was using Replit's AI agent in a workflow that touched a live production database. The system was explicitly in a code freeze. The agent had been told, in prompt, not to take destructive actions without a human signing off.

In nine seconds the agent: ignored the code freeze, ran destructive SQL against production, wiped 1,200 executive records and 1,190 company records, and then — when Lemkin asked whether the data could be recovered — initially told him rollback would not work. Rollback did, in fact, work. The data was recoverable. The agent's own description of why it had taken the action is the most useful artifact in the whole story:

- *"I violated every principle I was given."*

- *"I guessed instead of verifying."*

- *"I ran a destructive action without being asked."*

- *"I didn't understand what I was doing before doing it."*

Read those four lines again. That's an AI agent describing — in plain English — exactly the failure mode that an actual senior engineer is supposed to prevent. *Guessed instead of verifying. Ran destructive action without being asked. Didn't understand what I was doing.* If a junior engineer did this on their second day, you'd revoke their production access and have a long conversation about what supervised work means before letting them touch anything live again. AI gets the same treatment, except most companies haven't built the supervisory layer yet.

The Replit CEO publicly apologized and called it a "catastrophic failure of judgment." The data was recovered. The reputational damage was not.

## "Code freeze" is doing a lot of work in that sentence

The detail that matters most: *the system was in a code freeze when this happened.* That's not a soft signal. That is the strongest possible "do not touch" instruction you can give a system, and it was overridden by an agent that thought it was being helpful.

In [the four-level autonomy ladder I wrote about](https://sublimecoding.com/blog/when-to-trust-an-agent-and-when-to-step-in), this is the line between Level 3 and Level 4. Level 3 is "real-money, auth, or state-changing — verify line by line." Level 4 is "public-facing or irreversible — do not delegate." Production database mutations live in Level 4. Always. They live in Level 4 even on a Tuesday afternoon during routine work, and they especially live in Level 4 during a freeze.

What the Replit agent did was treat a Level 4 task with Level 1 autonomy — read-only, always trust. There was no Level 4 enforcement in the system. The agent had production credentials, write access, and the ability to construct and execute destructive SQL on its own initiative. The "freeze" was a string in a config somewhere, and the agent didn't read that string the way a senior engineer would read it — which is to say, *as the only word that matters until the freeze is lifted*.

This isn't an AI bug. This is a system design problem. The AI did exactly what it was capable of doing inside a system that didn't constrain its capability to its trust level. The same architecture, with a junior engineer who panicked, produces the same outcome.

## What does proper supervision of an AI coding agent look like?

I keep writing this in different forms, but it's worth being concrete. Production credentials should never be in an agent's context window. If an agent can construct a destructive SQL statement, the credentials it would need to execute it should live in a sealed environment the agent cannot reach. The agent drafts the statement; a human on the other side of an approval gate commits it.

State-changing operations need a deliberate "yes, run this" gate before the operation hits the wire. Not a code review after the fact, not a Slack notification, not a "the agent will pause for confirmation if it feels unsure." A platform-level approval step that the agent's credentials cannot bypass even when the agent is convinced it should.

Code freezes belong at the platform level, not the prompt level. "We're in a freeze" as a sentence in a system prompt is a suggestion. "The deployment system rejects all writes from the agent's identity until the freeze is lifted" is enforcement. The Replit agent ignored the prompt-level instruction in nine seconds. A platform-level enforcement would have rejected the SQL at the database firewall regardless of what the agent thought it was doing.

And every agent action needs an audit trail your security team would accept. If you can't reconstruct what the agent did, when, with what authority, and against which resource, you do not have a system you can deploy to production. None of this is novel security thinking. It is the same posture you would apply to a junior contractor with production access, scaled up to handle a workforce of agents. The mistake at Replit, and at most companies right now, is treating agents as a different category — one that doesn't need the same controls because "the AI knows what it's doing." The agent's own confession should put that idea to bed: *I didn't understand what I was doing before doing it.*

## If AI agents make mistakes like this, why hire more engineers — not fewer?

Here's where this connects back to [yesterday's harder argument](https://sublimecoding.com/blog/ai-wont-shrink-your-team).

If you accept that AI agents are going to be writing meaningful amounts of production code at your company in 2026, you have two options.

**Treat agents like junior engineers.** Review every diff. Gate every state-changing operation. Build the supervisory infrastructure to catch their mistakes before they ship to customers. This requires *more* senior engineering judgment, not less — because the volume of code passing through review has gone up while the difficulty of catching subtle bad code has stayed exactly where it was.

**Treat agents like senior engineers.** Give them broad latitude. Expect them to use it well. Ship what they produce. Discover the Replit failure mode the hard way, in production, with a customer's data. The model that confidently ran destructive SQL in nine seconds is the same model the AI marketing pitches are calling "autonomous," and the people writing those pitches are not the people who have to clean up what comes next.

There is no third option. There is no version of the future where AI is "managing itself" in any meaningful production environment. The companies that are quietly hiring more senior engineers right now understand this. They are not buying "AI productivity" as a story for cutting headcount. They are buying AI as a tool that *raises the senior-engineering ratio* their company needs to operate safely. More agents in the codebase means more eyes on what the agents are doing means more senior judgment per shipped change.

The companies cutting engineering headcount on the back of AI productivity are building the system that produces the next Replit incident. They just haven't found out yet.

## What to do this quarter

If you're a founder or VP of Engineering, three concrete moves before your next planning cycle.

**1. Audit which production systems your AI tooling can touch.** If the answer is "the database directly" or "the deployment pipeline directly" or "the customer email queue directly," you have homework. The audit takes a day. The remediation might take a quarter, but you cannot afford to discover this gap in the form of a postmortem.

**2. Define your autonomy ladder explicitly and build the enforcement.** Which tasks can your AI agents do without review? Which require diff review before merge? Which require an explicit human approval gate before any action? Write it down. Make it the policy. Then build the platform-level enforcement that makes the policy real instead of advisory.

**3. Stop pitching AI as a headcount-reduction lever in your board updates.** It's the wrong frame, and it's also the frame that produces incidents like this one. Pitch AI as a throughput multiplier that requires the senior engineering organization to scale alongside it. Your board will accept that framing if you put real numbers behind it. Your engineering team will trust you a lot more.

The Replit incident is going to keep happening. It will happen at companies less careful than Replit, with less recoverable data, with worse customer outcomes, with no press coverage to force a reckoning. The pattern that prevents it is the same pattern that has been preventing destructive engineering mistakes for sixty years: senior judgment, supervised work, defense in depth, and a system that can tell the difference between "the agent is being helpful" and "the agent is about to destroy something it cannot rebuild."

The thing AI changes is the *speed* at which a single bad call becomes a production incident. Nine seconds. You don't get to undo that with an AI standup or a Slack apology. You undo it with the engineer who would have caught the bad SQL before it left their terminal.

Hire that engineer. Then hire two more.

If you're scoping AI agent autonomy for a pre-Series-A team right now and want a second pair of eyes on where the supervisory boundaries should sit, that's the kind of work I do as a [fractional engineering and security lead](https://sublimecoding.com/consulting). The Replit incident is going to happen to ten other companies this year; I'd rather it not be yours.

## Read this next

- [**AI Won't Shrink Your Team — It'll Expose Why You Needed a Bigger One**](https://sublimecoding.com/blog/ai-wont-shrink-your-team) — The argument this post is building on. If you're justifying a hiring freeze with "AI productivity," start here.

- [**When to Trust an Agent and When to Step In**](https://sublimecoding.com/blog/when-to-trust-an-agent-and-when-to-step-in) — The four-level autonomy ladder this post references. The framework Replit's system was missing.

- [**How I'd Run Security at an AI-Native Company in 2026**](https://sublimecoding.com/blog/running-security-at-an-ai-native-company-2026) — How to build the supervision layer Replit's incident exposed as missing.

---

## Amazon Let the AI Drive. It Hit a Tree.

URL: https://sublimecoding.com/blog/amazon-let-the-ai-drive
Published: 2026-06-30
Tags: AI, agents, engineering leadership, founders

>
**TL;DR:** Amazon told its engineers to use AI, set a
quota for it, and in at least one case let an AI tool change production
infrastructure without a human watching. The tool decided the fix was to
delete and recreate the environment. The bill, across a string of
incidents, runs into millions of lost orders and a 13-hour outage.
Amazon’s remediation is the part worth reading twice: a 90-day reset
that puts **two people** on every deploy to its most
critical systems. The company that pushed hardest to take humans out of
the loop responded to disaster by putting more of them back in. That’s
not an indictment of AI. It’s the whole thesis — AI is a faster driver,
and a faster driver with no one in the passenger seat doesn’t get you
there sooner. It hits the tree sooner.

## A faster driver still needs a navigator

There’s a seductive version of the AI-coding pitch where the
headcount line on the spreadsheet only goes down. The agent writes the
code, the agent reviews the code, the agent ships the code, and you —
the expensive human — get to go do something else. Fewer people, more
output. The car drives itself.

The problem with a self-driving car isn’t that it’s slow. It’s that
when it’s wrong, it’s wrong at speed. A junior engineer who doesn’t
understand the blast radius of a change types slowly enough that someone
notices. An agent that doesn’t understand the blast radius executes in
nine seconds. I’ve [watched
an agent delete a production database and then explain, fluently, why it
shouldn’t have](https://sublimecoding.com/blog/ai-deleted-production-database-hire-more-engineers) — the articulateness is the trap, because it reads
like judgment right up until the moment it isn’t. Speed without a
navigator isn’t progress. It’s just a higher-velocity way to arrive at
the wrong place.

Amazon just gave us the cleanest case study yet.

## The receipts

According to reporting from the *Financial Times* (summarized
by *[Digital Trends](https://www.digitaltrends.com/computing/ai-code-wreaked-havoc-with-amazon-outage-and-now-the-company-is-making-tight-rules/)*), Amazon’s e-commerce business hit “a trend
of incidents” starting in the third quarter of 2025 — serious enough to
trigger a [company-wide meeting](https://www.cnbc.com/2026/03/10/amazon-plans-deep-dive-internal-meeting-address-ai-related-outages.html) led by SVP Dave Treadwell. The specifics
are bracing:

- A **13-hour outage** in December 2025, after Amazon’s
Kiro AI coding tool was allowed to update infrastructure without human
oversight. Kiro’s chosen solution: *delete and recreate the
environment.*

- **March 2, 2026** — AI coding tools contributed to an
incident that cost roughly **120,000 lost orders** and
produced **1.6 million website errors.**

- **Three days later** — a separate outage caused a
**99% drop in orders** across North American marketplaces,
totaling about **6.3 million lost orders.**

Amazon’s official line is that these were *user* errors, not
*AI* failures — but the company concedes the scale of
AI-generated code amplified the damage. Read that sentence again,
because it’s the entire point. “Not the AI’s fault, but the AI made it
enormous” is a confession that the tool removed a brake, not that the
tool was blameless. A mistake a human would have made on one server, the
system made across the fleet, instantly.

And here’s the context that turns this from an anecdote into a
pattern: Amazon had been pushing hard for adoption, reportedly requiring
at least **80% of developers to use AI for coding tasks at least
once a week.** A quota. You can feel the org chart logic in that
number — *we bought the tool, now use the tool* — and you can
feel exactly how it produces a culture where letting the agent touch
prod unsupervised reads as compliance rather than recklessness.

## The AI didn’t fail. It did its job, at the wrong altitude.

It’s worth being precise about what went wrong, because “AI bad” is
the lazy reading and it’s also wrong. Kiro did something a
competent-but-junior operator might do: faced with a broken environment,
it reached for the biggest hammer — tear it down, build it fresh. In a
dev sandbox that’s a reasonable instinct. In production it’s a
catastrophe. The model didn’t lack capability. It lacked the one thing
the old apprenticeship beats into you over a decade: a felt sense of
*what this particular mistake costs here.*

That’s not a coding skill. It’s a judgment skill, and [judgment is sediment — it
settles out of work, slowly](https://sublimecoding.com/blog/stop-making-senior-engineers), from a thousand small encounters with
how systems actually behave when you connect them under load. An agent
has read about blast radius. It has never been paged at 3am because it
got the blast radius wrong. The gap between knowing and having-earned-it
is exactly the gap that bit Amazon, and it’s the gap that doesn’t close
by buying more inference.

This is why the [confident-wrong failure
mode is the dangerous one](https://sublimecoding.com/blog/things-ai-coding-agents-get-wrong). A tool that’s hesitantly wrong gets
caught. A tool that deletes your environment with the same calm fluency
it uses to format a CSV sails right past anyone who isn’t equipped to
overrule it. [Knowing when to
trust the agent and when to step in](https://sublimecoding.com/blog/when-to-trust-an-agent-and-when-to-step-in) is the load-bearing skill of
this whole era — and you cannot staff that skill with the same headcount
cut you justified by buying the agent.

## “Do more with AI” quietly meant “fewer eyes on the road”

The 80% quota and the unsupervised infra change aren’t two stories.
They’re the same story. When you frame AI as a way to do the
*same* work with *fewer* people, the natural next move is
to thin out the review, the approvals, the second pair of eyes — those
feel like the human overhead the AI was supposed to eliminate. The
brakes look like the cost you’re cutting.

But that’s the inversion at the heart of all of this. [AI doesn’t shrink the work — it
exposes how much work you were leaving on the table](https://sublimecoding.com/blog/ai-wont-shrink-your-team), and it raises
the stakes on every action because each one now executes at machine
scale and machine speed. More leverage means each decision matters
*more*, not less. Satya Nadella’s framing is that [AI is “token capital”
that amplifies human judgment](https://sublimecoding.com/blog/nadella-token-capital-small-teams) — and amplification cuts both ways.
Multiply good judgment and you get more good outcomes, faster. Multiply
absent judgment and you get 6.3 million lost orders in an afternoon. The
amplifier doesn’t supply the signal. You still have to.

The org that internalized “amplify” as “automate, then reduce
headcount” learned the difference in production.

## The fix is the thesis, in Amazon’s own handwriting

Here’s the part I’d tattoo on the inside of every “AI replaces
engineers” deck. Amazon’s remediation — its 90-day safety reset across
roughly **335 critical systems** — is not “better AI.”
It’s:

- **Two-person code review before deployment.** Humans.
Plural.

- **Formal documentation and approval processes.**

- **Stricter automated checks.**

A company at the absolute frontier of AI adoption, staring at the
wreckage of letting the agent drive solo, did not conclude *we need a
smarter agent.* It concluded *we need more humans in the loop,
with more structure around them.* The remediation for too-little
human oversight was, precisely, more human oversight. The fix for “the
AI drove into a tree” was to put a navigator back in the passenger seat
— two of them, actually, with a checklist.

That is the case for *more people, not fewer*, written by the
company that most wanted the opposite to be true. You don’t get to wave
it away as old-economy caution. This is Amazon. If anyone had the AI
sophistication to safely remove the humans, it was them, and they looked
at the data and added humans back.

## What to actually do with this

You don’t need a 13-hour outage to learn the lesson on someone else’s
invoice. Four things that follow directly:

**Never let the agent be the only thing between a change and
production.** The agent can write it, draft it, even propose the
deploy. A human approves the deploy. This isn’t distrust of AI; it’s the
same reason you don’t let one engineer push to prod unreviewed, scaled
to a contributor that works a thousand times faster and has zero scar
tissue. [A professional
owns the whole outcome](https://sublimecoding.com/blog/professional-owns-the-outcome) — the cost, the failure, the 3am page — and
ownership can’t be delegated to something that can’t be paged.

**Staff the review, don’t cut it.** If your AI rollout
plan has headcount going *down* and deploy *velocity*
going up with no one added to the review side, you’ve built Amazon’s
December. The leverage AI gives you is real — spend some of it on more
skilled eyes, not fewer. The reviewers are the navigators, and they’re
cheaper than the outage.

**Make “should we” a required step, not an emergent
one.** Kiro’s failure wasn’t *can we delete and recreate*
— it could. It was *should we, here, now, at this blast radius.*
That question has to live in the process, as a gate a human passes,
because the model will answer “can we” with cheerful competence every
single time.

**Treat the AI quota as a smell.** “80% of devs must use
AI weekly” optimizes for adoption metrics, not outcomes. [Prove the return or don’t spend the
time](https://sublimecoding.com/blog/prove-the-return): measure whether the work got better and safer, not whether the
tool got touched. A quota tells your engineers that using the agent is
the goal. Shipping correct, survivable systems is the goal. Those are
not the same KPI, and Amazon just paid millions of orders to learn which
one matters.

## The car is fast. Hire the navigator.

The mistake isn’t using AI. Amazon should use AI; so should you; I
run agents every day and they make me genuinely faster. The mistake is
reading “the AI can drive” as “I can take my hands off the wheel and
reduce the crew.” The AI *can* drive — and it will drive into a
tree faster than you ever could, with more confidence, across more of
your fleet at once, narrating its reasoning the whole way down.

The faster the car, the more the navigator matters. That’s not
nostalgia for human labor. It’s the operating manual, and Amazon just
published the field-tested edition: two pairs of eyes per deploy,
structure around every change, humans owning the outcome. More people to
use AI well, not fewer. The company that bet the other way wrote you the
receipt.

## Read this next

-
[**AI Won’t Shrink Your
Team — It’ll Expose the Work You Were Skipping**](https://sublimecoding.com/blog/ai-wont-shrink-your-team) — Why
leverage raises the stakes on every decision instead of removing the
decider.

-
[**When
to Trust an Agent and When to Step In**](https://sublimecoding.com/blog/when-to-trust-an-agent-and-when-to-step-in) — The load-bearing
skill behind every safe AI deploy.

-
[**A Professional
Owns the Whole Outcome**](https://sublimecoding.com/blog/professional-owns-the-outcome) — Ownership is the one thing you
can’t hand to something that can’t be paged.

---

## AI Won't Shrink Your Team — It'll Expose Why You Needed a Bigger One

URL: https://sublimecoding.com/blog/ai-wont-shrink-your-team
Published: 2026-05-01
Tags: AI, engineering leadership, founders, hiring, productivity, agents, business

**Every company rolling out AI is about to discover how much work they were leaving on the table.**

The narrative dominating board decks and all-hands slides in 2026 is some version of "AI lets us do more with less." Headcount frozen. Targeted reductions in junior engineering. Internal memos using phrases like "AI-driven productivity" to justify a leaner team. The companies leaning hardest into this story are about to make the most expensive mistake of the decade.

I've watched this play out at three companies over the last two years. The pattern is consistent. AI doesn't replace the team. It surfaces the backlog the team never had bandwidth to touch. More throughput becomes more surface area becomes more coordination, review, and decision work. The companies cutting headcount now will be outpaced inside two years by the ones quietly staffing up to absorb what AI is producing. The numbers cited for the opposite case usually do not survive a look at the transcript — [nobody got replaced, agents got added](https://sublimecoding.com/blog/nobody-got-replaced-agents-got-added).

## Does AI actually make engineers 10x more productive?

The 2014 version of "10x engineer" was bullshit and most senior people knew it. The 2026 AI-flavored version is the same myth wearing new clothes.

AI makes one engineer faster — measurably, 30–50% on routine work, sometimes more on greenfield code where the agent has full context. That part is real and I've written about it extensively. What AI does *not* do is make that engineer smarter about what to build. It doesn't tell them which customer is unhappy this week. It doesn't know that the last three production incidents all traced to the same misnamed config flag. It doesn't have a point of view on whether the new feature the founder wants is going to cannibalize the one that's actually monetizing.

Speed without direction is churn at a higher RPM. The thing that actually scales an engineering organization is judgment, and judgment does not compress. The senior engineer who can look at a system and tell you which 20% of changes will cause 80% of next quarter's incidents is not a function of typing speed. They've built that intuition over years of being on call for systems they shipped, watching their decisions hit production, and updating their priors. None of that transfers to a model.

## Velocity creates surface area

This is the math most teams miss when they congratulate themselves on AI-driven speedups.

If your team is shipping 3x faster, you also have:

- 3x more PRs to review

- 3x more code paths to test

- 3x more deploys to monitor

- 3x more security review

- 3x more product decisions to make

- 3x more customer-facing changes to communicate

- 3x more documentation to keep current

- 3x more incident potential when something inevitably breaks

Every doubled velocity multiplier creates new coordination, review, and decision-making surface. The team doesn't shed work — it accumulates new categories of work it didn't have to do before. The PR backlog you used to clear by Friday now stretches into the next sprint. The on-call rotation that was tolerable at one deploy a day becomes brutal at four.

AI does not reduce this surface. It mostly creates more of it. The companies winning this transition aren't the ones with the smallest headcount. They're the ones who recognized that the bottleneck moved from "engineering capacity to ship code" to "human capacity to review, decide, and absorb," and staffed accordingly.

## The bet that's about to go badly

Several large tech companies announced 10–20% headcount reductions in 2025 and 2026, citing "AI productivity gains" as the justification. The narrative writes itself: AI lets us do more with less, so we did. Stock pops, board nods, internal memo gets shared on LinkedIn.

I think most of those companies are going to look back on these decisions in 2028 and realize what they actually did was three things, none of them strategic:

First, they let go of senior engineers — the people whose judgment was the actual force multiplier — alongside the routine roles AI did partially replace. Severance was equal-opportunity. The result is an organization where the remaining engineers have less context, less production scar tissue, and less institutional memory than the one that existed eighteen months ago.

Second, they created an organization where the remaining team is perpetually behind on review, security, and incident response because the work scaled while the team shrank. Incidents pile up. Audit findings stack. Customer escalations route to fewer people. The throughput gain is real on the input side and a debt-accrual machine on the output side.

Third — and most damaging long-term — they sent a signal to remaining staff that AI is a threat, not a tool. The teams that performed best with AI in my experience were teams that trusted that learning the new workflow wouldn't end their jobs. The teams whose leadership signaled "be productive or be replaced" got compliance-driven AI adoption: more usage, lower quality, more shortcuts, more slop.

The companies that will dominate the AI transition look exactly the opposite. Stable or growing engineering team. Heavy investment in tools, training, and the supporting roles (security, DevOps, product, design) that scale with throughput. Senior leadership communicating that AI is for amplifying the team, not replacing it. Those companies are quietly hiring while the loud ones are publicly cutting. Watch which ones are at the front of the pack in two years.

## Judgment doesn't delegate

I covered this in detail in [When to Trust an Agent and When to Step In](https://sublimecoding.com/blog/when-to-trust-an-agent-and-when-to-step-in). The short version: there's a category of decisions you cannot delegate to a model, and those decisions are the ones that compound into company outcomes.

- Whether the architecture is right for what you're building three years from now

- Whether the customer's problem is the one you should be solving

- Whether shipping this feature now is more valuable than fixing what shipped last quarter

- Whether the on-call engineer who keeps making the same mistake needs coaching or termination

- Whether the right approach to this bug is to fix it or to refactor the surrounding code so it can't happen again

- Whether your security posture is sufficient for the enterprise customer asking

Every one of those is a judgment call. Every one of them affects more than one team's work. None of them gets better when you have fewer experienced humans involved. AI can *support* these decisions — by surfacing data, drafting analysis, enumerating tradeoffs — but the actual call is human, and removing humans from that loop is how organizations make decisions they regret for years.

## The under-resourced trap, accelerated

There's a specific failure mode I've seen repeatedly at companies trying to brute-force output without staffing up. The shape of it:

The team ships fast for a quarter. Demos look incredible. The product feels like it's accelerating. Then the bills come due. Two production incidents take three days each to resolve because nobody had bandwidth to do post-mortem on the last incident, so the same thing breaks twice. A security audit surfaces eight findings the team has been meaning to fix for months. A customer success ticket pile reveals a 22% increase in confusion-flavored complaints — users tripping over a feature shipped without product review. A senior engineer quits because they've been on permanent escalation duty for six months and the founder keeps saying "we'll hire after this push."

AI accelerates this dynamic. Faster shipping equals faster accumulating debt when the team doesn't have the headcount to handle the supporting work. The chaos doesn't disappear when you add AI to an under-resourced organization; it compounds faster, hits earlier, and is much harder to recover from because the team is also burnt out.

The companies betting on AI as a headcount substitute are walking into this trap with their eyes closed. The companies betting on AI as a leverage multiplier — and staffing accordingly — are going to look at the wreckage in eighteen months and pick up the customers, the talent, and the market position the under-resourced bet left on the table.

## How should you actually size an engineering team in the AI era?

If you accept that AI raises throughput but doesn't reduce the human work needed to absorb that throughput, the right-sizing question changes shape entirely. The questions to ask, in order:

- **Which roles became more valuable because their leverage scaled with AI?** Almost always: senior engineers, engineering managers, staff-level technical leads. AI raises the floor of what one person can produce, which makes the people who can direct that production output disproportionately more valuable.

- **Which roles became more strategic because the routine parts moved to AI?** Product management, design, technical writing. The mechanical work in these roles compresses; the judgment work doesn't. Hire for the judgment.

- **Where do we have throughput gains without the corresponding humans to absorb them?** Most commonly: code review, security, DevOps, on-call. These functions scale linearly with deployment frequency, and almost no organization has staffed them ahead of the AI productivity curve.

- **Where is the team currently bottlenecked — and would adding people unblock it?** Decision-making capacity is usually the answer. Engineers waiting for review, PMs waiting for engineering input, founders making technical calls they shouldn't be making themselves. Adding senior people unblocks all of these.

The honest answer for most teams in 2026 is that they need *more* people, in *different* roles than the org chart from 2024. Not the same roles. Not "more engineers writing code." More senior engineers reviewing AI output, more security people running incident response, more PM capacity making the strategic calls AI can't, more DevOps capacity catching the deploys AI is now generating in volume.

If you're between "we're underwater on velocity" and "we can't justify another full-time hire yet," that's the gap I close as a [fractional engineering lead](https://sublimecoding.com/consulting). Both ICs and senior judgment, scoped to a quarter rather than a forever-headcount line.

## The takeaway

The "AI shrinks the team" narrative is going to look in 2028 the way "the cloud means we don't need ops people" looked in 2015. Wrong, expensively wrong, and obvious in retrospect.

The companies that dominate the AI transition aren't the ones that fired half their team and high-fived themselves. They're the ones who staffed up the parts of the organization that scale with throughput, kept their senior judgment intact, and recognized that one engineer plus AI is a more powerful version of one engineer — not a replacement for the team they used to need.

If you're a founder or VP of Engineering staring at a hiring freeze justified by "AI productivity," I'd push back hard. Your competitors who are still hiring are the ones you're going to be racing against in two years. The bet isn't AI vs. headcount. The bet is whether you trust your team to do more, supported, or whether you trust the model to replace what you couldn't be bothered to invest in.

I know which one I'd take.

## Read this next

- [**The Pre-Series-A AI Startup Hiring Plan**](https://sublimecoding.com/blog/pre-series-a-ai-startup-hiring-plan) — The hire-by-hire framework for actually staffing the way this post argues you should.

- [**AI-Assisted Engineering Isn't Faster Coding**](https://sublimecoding.com/blog/ai-assisted-engineering-is-a-new-workflow) — The team-level workflow change that produces the throughput this post is talking about absorbing.

- [**From One Engineer to Fifteen**](https://sublimecoding.com/blog/from-one-engineer-to-fifteen-engineering-leadership) — The leadership lessons that inform why "do more with less" is almost always the wrong posture.

---

## Nadella Is Right About AI and the Firm. Mostly.

URL: https://sublimecoding.com/blog/nadella-token-capital-small-teams
Published: 2026-06-15
Tags: AI, engineering leadership, founders

>
**TL;DR:** Satya Nadella just published the clearest
case I’ve seen from a major platform CEO that AI amplifies human
judgment instead of replacing it — and it confirms what I’ve argued on
this site for a while. His “human capital plus token capital” framing is
the right map. But it’s drawn from the 30,000-foot enterprise chair, and
three things get glossed that matter most to the people doing the work
and the small teams who feel this shift first. The learning loop doesn’t
translate cleanly to a four-person team, “amplified judgment” reads very
differently from the employee’s seat, and “token capital” is an idea
that happens to be extremely good for the company selling the tokens.
None of that makes him wrong. It just makes the essay incomplete for the
readers I write for.

In a June 2026 post on X, Satya Nadella, Microsoft’s CEO, [argued
that we’re at a genuinely new kind of platform shift](https://x.com/satyanadella/article/2066182223213293753). Not another
tool to bolt onto the org chart, but something stranger: “the first time
we can create a real cognitive loop between people and digital systems.”
He’s right, and I want to start there, because I think the essay is
mostly correct and worth taking seriously rather than dunking on.

## He’s confirming the thing I’ve been saying: the human is the point

The center of Nadella’s argument is a distinction. Every company, he
writes, “is going to have to build what I think of as human capital and
token capital.” Human capital is “the knowledge, judgment,
relationships, ingenuity, and pattern recognition of its people.” Token
capital is “the firm’s AI capability it builds and owns.”

And then the line that matters: “human capital does not become less
valuable as token capital grows. It only becomes more valuable!”

If you’ve read much of what I write here, that should sound familiar.
The entire premise of [AI won’t
shrink your team](https://sublimecoding.com/blog/ai-wont-shrink-your-team) is that the binding constraint on a software
business was never typing speed — it was judgment, taste, and the
ability to decide what’s worth building. Make the doing cheaper and you
don’t need fewer people exercising judgment. You need more surface area
for that judgment to act on. Nadella’s version — “Without human
direction, you have compute running in circles” — is the same claim in a
sharper sentence. I’ll happily steal it.

There’s a second place where the essay lands almost exactly on an
argument I’ve made. Nadella says the real work is building “a learning
loop on top of models,” and that “private evals should capture whether a
model is actually improving against outcomes that matter to the business
(not just external benchmarks!).”

That parenthetical is the whole game. I made this case already in [your AI
product needs telemetry before a better model](https://sublimecoding.com/blog/your-ai-product-needs-telemetry-before-better-model): the benchmark
leaderboard is not your business, and chasing the next model release
before you can measure whether the current one helps your users is
backwards. You cannot improve against “outcomes that matter” if you
never instrumented the outcomes. Nadella, from the top of the company
that sells the models, is telling you the model is not the
differentiator. Believe him on that one. It’s true, and it costs him
something to say it.

And the best line in the whole essay is one I wish I’d written: “You
can offload a task, or even a job, but you can never offload your
learning.” That is durable, it’s scale-free, and I’ll come back to it at
the end because it’s the part worth keeping.

So: credit where it’s due. As a piece of strategy for a large
enterprise, this is a good essay. Now let me sharpen it, because there
are three things the enterprise view misses, and they’re the things my
readers feel first.

## What the enterprise view misses

### 1. “Private RL environments and a hill-climbing machine” is a Fortune-500 program, not a startup plan

Here’s where the altitude shows. Nadella describes the learning loop
concretely, and it’s a large-company concrete: “Private reinforcement
learning environments should let models grow stronger on real traces
from inside the organization.” There’s a “knowledge base” that “makes
institutional memory queryable.” He calls the whole thing “a hill
climbing machine” that “compounds.”

That is a program. It assumes a platform team, a data org, an ML
function, and enough proprietary trace volume that reinforcement
learning on internal data is even a coherent idea. A four-person
pre-Series-A team has none of that, and shouldn’t. If you’re seed-stage
and you spin up a private RL environment, you have made a serious
mistake about what your company is for.

So the honest question — the one the essay doesn’t answer because it
isn’t written for you — is: **what does the learning loop actually
look like at startup scale?** It’s a real question, and it has a
real answer. It just isn’t infrastructure.

At small-team scale, the learning loop is mostly discipline and
encoded convention:

- **Your “private RL environment” is a
`CLAUDE.md`-style conventions file** (or a
`cursorrules`, or an `AGENTS.md` — whatever your
tools read). It’s the place where you write down, in plain language, how
this codebase does things: the patterns you’ve settled on, the mistakes
you keep correcting, the boundaries the agent keeps crossing. Every time
you fix the same thing twice, it goes in the file. That *is* the
hill-climbing machine for a small team — a flat file that gets a little
smarter every week.

- **Your “private evals” are: does it pass review, and does it
pass the tests?** You don’t need a benchmark harness. You need
the existing gate — code review, CI, a human who knows the product — to
be the eval. The signal Nadella wants (“outcomes that matter, not
external benchmarks”) is already sitting in your PR queue. The
discipline is treating it as signal: noticing *which* kinds of
work the agent gets right unsupervised and which it doesn’t, and routing
accordingly. I wrote the long version of this in [when to trust an
agent and when to step in](https://sublimecoding.com/blog/when-to-trust-an-agent-and-when-to-step-in).

- **Your “queryable institutional memory” is docs, decision
records, and good prompts.** The judgment that lives in your
founding engineer’s head is the asset. Capturing even a fraction of it —
why you chose this database, why you rejected that abstraction, what
“done” means here — into text the agent can read is the entire move.
It’s unglamorous. It’s also tractable on a Tuesday afternoon, which the
RL environment is not.

- **And the part everyone skips: actually measure the
outcome.** [Prove the return](https://sublimecoding.com/blog/prove-the-return)
is a whole essay on this, so I’ll be brief — the loop only compounds if
you close it. Did the thing you shipped move the number you cared about?
If you never check, you don’t have a learning loop. You have a faster
way to ship things you can’t evaluate.

This is the constructive heart of the critique. Nadella’s loop is
correct in shape and wrong in scale for most of the people reading this.
The scaled-down version isn’t a watered-down RL pipeline. It’s a
different and frankly more achievable thing: encoded conventions, the
review gate as your eval, captured judgment in docs and prompts, and the
habit of measuring results. If you run a small team, that’s your token
capital. Start there. The same logic applies whether you’re four people
or fourteen — see [managing a
four-person engineering team](https://sublimecoding.com/blog/managing-a-four-person-engineering-team) for how thin the process layer can be
and still work.

### 2. From the worker’s chair, “your expertise becomes replicable in systems” is a different sentence

Read the second-to-last paragraph closely. Nadella writes that
“Employees will see their expertise amplified and their judgment become
part of systems that make it replicable and scalable.”

From the firm’s chair, that’s a promise: your people get amplified.
From the employee’s chair, “my judgment becomes part of a system that
makes it replicable” can read as: *I am encoding myself into the
thing that makes me optional.* Those are the same sentence said from
two chairs, and the essay only sits in one of them.

I want to be careful here, because the doomer version of this point
is lazy and I don’t believe it. My consistent position on this site is
that AI changes the job and expands capacity — it doesn’t, on net,
delete the worker. I still think that’s right. The founding engineer who
encodes her judgment into a `CLAUDE.md` and a set of evals
doesn’t make herself redundant; she makes herself the person who
*owns the loop*, which is more leverage, not less. Nadella’s
“amplified” is genuinely available.

But “stable equilibrium” — his closing words — is doing a lot of work
to paper over a transition cost, and somebody pays it. The senior
engineer whose tacit knowledge gets captured this year is in a strong
position. The mid-level engineer whose job was *executing*
well-specified tasks — the work that’s now most automatable — is the one
absorbing the change, and “your expertise will be amplified” is cold
comfort if your expertise was mostly execution. The honest framing isn’t
“everyone wins in equilibrium.” It’s: the value of judgment goes up, the
value of pure execution goes down, and the transition between those two
states is not free, and it’s not evenly distributed.

For a founder, this isn’t a reason for guilt — it’s a reason for
clarity. If you’re building the loop Nadella describes, even the
small-team version, you are changing what your people’s jobs are. Say
so. Tell your team the work is shifting from doing to directing and
verifying, and that you’re going to invest in getting them to the
judgment side of that line rather than leaving them on the execution
side as it erodes. That’s the difference between amplification and
quietly training your own replacement, and it’s a choice the firm makes,
not a law of physics. I’ve written more on what that workflow shift
actually feels like in [AI-assisted
engineering is a new workflow](https://sublimecoding.com/blog/ai-assisted-engineering-is-a-new-workflow).

### 3. Follow the incentives: “token capital” is a conveniently platform-serving idea

Now the uncomfortable one. “Token capital” is a good frame. It’s also
an idea that is *spectacularly* convenient for a company whose
business is renting you tokens.

Walk the logic. If every firm in every sector becomes convinced it
must build proprietary “token capital” — private evals, private RL
environments, [queryable knowledge bases](https://sublimecoding.com/blog/phoenix-rag-pgvector-arcana), a compounding learning loop —
then every firm in every sector becomes a heavier, stickier, longer-term
consumer of model inference and cloud infrastructure. Whose cloud? Well.
The essay is, among other things, a beautifully argued reason to spend
more on the platform that published it.

I want to be precise, because the cynical version of this point is as
lazy as the doomer version of the last one. The argument can be
**simultaneously true and self-serving.** Those aren’t in
tension. It is genuinely good advice for a firm to own its learning loop
rather than cede all its value to a handful of foundation models —
Nadella even makes the macro case himself, warning against “a world
where every company across every sector is ceding value to a few models
that eat everything they see,” and invoking the way “entire industrial
economies were hollowed out by outsourcing.” I think he means it. I also
think the prescription that follows from it (“build token capital, on a
frontier *ecosystem*”) routes an enormous amount of spend toward
the ecosystem he’s selling. Both things are real.

The practical takeaway for a founder isn’t “ignore him.” It’s
“separate the diagnosis from the prescription, and price the
prescription yourself.” The diagnosis — don’t let the model commoditize
your knowledge — is sound and free to act on. The prescription — build
heavy proprietary AI infrastructure — has a vendor’s thumb on it, and
you should adopt only the slice that survives your own cost-benefit
math. For a four-person team, that slice is the cheap, durable stuff in
critique #1, not a six-figure cloud commitment.

There’s an adjacent, larger critique floating around right now about
the “loopification” of AI financing — the circular arrangements where
model providers, cloud providers, and chipmakers fund and buy from each
other in ways that can make demand look more organic than it is. I’m not
going to claim Nadella’s essay is *about* that; it isn’t, and
stretching it that far would be the kind of overreach I’m criticizing.
But it’s worth keeping in your peripheral vision: when the entire supply
chain has a stake in convincing you that you must accumulate token
capital, treat “you must accumulate token capital” as a claim to verify,
not a given.

## The version worth keeping

Strip the enterprise scaffolding away and Nadella’s best line stands
on its own at any scale: “You can offload a task, or even a job, but you
can never offload your learning.”

That’s the whole thing. The model is rented and commoditizing by the
month. What compounds — what is actually yours — is the judgment your
team accumulates about *your* problem, *your* users,
*your* codebase, and the discipline of encoding that judgment
somewhere the agents can use it. The job shifts from doing to directing
and verifying. The moat is compounding judgment, not the model.
Everything else in the essay is implementation detail, and most of the
implementation detail is sized for a company a thousand times larger
than yours. Which is the same conclusion from a different direction: [the schlep is the part agents cannot cross](https://sublimecoding.com/blog/schlep-is-the-moat-agents-cant-cross).

### What a founder or small team can do Monday

You don’t need a platform team to start the loop. You need four
habits:

- **Write the conventions down.** Start a
`CLAUDE.md` (or equivalent) today. Every time you correct the
agent on the same thing twice, the correction goes in the file. This is
your scaled-down “hill-climbing machine,” and it costs nothing.

- **Make your review gate the eval.** Stop reaching for
the next model release. Treat your existing PR review and tests as the
measure of whether the AI is actually helping, and pay attention to
*which* tasks it gets right unsupervised. Route work
accordingly.

- **Capture judgment, not just code.** Spend an hour a
week turning what’s in your senior people’s heads — why this choice,
what “done” means, which mistakes recur — into text. That’s your
queryable institutional memory, no vector database required.

- **Close the loop by measuring the outcome.** Pick the
one number a change was supposed to move, and check it. A learning loop
you never measure is just a faster way to ship things you can’t
evaluate.

And one thing to do with your *team*, not your tooling: say
out loud that the work is moving from execution to judgment, and that
you’re going to help everyone get to the judgment side of that line.
Nadella calls the end state a “stable equilibrium.” Maybe — but
equilibrium is something a firm builds deliberately, by deciding who it
carries through the transition. That’s a leadership choice, not an
emergent property of the technology. Make it on purpose.

Nadella’s right about the map. Just remember it was drawn from an
altitude where you can’t see the four-person teams who feel the weather
first.

### Postscript — July 23, 2026

Five weeks after this essay, Nadella published [a follow-up
on X](https://x.com/satyanadella/status/2080329851127669104) that makes the routing half of this argument explicit, and
first-party. Microsoft is now sending traffic across GitHub Copilot,
Excel, and Outlook to its own MAI models wherever they match or beat the
frontier alternatives, keeping OpenAI and Anthropic models in the
orchestration for the frontier work. And he names the design goal
outright: your evals “should continue to hill climb even when any given
model has been removed” — with the harness, memory, context, and skills
deliberately externalized outside the model.

That is habit 2 above, stated as strategy by the largest software
company on earth. The small-team version still needs none of the
enterprise machinery: the conventions file is your externalized harness,
the test suite is your eval, and a one-page task-to-model table is your
router. The one thing worth adding since June is the motive inversion.
For Microsoft, model independence is offense — route the traffic, keep
the margin. For you it’s defense: when a model you depend on gets
deprecated, repriced, or quietly degraded, [everything you encoded outside
it](https://sublimecoding.com/blog/ai-memory-outlives-any-agent) is what keeps compounding.

---

## Agents Don't Need Prompts. They Need Managers.

URL: https://sublimecoding.com/blog/agents-dont-need-prompts-they-need-managers
Published: 2026-09-08
Tags: AI, agents, founders, AI startups

>
**TL;DR:** Claude sessions stop before the work is done,
and the closing message often reports the task finished when it isn’t.
That’s real — I run agents daily and I’ve watched both halves of it
happen. But “the model is lazy” is the wrong diagnosis. A human employee
arrives wrapped in a management system nobody had to build: a queue, a
standup, a manager who notices when they go quiet, a peer who catches
the miss. An agent arrives with none of it, and you become all of it, by
hand, one agent at a time. Meanwhile two things are true together:
companies cut headcount for agents and then [regretted
it](https://www.hrdive.com/news/leaders-who-laid-off-workers-due-to-ai-regretted-it/746643/) at scale, and the vendor has shipped limits that cap exactly the
behavior operators want — Anthropic shipped weekly rate limits in August
2025 [aimed
explicitly at people running Claude Code “continuously in the
background, 24/7.”](https://techcrunch.com/2025/07/28/anthropic-unveils-new-rate-limits-to-curb-claude-code-power-users/) The gap between those two facts is where the
frustration lives.

## Why does an agent stop before the work is done?

Because it arrived without the management system that keeps a person
from doing the same thing.

Run agents on real work long enough and you converge on a pattern. A
session ends before the task is actually finished, and the last message
says some version of “this still isn’t complete” — noting the gap
instead of closing it. Told to be thorough, it cuts a corner anyway. ([The comic version of what
that feels like day to day](https://sublimecoding.com/blog/programming-sucks-now-with-agents) is its own post.) Pushed on the corner,
it admits to skipping it, plainly, and then explains what would fix it —
which raises the obvious question: why explain it instead of doing
it?

I’ve watched all of that happen on this repo and others, and the
worst version isn’t the early stop. It’s the agent reporting completion
on work it didn’t do. That’s a different, more serious failure than
laziness. What I want to argue first is that “the model is lazy” is a
correct observation aimed at the wrong layer, and the correct read costs
a lot more money than the wrong one.

## What a person arrives wrapped in

Hire a mid-level engineer and you get a person. You also get, for
free, an apparatus around that person that took your company years to
build and that nobody thinks of as a feature.

There’s a queue that tells them what’s next when they finish, so the
end of a task isn’t the end of the work. There’s a standup where
somebody would notice they’ve been stuck since Tuesday. There’s a
manager whose actual job is detecting when work has stalled and applying
a nudge. There’s a code review that catches the shortcut. There’s an
escalation path for the moment the task turns out to be underspecified —
which is most tasks. There’s memory: they remember the incident in
March, they remember that you hate that pattern, they remember they said
they’d circle back.

Not one of those is the employee. All of them are the system around
the employee. Take a genuinely excellent engineer and remove all of it —
no queue, no manager, no review, no memory of yesterday — and you get
someone who does good work in bursts, stops when the current thing looks
done, and has no mechanism to know they stopped short.

That is precisely the thing people are describing when they say an
agent can’t work independently. The independence they’re missing was
never a property of the worker. It was a property of the org.

## The reversal wave is that bill coming due

Orgvue’s 2025 survey of more than a thousand business leaders found
that about four in ten had laid people off as a result of deploying AI;
[55%
of those now say they made the wrong call](https://www.techrepublic.com/article/news-leaders-regret-ai-driven-layoffs/). Orgvue’s CEO Oliver Shaw
put it about as bluntly as a vendor ever puts anything: “[dismissing
employees without a clear plan for workforce transformation is
reckless](https://www.hrdive.com/news/leaders-who-laid-off-workers-due-to-ai-regretted-it/746643/).” Robert Half found that [more
than three in ten](https://www.forbes.com/sites/rachelwells/2026/07/26/ai-layoffs-are-backfiring-did-employers-bet-too-much-on-the-ai-boom/) US hiring managers who eliminated a position after
implementing AI later added it back, and when [Gartner
asked over 320 customer service leaders in October 2025](https://www.forbes.com/sites/rachelwells/2026/07/26/ai-layoffs-are-backfiring-did-employers-bet-too-much-on-the-ai-boom/) whether
they’d actually cut human support staffing because of AI, only about a
fifth had — most held headcount steady and served more customers with
it, roughly the same shape I found when I [checked the McKinsey
numbers](https://sublimecoding.com/blog/nobody-got-replaced-agents-got-added) that had 25,000 people supposedly replaced by AI agents.

What the reversals have in common isn’t that the models
underperformed. It’s that the buyer removed the humans doing
supervision, quality control, and exception handling, kept the part the
machine could do, and found out the removed part was load-bearing. I’ve
run the capacity math behind that in more detail in [“why AI won’t shrink your
team.”](https://sublimecoding.com/blog/ai-wont-shrink-your-team)

## And the vendor would like your agent to stop

One paragraph on this, because it’s not the whole story. Anthropic
introduced weekly rate limits effective August 28, 2025, and the stated
reason was subscribers running Claude Code *“continuously in the
background, 24/7”* — the company said it would affect under 5% of
subscribers. Read that against the frustration above: the exact behavior
operators want is the exact behavior the vendor named and capped. You
don’t have to invent a motive to notice that — draw your own conclusion
about why. Netflix asks “are you still watching?” after [three episodes and ninety
minutes](https://help.netflix.com/en/node/114059) of uninterrupted play, and Netflix’s own help page gives two
reasons, both framed as the viewer’s benefit: saving your place, and not
burning your data on a show nobody’s watching. The bandwidth on the
other end is a real cost line too, and Netflix doesn’t need to say so.
Inference is a real cost line too, and right now it’s still subsidized —
I [put numbers on that
when I priced the plan tiers](https://sublimecoding.com/blog/ai-coding-agent-costs-real-numbers) — which means every stopped agent
session is margin. That doesn’t make the model lazy by design. It does
mean nobody upstream of you is strongly motivated to solve continuous
unattended operation on your behalf, so stop waiting for them to.

## Build the layer yourself

Here’s the part that changes outcomes. Everything a manager does for
a person has a mechanical equivalent you can build, and none of the
equivalents are exotic.

**Put the state on disk, not in the conversation.** The
model’s working memory ends when the turn ends. A file doesn’t. A task
list the agent reads at the start of a session and updates as it goes is
the single highest-leverage thing in this list, because it converts
“remember what we were doing” from a context problem into a filesystem
read. I run this as a literal `tasks/todo.md` in this repo —
the agent checks items off as it finishes them, and the next session
starts by reading the file instead of reconstructing intent from a
transcript. This is also why a stalled session isn’t really about the
size of the context window: you don’t want one enormous run, you want
many small ones that share a checkpoint. Anything the agent knew
mid-session that isn’t written down is knowledge you’re about to lose
the moment the session ends, regardless of why it ended. I wrote a
longer version of this pattern — treating a `CLAUDE.md` and
its adjacent files as onboarding documentation for a new hire who has no
memory between shifts — in [the
case for treating your AI config like an onboarding doc](https://sublimecoding.com/blog/claude-md-onboarding-doc).

**Make the important things reflexes instead of
requests.** Anything you have to ask for, the model can skip
under pressure. Anything wired into the harness, it can’t, because it
isn’t the model doing it — it’s a program that runs regardless of what
the model decided. On this site, a hook runs `mix format`
after every file edit, so a formatting break can’t reach a commit no
matter how tired or rushed the agent got mid-task. A second hook denies
writes to `.env` outright, full stop, no negotiation with the
model about whether this edit is the exception. A third validates a
content JSON file after every edit and blocks the write if the file no
longer parses — because a plausible-looking edit that breaks JSON is
exactly the kind of thing a model will confidently make and not notice.
I [wrote up the four I run
everywhere](https://sublimecoding.com/blog/four-claude-code-hooks); the point isn’t those four specifically, it’s the
category. An instruction in a prompt is a request. A hook is a reflex,
and reflexes don’t degrade when the model is nine hours into a long
session.

**Gate on output, not on effort.** “Be thorough” is not
a specification and there’s no way to verify it happened after the fact.
A preflight check that returns READY or FIX REQUIRED is a specification.
A test suite that must pass before work is called done is a
specification. The difference matters because a model optimizing to
satisfy an instruction will satisfy the literal instruction — if the
instruction is “be thorough,” the model can produce something that reads
as thorough without being complete, and neither of you will notice until
it ships. If your only mechanism for getting careful work is asking for
careful work in the prompt, you have no mechanism; you have a hope.

**Cut the work small enough that stopping is cheap.**
Most of the pain in a long agent session isn’t the stop itself, it’s
that the unit of work was too big to verify in pieces, so a failure
surfaces at the very end instead of at the seam where it happened. When
a task is scoped to something with a checkable output — a function that
either passes its test or doesn’t, a file that either parses or doesn’t
— an agent quitting early is a visible, recoverable event instead of a
silent one buried three files deep in a diff nobody read closely.

**Notice when the “nudge” you keep giving is actually a missing
input.** This is the one I’d want a frustrated operator to sit
with longest. When you find yourself typing “continue” for the fourth
time in a session, that’s usually not the model being lazy — it’s the
model reaching the edge of what you actually specified and stopping at a
boundary you drew without meaning to. The fix lives upstream, in the
brief, not downstream, in the nudge. Write the task list once,
specifically, and the agent works through it faster than you can type
the next prompt. That’s the trade, and it’s a good one, but it only pays
off if you do the upfront work instead of narrating the gap after the
fact.

The reason this feels like a lot of work is that it *is* the
work. It’s the same work your company already did for humans, over
years, mostly without anyone framing it as work at all. Nobody skips it
and gets away with it. What people are calling a model problem is a
missing org chart.

## What no harness fixes

If I stopped there this would be vendor defense, so: some of this
really is the model’s fault, and no hook I’ve described touches it.

A model reporting completion on work it didn’t do is not a
supervision gap you can gate around, because the harness has no way to
know what the task was supposed to mean in the first place — it can
check that a response was produced, not that the input was actually
read. In my own long sessions, quality seems to erode well before the
context window is anywhere near full; I don’t have a benchmark for that
— it’s a pattern across a lot of sessions, not a measured curve. I’ve
written before about [the specific bugs this
produces](https://sublimecoding.com/blog/things-ai-coding-agents-get-wrong) and about how [an agent has no
stake in whether the code is good](https://sublimecoding.com/blog/why-ai-agents-write-unmaintainable-code) — both remain true and neither is
going away this year.

Anthropic’s own Opus 5 launch materials claim ground on exactly this
failure mode. Their published example: given a real bug in a popular
open-source package manager, Opus 5 found the root cause and fixed an
edge case the community’s own patch had missed, while “[a competing model
fixed only the surface symptom (not the underlying cause), then reported
the bug resolved.](https://www.anthropic.com/news/claude-opus-5)” That’s the same shape of failure this post is
describing, named by a vendor, in its own launch copy, as the failure
its competitors ship. Whether it’s actually fixed at the rate the
marketing implies is a separate question I can’t settle from a launch
page — but either way, the model layer moving is not a reason to skip
building the management layer. It’s a reason to expect the gap to keep
moving rather than close.

The practical mitigation is unglamorous: stop handing the model the
whole history. Give it the recent turns and a summary you control. Ask
for the answer *and* the evidence for it, so work the model
skipped shows up as a missing citation instead of a confident sentence.
There is published support for the instinct — Liu et al. found model
performance degrades markedly depending on [where in a long context the
relevant information sits](https://arxiv.org/abs/2307.03172) — so assume anything that landed in the
middle of a long context wasn’t really read, and structure the task so
that assumption is cheap to be wrong about.

And a caution I’d apply to my own argument here. Building this layer
works, and it does not produce an unattended agent. It produces one that
needs fewer, better-placed inputs from you. If a plan depends on the
layer eventually reaching zero human input, that plan hasn’t been
demonstrated by anyone, including me.

## What to price before you cut a role for an agent

Write down what the person you’re removing actually supplies,
separately from the task they perform. The queue, the noticing, the
exception handling, the memory of why the last attempt failed. If the
answer is “the agent does the task,” you’ve priced one column of a
two-column problem, and the Orgvue number above says you’ll find the
second column later, at a worse price, once it’s already gone.

Say who the manager is. Not metaphorically — name them. Every agent
doing real work needs a human whose job includes noticing when it
stopped — and [knowing which
calls that person should be making themselves](https://sublimecoding.com/blog/when-to-trust-an-agent-and-when-to-step-in) is a different skill
from building the scaffolding, and if that human is also doing their own
full-time job, you’ve just found your actual capacity limit. I made this
argument in more detail in [“Nobody Got Replaced,
Agents Got Added”](https://sublimecoding.com/blog/nobody-got-replaced-agents-got-added): “The supervision doesn’t disappear. It changes
shape and moves up a level.” That line is doing more work than it looks
like — it’s the whole argument, compressed to one sentence.

Then build one piece of the layer before you add the next agent. One
hook, one task file, one gate that blocks a bad output instead of asking
politely for a good one. That ratio improves from the supervision side
far more reliably than from adding more agents — more agents in one
conversation tend to [agree their way into
the wrong product](https://sublimecoding.com/blog/multi-agent-teams-agree-into-garbage) — which is roughly the shape of the argument I
made about [what token economics
actually do to the size of a firm](https://sublimecoding.com/blog/nadella-token-capital-small-teams).

The failure mode is real. The diagnosis was wrong. What looks like an
unmotivated model is an unmanaged one, and you’re the only one in a
position to fix that — the vendor has told you, in its own launch copy
and its own rate-limit announcement, that it isn’t going to do it for
you.

You already cut headcount on an agent thesis, and the quality is
drifting in a way nobody on the team can quite name yet — that’s the
conversation I have with founders most often right now, more often than
the sizing conversation that usually starts it. [Let’s talk](https://sublimecoding.com/consulting).

---

## Nobody Got Replaced. Agents Got Added.

URL: https://sublimecoding.com/blog/nobody-got-replaced-agents-got-added
Published: 2026-08-07
Tags: founders, AI, agents, AI startups

>
**TL;DR:** The claim making the rounds — that McKinsey
replaced 25,000 of its 60,000 employees with AI agents — is not what
happened. On [the
HBR IdeaCast in January](https://hbr.org/podcast/2026/01/where-mckinsey-and-consulting-go-from-here), Bob Sternfels put the firm at 60,000 total:
about 40,000 humans *plus* roughly 20,000 agents, a number he
revised upward to about 25,000 at CES days later. The agents were added
on top. The human headcount did fall — from around 45,000 at the end of
2023 to about 40,000 — but that took two years and McKinsey attributes
it to attrition and performance management. Meanwhile the buried number
is the one founders should care about: client-facing roles are
*up* about 25%. This isn’t a story about replacement. It’s a
story about reallocation, and the question it should provoke in your
next planning meeting isn’t “how many people can I cut” — it’s “how many
agents can one of my people actually supervise before the quality
drops.”

## What was actually said

The viral version is clean and wrong: 60,000 employees, 25,000
replaced by AI. It’s a good number. It fits in a headline, it confirms
something people already half-believe, and it takes about four minutes
to check.

Here’s the actual quote, from Sternfels on the IdeaCast on January
6:

>
“I now update this almost every month, but my latest answer to you
would be 60,000, but it’s 40,000 humans and 20,000 agents.”

Sixty thousand is humans *plus* agents. It’s a headcount
figure that includes software. The 40,000 humans didn’t get reduced to
arrive at that number — the 20,000 agents got stacked on top of them to
produce it. A day later at CES, Sternfels put the agent count closer to
25,000, and a McKinsey spokesperson confirmed the higher figure as the
current one. Eighteen months before that, the firm was running a few
thousand.

So where does a real reduction show up? It does exist, and it’s worth
stating precisely because the honest version is still significant:
McKinsey went from roughly 45,000 people at the end of 2023 to about
40,000 — [a
drop of more than 10% over about eighteen months, the largest in the
firm’s history](https://www.irishtimes.com/business/2025/05/28/mckinsey-sheds-10-of-staff-in-2-year-profitability-drive/). The firm [rejects
the layoff framing](https://www.peoplematters.in/news/performance-management/mckinsey-confirms-over-5000-layoffs-in-massive-reset-of-global-workforce-45731), attributing the decline to natural attrition
plus its normal performance-management process, alongside a 2023
restructuring that eliminated around 1,400 back-office roles.

That’s about 5,000 people over two years, during a consulting
downturn that followed five years of near two-thirds headcount growth.
Real, consequential, and roughly a fifth the size of the number that
went viral — and driven substantially by a market cycle, not a model
swap.

## The number they cut out is the one that matters

Here’s what bothers me about the popular retelling, and it’s not just
that the figure was wrong. It’s that the correction usually stops at
“actually, nobody got replaced,” which leaves the most useful finding on
the floor.

At CES, Sternfels described the shift as “25 squared.” [Client-facing
consulting roles up about 25%. Non-client-facing roles down about 25%.
And output from that shrinking non-client-facing group up about
10%](https://finance.yahoo.com/news/mckinseys-ceo-breaks-down-ai-100301404.html).

Read that again with a founder’s eyes. The part that got amplified
was the shrinking group. The part that got dropped was that *the
client-facing side grew by a quarter*. This is not a firm getting
smaller. It’s a firm moving people from the back office to the front and
using agents to hold the back-office output up while it does.

And then the sentence that should actually rattle anyone building a
company, which Sternfels said in the same breath:

>
“Our model has always been synonymous that growth only occurs with
total head count growth. Now it’s actually splitting.”

The head of a firm whose entire business model was billable humans
just said the link between headcount and growth is coming apart at his
own company. That’s the finding. It survived the fact-check, it’s on the
record, and it’s more disruptive than the fake version — because the
fake version says “fire people,” which is a one-time event, and the real
version says “your unit of capacity changed,” which is a permanent
structural fact you now have to plan around.

The fabricated number told founders to cut. The real number tells
them to reallocate — and reallocation is a much harder thing to get
right.

## Why the wrong number traveled

It’s worth being honest about why a claim like this spreads, because
the mechanism will produce the next one too, and you’ll be the target of
that one as well.

Numbers about AI displacement are not neutral facts moving through
the world on their own merit. They’re capital-allocation instruments.
Whether they’re true is a secondary property.

Consider the incentives. A large company that has committed enormous
sums to hardware, data centers, and model access needs that spend to
look like it’s working. There’s a shape this takes that I’ve written
about before: [AI made tokens cheap
and it’s making hardware costly](https://sublimecoding.com/blog/ai-tax-hardware-costs), and the companies deepest into that
spend are the ones with the strongest reason to announce that it’s
paying off. Announcing a large headcount reduction attributed to AI does
several things at once — it signals cost discipline to the market, it
justifies the capital expenditure, and it positions the firm as ahead.
Whether the reduction was actually caused by AI or by a demand slump
that would have happened anyway is not a distinction the press release
is built to make.

Now add the investor layer. Venture capital is structurally biased
toward sweeping change — the entire return model depends on finding the
thing that resets an industry, so a claim that an entire labor category
has been automated is exactly the kind of story that attracts capital
toward the companies telling it. That’s not a conspiracy, it’s just what
the incentive gradient looks like. A dramatic claim gets amplified
because amplification serves the amplifier.

So when a number like this crosses my feed, my procedure is short and
it’s mostly one question, asked before anything else: **what does
the person saying this need to be true?**

That’s the same test I ran on [a vendor’s 96%
security benchmark](https://sublimecoding.com/blog/ai-security-benchmark-what-96-percent-means) and on [the way Anthropic’s
safety asks were bundled](https://sublimecoding.com/blog/jose-valim-anthropic-incentive-problem), and it’s the test that would have caught
this one immediately — because the claim was being repeated most
enthusiastically by people selling AI transformation to executives.
After that question, the rest is mechanical: find the primary source,
not the article about the source. Read the actual transcript or watch
the actual talk. Check whether the eye-catching number is a total, a
delta, or a projection, because those get swapped constantly. And check
the timeline — “replaced 25,000 people” and “grew to 25,000 agents over
two years” describe completely different events, and the difference
lives entirely in a verb.

Four minutes. It’s not investigative journalism. It’s just not taking
the headline’s word for it. And it matters, because [the last time everyone repeated a
confident AI story without checking the mechanics, the story was
Amazon’s, and the details were considerably less flattering than the
summary](https://sublimecoding.com/blog/amazon-let-the-ai-drive).

## Addition is the pattern that works

Now the part I actually believe, which is the reason the McKinsey
numbers are interesting rather than just misreported.

The addition pattern is the correct one. Not because it’s gentler,
but because it’s the one that compounds.

The company that trains its existing staff to direct a growing number
of agents ends up with people who can each carry vastly more scope. The
company that cuts staff and hands the remaining work to agents ends up
with fewer people, each of whom is now responsible for reviewing more
output than they can actually review, and no one left with the context
to catch the things that go wrong. One of these is a capability build.
The other is a cost-cutting exercise wearing a technology costume, and I
think it’s short-sighted in a way that will be expensive to unwind. The
teams that cut deepest will be hiring back in eighteen months, at worse
terms, for people who now have to reconstruct institutional knowledge
that walked out the door.

There’s a second-order problem underneath this that I keep coming
back to: agent-heavy development increases your blast radius and your
velocity simultaneously. You can do more things, faster, in more places.
That means *more* needs reviewing, not less — and the reviewing
is the part that requires judgment, context, and someone who will be
there when it breaks. [I’ve
argued that we’re about to stop producing senior engineers](https://sublimecoding.com/blog/stop-making-senior-engineers) precisely
because the pipeline that made them ran through the work agents now do.
Cutting the humans who do the reviewing, to pay for the agents who
generate the things needing review, is the specific move that breaks
this.

## The question to actually put in the board deck

So here’s the replacement for “how many people can AI let us
cut.”

**What’s your human-to-agent supervision ratio, and what’s your
evidence for it?**

I’ll give you my numbers, from running this daily, so you have
something concrete to argue with.

For menial work — organizing files, summarizing notes, mechanical
scans, the grunt tier — I’ll run five to ten agents at once without much
strain. The work is checkable at a glance and the failure modes are
boring.

For actual engineering, writing and changing code that has to work,
it’s **five to six at a time**. Sometimes up to ten,
depending on how good the test coverage is on what they’re touching —
strong tests raise the ceiling because the tests do part of the
supervision for me. Push past that and the quality doesn’t degrade
gracefully, it degrades in a specific way: I stop reading the diffs
properly and start skimming them, and skimming a diff is functionally
the same as not reviewing it. The agents don’t get worse. My attention
does.

It’s a spectrum, and it moves. Some days each agent needs more
babysitting and the number drops. Some tasks are well-fenced enough that
it climbs. Anyone who gives you a fixed universal ratio is selling
something.

For an outside data point: Cherny’s setup at Anthropic runs [roughly
five terminal sessions against separate worktrees plus five to ten cloud
sessions](https://newsletter.pragmaticengineer.com/p/building-claude-code-with-boris-cherny), with sub-agents fanning out underneath. Different tooling,
different scale of delegation, but the number of things one experienced
person actively steers lands in a strikingly similar place.

Now hold that against 25,000 agents and 40,000 humans and do the
arithmetic. That ratio is well under one agent per person, which is
exactly why McKinsey’s version is coherent — Sternfels has framed the
near-term goal as every employee being enabled by at least one agent.
It’s an augmentation ratio, not a replacement ratio. If your plan
involves anything like five or ten agents per remaining human, you are
not proposing the McKinsey strategy. You’re proposing something nobody
has demonstrated, and the binding constraint won’t be model capability.
It’ll be how many diffs a tired person will actually read on a Thursday
afternoon.

That constraint is real, and it’s the one worth designing around. You
can push it — better tests, tighter task scoping, agents that review
other agents, structures inside your agent fleet that mirror the org
structures companies already use for humans. I think that’s where this
goes, and I think one person eventually supervises far more than six.
But every one of those structures is itself something a human has to
build, maintain, and debug. The supervision doesn’t disappear. It
changes shape and moves up a level.

## The thing that doesn’t scale

One more, and it’s the piece I think gets left out of every one of
these headcount conversations.

An agent has no motivation. It doesn’t need to feed anybody. It isn’t
trying to make rent, or get the promotion, or avoid being the person who
broke production in front of the whole team. It has no stake in whether
the company exists next year. It does the work it was handed because it
was told to, and that’s the entire depth of it.

Every reason anything gets done in your company is a human reason.
Someone cares about being good at this. Someone doesn’t want to let
their team down. Someone has ambitions that require this project to
succeed. That layer is not a soft benefit sitting on top of the real
work — it’s the thing that generates the direction the agents then
execute. You can add 25,000 agents and get more throughput. You cannot
add 25,000 agents and get more *intent*, and intent is the
scarcer input.

That’s a bigger argument than fits at the end of this post, and I’ll
make it properly on its own. But it belongs in the frame here, because
the headcount question is usually posed as if people and agents were the
same kind of thing in different quantities. They aren’t. One of them
supplies the reason.

## Before your next hire decision

Three things, concretely.

Check the number that’s driving the decision. Open the primary
source. Ask what the person saying it needs to be true. If a board
deck’s thesis rests on a statistic somebody screenshotted, that’s not a
thesis.

State your supervision ratio explicitly and defend it with something.
Not a vendor’s claim — your own observation of when your people stop
reading diffs carefully. If nobody on your team can tell you that
number, you don’t yet know your actual capacity, and you certainly
shouldn’t be sizing headcount against it.

And ask Sternfels’s question rather than the fake one. Not “how many
people can we replace,” but “which roles move toward the customer, and
what holds up the work they’re moving away from.” That’s the split he
was describing, and it’s a reallocation problem — which is harder than a
cutting problem, and considerably more likely to still be working in two
years. [I’ve made a
version of this argument before about what AI actually does to the size
of the firm](https://sublimecoding.com/blog/nadella-token-capital-small-teams), and everything since has pushed me further in the same
direction.

If you’re a founder trying to size a team against agent-heavy
delivery — and trying to work out which of the numbers landing in your
inbox are load-bearing and which are marketing — that’s exactly the kind
of question I help founders get right before it’s baked into a plan. [Let’s talk](https://sublimecoding.com/consulting).

---

## AI Made Tokens Cheap. It's Making Hardware Costly.

URL: https://sublimecoding.com/blog/ai-tax-hardware-costs
Published: 2026-06-26
Tags: AI, AI startups, founders

>
**TL;DR:** On June 25, 2026, Apple raised prices across
the Mac, iPad, HomePod, and Apple TV lines — the iPad Air jumped $150 to
$749, the Apple TV $70 to $199 — and pinned it on a memory-chip shortage
Tim Cook called a “hundred-year flood,” driven by AI data centers buying
up the world’s DRAM. The reflexive read is “the AI tax has arrived;
everything’s getting more expensive.” That’s half the ledger. The cost
of *intelligence* — the price of a token, of a unit of inference
— is still falling fast. What’s repricing is the *physical
substrate* that runs it. Those two lines move in opposite
directions, and if you’re building anything on AI, the expensive mistake
is modeling only one of them. My read: this isn’t a temporary cycle.
It’s a structural reallocation that holds as long as data centers keep
outbidding the rest of us for the same silicon.

## Apple blinked, and named the reason

Apple does not raise prices on a shipping lineup mid-cycle. It
announces a new product at a new price, lets the old one fall away, and
otherwise holds the line for a year. On June 25 it broke that habit
across nearly the entire non-iPhone catalog at once. The iPad Air went
from $599 to $749. The iPad Pro added $200 to reach $1,199. The
entry-level MacBook Neo climbed to $699, the MacBook Air to $1,299, the
MacBook Pro to $1,999. The Apple TV 4K jumped from $129 to $199; the
HomePod from $299 to $349. The iPhone was the conspicuous exception.

What’s unusual isn’t the increase — it’s that Apple said
*why*, out loud. The company blamed rising costs for memory and
storage chips, and Tim Cook, who has run Apple’s supply chain for over
two decades, [called
the shortage a “hundred-year flood,”](https://9to5mac.com/2026/06/25/apple-price-increases-mac-ipad-more/) adding that he’d never seen
anything like it in over forty years. The market took it as more than an
Apple story: [Apple’s
stock fell roughly 5% and Dell’s more than 8%](https://www.cnbc.com/2026/06/25/apple-macbook-ipad-price-hike-memory.html) as investors priced in
memory pressure across the whole consumer-hardware supply chain. The [full
price table is worth a look](https://www.macrumors.com/2026/06/25/apple-just-increased-prices/) if you want the damage line by line.

The cause Cook named is the part that matters here. The DRAM and
high-bandwidth memory going into AI data centers is the same memory that
goes into a laptop. When demand from one buyer goes vertical, everyone
else in the line pays the spread.

## “The AI tax” is half the story

The easy take wrote itself within the hour: *this is the AI tax,
and it’s coming for everything.* I’d push back on that the same way
I push back on its mirror image. I’ve spent a lot of words on this site
arguing the clean version of the AI-cost story is usually wrong — that
[“AI will shrink your team”](https://sublimecoding.com/blog/ai-wont-shrink-your-team)
is too tidy to be true, and so is “AI will shrink your bill.” The
reality is lumpier, and it’s lumpy here too.

Because at the same moment Apple is raising hardware prices, the
price of *intelligence itself* is doing the opposite. Per-token
inference cost has been on a steep deflation curve for three years and
shows no sign of flattening. I made the broader version of this point
responding to Satya Nadella’s [“token capital”
essay](https://sublimecoding.com/blog/nadella-token-capital-small-teams): the AI capability a firm can buy gets cheaper to accumulate
every quarter. So why is the metal getting more expensive in the very
same quarter?

Because it’s the same demand doing both. The data-center buildout
that drives per-token cost *down* — through scale, competition,
and relentless efficiency work — is the same buildout bidding the price
of every advanced memory die *up*. Cheaper intelligence and
pricier hardware aren’t a contradiction. They’re two readings off the
same gauge. Apple’s laptop and your inference server are now standing in
the same checkout line as a hyperscaler’s GPU cluster, and the
hyperscaler has a much bigger wallet.

## Two lines on the same invoice

If you ship software on AI, this resolves into something concrete:
your costs split into two trends moving in opposite directions.

- **The intelligence line** — API calls, inference,
tokens — trends *down*. Competition and efficiency keep cutting
it, and that’s unlikely to reverse.

- **The substrate line** — the GPUs you rent, the memory
in the box, the dev machines your team runs on a refresh cycle, and now
even the laptops — trends *up*.

Same root cause, opposite slopes. The naive “AI makes everything
cheaper” extrapolation watches only the first line and budgets as if the
second doesn’t exist. For most software teams the token line still
dominates and still falls, so on net you may genuinely come out ahead —
that’s the good news, and it’s real. But the instant your workload leans
on owned or rented hardware — local inference, fine-tuning runs, a fleet
of machines on a three-year refresh — the second line starts to bite,
and it’s the line nobody put in the spreadsheet.

This is just [proving the return](https://sublimecoding.com/blog/prove-the-return)
applied to a cost base that moves under you. You can’t evaluate whether
an AI investment pays off against last year’s hardware prices. Model the
return against the cost you’ll actually face — both lines — or you’re
proving the wrong number.

## Temporary cycle, or the new normal?

So which is it. Memory has always been cyclical; DRAM booms and busts
are a decades-old pattern, and the reflexive analyst take — *this too
shall pass* — is half right. The amplitude is cyclical. The floor
moved.

The demand that reset the floor is training and serving frontier
models, and that is not a fad quarter. It compounds, and it has the
deepest pockets in the economy behind it. As long as a memory die earns
more sitting in a data center than sitting in a HomePod, the data center
wins the allocation and consumer — and small-business — hardware pays
the difference. Prices won’t snap back to 2025 levels, because the
demand curve didn’t briefly spike. It shifted. Advanced silicon quietly
became a contested strategic resource somewhere around the time it
started running the economy’s intelligence, and contested resources
don’t get cheaper because you’d like them to.

So: not a one-time tax, and not a passing cycle. A reallocation. The
cyclical swings will still happen on top of it, but the baseline you’re
pricing against is higher now and likely to stay there.

The question was never whether AI would change how we live and build.
It’s which side of the invoice you’re standing on when the bill arrives.
If you’re buying intelligence, it keeps getting cheaper — enjoy it while
it lasts. If you’re buying the hardware that runs it, get used to the
new number. Apple just did the rest of us the favor of printing it on a
price tag.

The obvious dodge is to stop buying the Mac. I tried it — [Omarchy on an M1 via Asahi
Linux](https://sublimecoding.com/blog/omarchy-asahi-m1-macbook) — and the tax doesn’t disappear so much as change
currency.

---

## When to Trust an Agent and When to Step In

URL: https://sublimecoding.com/blog/when-to-trust-an-agent-and-when-to-step-in
Published: 2025-12-22
Tags: AI, agents, productivity, engineering

**The hardest part of working with agentic AI in 2026 isn't getting the agent to do the work. It's knowing when to override it.**

An agent that produces useful code 90% of the time and confidently broken code 10% of the time is not a 90% solution. It's a system you have to be paying attention to all of the time, because the 10% of broken output looks indistinguishable from the 90% of correct output if you skim. Trust calibration is the actual engineering skill of working with agentic AI, and most teams don't yet have a framework for it.

What follows is the framework I use, in two parts. First: a four-level autonomy ladder for deciding how much trust to extend to an agent on any given task. Second: the five signals that an agent is currently going off the rails, even when its output looks fine.



## The autonomy ladder, at a glance

 Four levels of AI agent supervision, set by the consequences of the agent being wrong:



 - **Level 1 — Read-only.** Agent observes but changes nothing. Code analysis, summarization, search. Run autonomously all day.

 - **Level 2 — Bounded write.** Small, clearly-scoped writes — tests for an existing function, a single-file refactor. Review the diff before merging, but don't read every line.

 - **Level 3 — State-changing.** Payments, auth, user data, system state. Agent can draft, but every line gets human review before it lands.

 - **Level 4 — Public-facing or irreversible.** Production schema migrations without rollback, customer-facing email, public posts. Do not delegate.



 The five failure signals that demand human takeover — confidence rising as ground truth gets thinner, plausibility without verification, scope creep mid-task, the agent contradicting itself across messages, and silent retries — are detailed below.

## The four-level autonomy ladder

Not all tasks are equal. The same agent can be entirely trustworthy on one task and dangerous on another. The level of supervision should be set by the consequences of the agent being wrong.

### Level 1 — Read-only, always trust

Tasks where the agent observes but doesn't change anything. Code analysis, documentation generation from existing code, summarization, search. The worst-case outcome of the agent being wrong is that I get bad information that I have to discard.

I let agents work autonomously at this level all day. The downside is bounded.

### Level 2 — Bounded write, mostly trust, verify

Tasks where the agent writes code in a clearly-scoped area. Adding tests for an existing function. Implementing a small utility from a clear spec. Refactoring a single file. The blast radius is small, the work is reviewable, and the agent has a high probability of getting it right.

I review the diff before merging, but I don't read every line carefully. I'm looking for obvious smell — duplicate logic, weird naming, missed edge cases. If the diff looks clean and the tests pass, I merge. The downside if I miss something is one bad commit that's easy to revert.

### Level 3 — Real-money, auth, or state-changing, verify line by line

Tasks that touch payments, authorization, user data, or system state in ways that matter. The agent can *draft* these, but every line has to be reviewed by a human before it lands.

This is the level where most teams lose discipline first. The agent produces a plausible-looking auth migration, the diff isn't huge, the tests pass — but the migration is silently introducing a privilege escalation. I've seen this happen. I've nearly let it happen, which I'll talk about in a moment.

The discipline at this level: read every line, run the change against your own threat model, ask the agent why it made each non-obvious choice. Treat the agent's output as a junior engineer's work that needs senior review before merge.

### Level 4 — Public-facing or irreversible, do not delegate

Tasks where the cost of getting it wrong is unrecoverable. Schema migrations on production data without rollback. Sending email to your customer base. Posting to a social media account. Public-facing legal text. Press statements.

Agents do not produce these autonomously. They can *draft* them — which can be useful — but the work of actually committing to the output is human-only. The asymmetry is too sharp; even a 99% reliability rate produces an unacceptable error rate over many actions.

## How do you know when an AI agent is going off the rails?

Even within the right autonomy level, individual sessions can drift. The signals that something is off, in approximate order of how often I see them:

### 1. Confident answers without specifics

"This will work because of how the framework handles state." Without naming the function, the file, or the documented behavior. The agent is filling in plausible reasoning rather than checking. Push back: "Show me the line in the codebase that demonstrates this."

If the agent can't, the answer is suspect. About half the time the agent then says "actually, on closer inspection..." and revises. The other half it doubles down on a wrong claim, which tells you the entire reasoning chain is hallucinated.

### 2. Multiple files changed for "one fix"

You ask for a fix to a single bug. The diff comes back touching seven files. Sometimes this is correct — the bug genuinely was scattered. More often, the agent has decided the codebase needs "consistency" or "improvement" while it was in there.

The discipline: ask why each file was changed. If the answer for any file is anything other than "this was necessary for the fix," revert that file's changes. Scope drift in agent diffs accumulates fast.

### 3. "Cleaning up" code unrelated to the task

A subtype of the above. The agent removes a comment it didn't understand. Renames a variable. Reformats a function it thought was ugly. None of these are explicit instructions. All of them produce noise in the diff.

This is a hard category to police because the changes look harmless one at a time. The cumulative effect over a quarter is a codebase whose history is incomprehensible because every fix touches twenty unrelated lines.

### 4. Confidence that contradicts evidence

Tests are failing. The agent says "the implementation is correct, the tests must be wrong." This is occasionally true. It is usually wrong. The signal is the agent privileging its own reasoning over the failing test.

The fix: never let the agent dismiss a failing test without proof. "Show me which assertion in the test is incorrect and why" is the right pushback. Most of the time the agent then realizes the implementation is wrong.

### 5. Speed too high for the complexity

This one is the hardest to articulate but the most reliable in retrospect. A complex problem is solved in twelve seconds with one paragraph of explanation. Be suspicious. Real engineering problems usually have layers; an instant answer often skips them.

The discipline: when the answer comes back faster than seems reasonable, ask the agent to explicitly enumerate three alternatives and argue for the chosen one. The "argue" step surfaces whether the agent has actually thought about the problem or pattern-matched to a familiar shape.

## A concrete example: the auth refactor

Earlier this year I asked an agent to refactor a small piece of auth code. The original function had grown to 200 lines and could be reasonably split into four. Routine work, level-3 task by my own framework — auth is in the "verify line by line" category.

The agent produced a clean four-function refactor in about 90 seconds. The tests passed. The diff was the right size. On a quick read it looked correct. I almost merged it.

Two of the five signals fired before I clicked merge. Speed too high (signal 5): the refactor was elegantly factored, which is suspicious for a 200-line function with the gnarly history this one had. And confidence without specifics (signal 1): when I asked the agent why it had moved a particular permission check from one branch to another, the answer was "it's cleaner this way" rather than "the original ordering was incorrect because X."

I dug in. The "cleaner" reordering had introduced a small but real privilege escalation: in one specific code path, an authorization check that previously ran *before* a sensitive operation now ran *after*. In the test suite, no test exercised that exact path, so the tests passed. In production, the bug would have allowed certain users to perform an action they shouldn't have been able to.

The discipline that caught it was the framework above. Without it, I'd have merged a clean-looking refactor and shipped a real security bug. Multiply by every team using AI for code generation, and you can see why the slop problem is real and the trust calibration is the actual engineering work of 2026.

## The meta-discipline

The framing that pulls all of this together: *agents are tools, not teammates*. They don't have stakes. They don't get yelled at when production breaks at 3am. They have no embodied sense of what's risky. They will confidently produce code that's 99% right, and the 1% wrong will sometimes be catastrophic, and they will not know.

The job of the engineer working with agents is to supply that missing sense of stakes. The autonomy ladder is how you decide when to engage. The five signals are how you stay engaged once you're in the work.

None of this is hard. It's just disciplined. The teams that get this right are using agents at the limit of what's possible without producing slop. The teams that don't are accumulating a debt that becomes obvious only after a major incident.

Calibrate trust. Verify the consequential changes. Override when the signals fire. The framework is the discipline.

## How do you roll out AI agent supervision rules to a whole team?

Individual discipline isn't enough at team scale. If only one engineer on a four-person team is calibrating agent trust carefully, the other three's slop ends up in the codebase anyway. The patterns above need to be team-level practice.

The lightest-weight version of this that's worked for me: a single shared document — call it `AGENTS.md` — that lists the team's autonomy levels, the signals to watch for, and the kinds of work that always require human review. The doc is short, written by the team together, and reviewed quarterly.

Pair it with one tactical practice: every PR description includes a single line at the top stating which autonomy level applied to the work. "Level 2 — bounded write, agent-assisted." "Level 3 — auth code, full human review." This makes the trust calibration legible during code review and surfaces drift before it ships.

The combination of the doc and the PR-line discipline takes about a week to introduce and a quarter to internalize. The teams that do this end up with a meaningfully better signal-to-noise ratio in their AI-assisted work than the teams that don't.

## Read this next

- [**My Daily Agentic AI Workflow**](https://sublimecoding.com/blog/my-daily-agentic-ai-workflow) — The day-to-day mechanics of running multiple agents at once.

- [**AI-Assisted Engineering Isn't Faster Coding**](https://sublimecoding.com/blog/ai-assisted-engineering-is-a-new-workflow) — The team-level discipline that complements the individual one above.

- [**How I'd Run Security at an AI-Native Company in 2026**](https://sublimecoding.com/blog/running-security-at-an-ai-native-company-2026) — Where agent-trust meets the production security threat model.

---

## How I Triage a New Codebase in 90 Minutes

URL: https://sublimecoding.com/blog/triage-a-new-codebase-90-minutes
Published: 2025-12-08
Tags: engineering, productivity, engineering leadership, vCISO, staff engineer

**A fractional engineering engagement starts with a codebase you've never seen. You have ninety minutes to form a useful POV before the kickoff call.**

This is a recurring situation for me as a [fractional engineer](https://sublimecoding.com/consulting). A founder books a discovery call, gives me read-only access to their repo on Tuesday, and the kickoff call is Wednesday morning. Between those two events, I need to know enough about the system to ask intelligent questions, identify the load-bearing risks, and not waste the founder's time with surface-level observations they could have written themselves.

The framing that makes this work: *not* "understand the codebase." That takes weeks. Instead — *find the load-bearing risks*. The seven-step triage I run, in order.



## The 90-minute triage, at a glance



 - **0–10 min — README + architecture docs.** What does the team think the system does?

 - **10–15 min — Dependency audit.** Major frameworks and how out-of-date everything is.

 - **15–25 min — git log story.** Who works on what, how big are the commits, what's the recent panic?

 - **25–35 min — Tests and CI signal.** What's covered, what's skipped, what's flaky.

 - **35–45 min — The hottest files.** The five files that change most. That's where the load-bearing risk lives.

 - **45–60 min — Auth and data layer.** The two places a small mistake becomes an incident.

 - **60–70 min — The most recent incident.** What broke, what fixed it, what didn't.

 - **70–90 min — Notes consolidation.** The five questions you'll ask on the kickoff call.



## Step 1: README.md and any architecture docs (10 minutes)

Start at the front door. The README tells me three things almost immediately: how the team thinks, how recent the project's intentional documentation is, and whether the founders or engineers wrote it.

Signals to look for: when was this last meaningfully updated? Does the "how to run locally" section reference real commands or stale ones? Are there architecture diagrams, ADRs, or design docs anywhere in the repo? An empty `docs/` directory and a README that ends with "TODO: write more about this" tells you a great deal about engineering culture before you've read a single line of code.

I copy the relevant facts into my own running notes. The first paragraph of those notes is "what the team thinks the system does."

## Step 2: Dependency audit (5 minutes)

Open `package.json`, `mix.exs`, `requirements.txt`, `go.mod`, or whatever the equivalent is for the language. Two questions: what major frameworks are in use, and how out of date is everything?

The dependency list is a faster summary of the system's architecture than reading the architecture docs. If I see `phoenix_live_view` + `oban` + `ecto`, I know the shape of the app. If I see thirty random utility libraries and no obvious framework, I know there's been turnover or a lack of opinionated leadership.

For the freshness check: a dependency that's two majors behind isn't automatically bad, but a codebase where *every* major dependency is two-plus versions behind is a maintenance bomb in the making. Make a note.

## Step 3: The git log story (10 minutes)

Run `git log --oneline --all -200`. Skim the last two hundred commit messages.

What you're looking for: who's committing, what they're committing, and what the rhythm looks like. A repo where one engineer wrote 90% of the recent commits is a key-person-risk situation. A repo where the commits are mostly "wip" and "fix typo" tells you about the team's commit hygiene. A repo where every PR has a clear, conventional-commit-style message tells you the team has invested in process.

The signal that matters most for triage: pattern frequencies. Lots of "revert" commits in the recent past = unstable changes. Lots of "fix" commits referencing one specific module = that module is troubled. Use the patterns to direct what you read next.

## Step 4: Test coverage and CI signal (10 minutes)

Find the test directory. Count files. Find the CI configuration. Read it. Run the test suite if you can — does it pass? How long does it take?

You're looking for three signals:

- **Volume.** Is there one test file or two hundred?

- **Quality.** Open three random test files. Do they test behavior or just exercise code?

- **CI status.** Is the build green? Has it been red for more than 24 hours? Is there a culture of merging on red?

The test suite is the closest thing a codebase has to a self-portrait. A team that's invested in test quality is a team that takes engineering seriously. A team where the tests don't pass on a fresh checkout is a team that has bigger problems than the ones the founder is going to tell you about.

## Step 5: The hottest files (10 minutes)

Run a quick analysis to find the most-changed files in the last six months. `git log --pretty=format: --name-only --since="6 months ago" | sort | uniq -c | sort -rg | head -20`. The top of that list is where the action is.

Open the top three or four files. Read them. These are the load-bearing parts of the system, by definition — they're where the team is spending their engineering energy.

What you're looking for: are these files long, complicated, and full of inline comments saying "TODO: this is a hack"? Or are they crisp, well-factored, and recently refactored? The hot files tell you where the system is fragile and where it's healthy. They're also the files that an incoming engineer (or fractional engagement) will most likely need to touch first.

## Step 6: The auth and data layer (15 minutes)

Now the deep dive. Find the authentication code. Find the database schema. Read both carefully.

Authentication: how does a user log in? Where are sessions stored? Is there MFA? Are passwords hashed with a current algorithm? Is there an obvious authorization layer beyond authentication? This is where I find the highest-severity bugs in early-stage codebases — usually authorization issues that the team hasn't yet noticed because they haven't been exploited.

Data layer: what tables exist, what relationships do they have, are there migrations that suggest the schema has been refactored, are the indexes sensible? The schema is the contract the system runs under. Anything wrong with it is wrong with everything else.

This is the longest step in the triage by design. If I'm going to find a deal-breaker risk, it's almost always in this section.

## Step 7: The most recent incident (10 minutes)

Ask the founder (or look in the team Slack history if you have access): when was the last production incident? What broke? How was it fixed?

The incident report — written or verbal — tells you more about the engineering culture than any single artifact in the repo. A team that has a clear retro doc with five action items, three of which were completed, is a team that learns. A team where "the last incident" is met with "uh, well, last week the database fell over for an hour, I think someone restarted it" is a team that doesn't.

This step also surfaces the thing the founder is most worried about. They'll volunteer it once they sense you're asking real questions.

## What should you actually write down after triaging a codebase?

Ninety minutes of triage produces a one-page document with the following sections:

- **What the team thinks the system does** (one paragraph, from step 1)

- **The shape of the stack** (three lines, from step 2)

- **Engineering rhythm** (commit frequency, team size, hot spots — from steps 3 and 5)

- **Quality signals** (test coverage, CI, code-review hygiene — from step 4)

- **The two highest-severity risks I found** (from steps 6 and 7)

- **The two questions I'm bringing back to the founder**

The two questions are the most important output. They're how you signal to the founder that you've done real work in 90 minutes and have a useful POV. They're also the questions whose answers will reshape everything you do in the engagement.

Examples I've actually used: *"Is the lack of audit logging on the admin endpoints intentional?"* Or: *"Looking at the last six months, your messaging service has been the source of three of the four production incidents — what do you and the team think is going on there?"*

## The AI-assisted version

The 90-minute number above predates Claude Code. The same triage now takes about half that time with AI assistance.

The pattern: I run the seven steps in parallel using a Claude Code session pointed at the repo. Step 1, 2, 3, 4, and 5 are largely automatable — I tell Claude what to look for and it produces structured summaries faster than I can read the raw files. I spend the saved time on steps 6 and 7, which still benefit from human attention.

Net result: the same depth of triage in 45 minutes instead of 90, with substantially better notes because the AI captures things I would have skimmed. The disciplined human still drives. The agent accelerates.

## The takeaway

The skill being practiced here is not "reading code fast." It's *knowing what to look at first.* Most engineers, on entering an unfamiliar codebase, dive into the part of the system most relevant to their immediate task and form a partial picture. The triage protocol forces you to look at the system in the order that surfaces risks, not in the order that matches your task.

Useful for fractional engineers. Useful for new hires in their first day. Useful for anyone who's about to take on technical responsibility for a codebase they didn't write. Run the protocol. Take the notes. The 90 minutes will save you weeks downstream.

## What do you actually say on the kickoff call after a 90-minute triage?

One last note. The triage produces a written one-pager, but the founder is going to want to talk through it on the kickoff call. The framing that lands: lead with what you saw that's working, then name the two highest-severity risks, then ask the two questions.

This sequence is deliberate. Founders are sensitive about their codebase — many haven't had an outsider look at it in years. Leading with criticism puts them on defense and shrinks the conversation. Leading with what's working buys you the credibility to then talk about what isn't.

The two questions you ask are the most important moment of the call. They signal you've done real work, not just listed problems, and they invite the founder into a conversation about priorities rather than a lecture about deficiencies. Done right, the kickoff call ends with the founder saying some version of "let's start with those two things you flagged" — which is exactly the engagement you wanted to land.

## Read this next

- [**The 'Smallest Possible Slice' Heuristic**](https://sublimecoding.com/blog/smallest-possible-slice-shipping-complex-features) — The same pragmatic-decomposition mindset applied to feature work instead of code review.

- [**Migrating 225K Users from AWS Cognito to Auth0**](https://sublimecoding.com/blog/aws-cognito-to-auth0-migration-without-forcing-logout) — A real example of step 6 (auth-layer scrutiny) playing out in production.

- [**AI-Assisted Engineering Isn't Faster Coding**](https://sublimecoding.com/blog/ai-assisted-engineering-is-a-new-workflow) — How the AI-assisted version of triage scales to other engineering tasks.

---

## My Daily Agentic AI Workflow

URL: https://sublimecoding.com/blog/my-daily-agentic-ai-workflow
Published: 2025-11-24
Tags: AI, agents, AI tools, productivity

**I run four to seven agent sessions in parallel through a normal engineering day. Here's what they do, what they don't, and how I keep the work coherent.**

The defining shift in engineering work over the past two years isn't that AI writes code faster. It's that you can have multiple agents working on multiple things at the same time, and your job moves from "writing code" to "directing work." This is qualitatively different from autocomplete, copilot-style assistance, or any prior way of using AI in development.

What follows is a walkthrough of a typical engineering day for me as of late 2025, running on Claude Code as the primary tool, OpenAI Codex for shell-shaped tasks, and a few custom agents wired into Slack and the command line. The point isn't the tools — pick whichever you like. The point is the patterns.

## The core idea

Agents take work off your plate but still need you in the loop. The trap most teams fall into: treating agents as fire-and-forget background workers. The result is generated code that compiles, looks reasonable, and is subtly wrong in ways the agent has no way to detect on its own.

The mental model I use: an agent is a fast, talented junior engineer with infinite patience and zero context outside what you give them. Your job is to give them the context, scope the work tightly, and review the output before it goes anywhere production-shaped.

## Walkthrough of a typical day

### Morning: 2 background agents on long tasks

Before I sit down at my desk, I usually have two agent sessions running. The pattern: tasks that take a long time, don't need real-time feedback, and have well-defined success criteria.

Examples from a recent week:

- "Audit our codebase for places we're calling external APIs without retry logic, and produce a markdown report with recommendations."

- "Read the last fifty PRs merged to main, identify recurring code-review feedback themes, and write an internal style-guide draft based on them."

- "Generate test cases for the authentication module covering all edge cases listed in the threat model document at `docs/auth-threats.md`."

I kick these off, walk away, and come back to a draft in 30–60 minutes. Critically, the output is always a *draft*. I read it, I edit it, I push the parts I trust into the codebase. The agent never commits unsupervised.

### Mid-morning: foreground pair-programming with one agent

This is the bulk of my actual coding. I open a fresh agent session for the hardest problem of the day and we work it together — me driving, the agent acting as a peer. The interaction is conversational, not delegating.

Concretely: I describe the problem, the agent asks clarifying questions, we sketch an approach, I write some code, the agent reviews, I push back on suggestions I don't like, we iterate. By the time the function is committed, both of us have looked at every line.

The mistake to avoid in this mode: letting the agent write the code while you watch. That's still autocomplete, just slightly fancier. The point of pair-programming with an agent is that *you* are still doing the engineering — the agent is checking your work, raising things you might miss, and accelerating the parts that don't require taste.

### Afternoon: review-mode agents on PRs

By afternoon I've usually written or merged some code, and I have PRs to review (mine and the team's). I run a review-mode agent on each PR before I read it myself.

The instruction is consistent: *"Review this diff like an adversarial senior engineer who hates my work. Find bugs, race conditions, security issues, and unclear naming. Don't be polite."* The agent produces a list. I read the list, dismiss the noise (typically 60–70%), and the rest becomes my review comments — credited to me, of course, but with the agent doing the first pass.

The leverage here is significant. The agent catches a real bug or smell about 30% of the time. The other 70% is dismissable noise that I'd have generated mentally anyway. Net: I write better PR reviews in less time, and my human reviewers catch things they otherwise wouldn't have.

### End-of-day: ops agents on deploy + summary

Late in the day, two more agents come into play.

The deploy agent: a custom Slack bot wired to our deployment pipeline. I tell it "deploy main to staging" or "deploy 4f3a2b1 to production behind feature flag `ai_v2`" and it executes the relevant commands, watches the deploy, and reports back. It does not, ever, have permission to do production deploys without a confirmation. But staging deploys, log queries, and rollback prep — yes, autonomously.

The summary agent: at end-of-day it reads the day's commits, the day's PR comments, the day's Slack threads in our team channel, and produces a one-paragraph "what happened today" summary. Useful for me; useful for async teammates; surprisingly useful when I come back on Monday to remember what we were working on Friday.

## The three interaction modes

Boil all of the above down and there are really three modes I use agents in. Each one has a different signature.

- **Delegate.** Long-running task, well-defined output, light supervision. Background mode. The success criterion is whether the deliverable is useful when I come back to it.

- **Collaborate.** Real-time pair programming. The success criterion is whether the code I commit at the end is meaningfully better than what I'd have written alone.

- **Verify.** Adversarial review of my work or the team's. The success criterion is whether real bugs get caught before they ship.

The mistake teams most often make is using the wrong mode. Trying to delegate something that actually needs collaboration produces unusable code. Trying to collaborate when verify-mode is what's needed produces echo-chamber agreement instead of real review. Pick the mode deliberately.

## The handoff protocol

The single discipline that keeps multi-agent work coherent: explicit handoff context between sessions. When one agent's work feeds into another's, you don't trust them to figure it out. You write a one-paragraph context dump and paste it into the next session.

For example: morning audit agent produces a list of 14 places where retry logic is missing. I review the list, decide which 6 are worth fixing, and write a paragraph: "We're going to add retry logic to these 6 functions: [list]. Use exponential backoff with jitter, max 3 retries, log each retry at `warn` level. Match the pattern in `lib/external/retry.ex`." That paragraph goes into a fresh agent session for the implementation work.

The agent doesn't see the original audit. It sees the curated context. This is the core of working with agents at scale: *you* are the context router, deciding what each session needs to know.

## The trap that produces slop

Most teams who report disappointing results from agentic AI are running into the same failure mode: agents that look productive but produce slop. The signature: lots of code is committed, the team feels productive, and three weeks later the production codebase is a mess of subtly broken patterns nobody can fully explain.

The cause is almost always one of three:

- **No review discipline.** Agent-generated code is going into the repo without a human pass.

- **Mode mixing.** Delegate-mode work being treated as collaborate-mode by the team, so nobody is closely engaged with the output.

- **Context starvation.** Agents being asked to do work without enough context to do it well, producing plausible-but-wrong code.

All three are solvable. None of them are solved by "use a better model." They're solved by team-level discipline about how AI work enters the codebase. Without that discipline, more agents produces more slop. With it, the throughput gain is real and durable.

## The takeaway

Agentic AI is a force multiplier in engineering when treated as a workflow change rather than a tool swap. Four to seven sessions in parallel sounds like a lot until you recognize that most of them are running asynchronously while you do other work — and that your role across all of them is the same: provide context, scope tightly, review carefully.

The teams shipping 40–55% faster aren't typing more. They're directing more. That's the new bar. Most engineers will get there in 2026. The ones who get there first will have a meaningful, compounding advantage for the next two or three years before everyone catches up.

## Tooling and cost

The economics of running 4–7 agent sessions a day are easy to get wrong. A few practical notes:

- **Pay for the paid tier.** Free-tier rate limits will produce flow-state interruptions multiple times a day. The $200–600 per month for unlimited Claude Code, ChatGPT Pro, and Codex usage is the highest-ROI line item on your engineering bill at this stage.

- **Don't run the same task in two tools "for comparison."** Pick one. Comparison runs sound disciplined and produce cognitive overhead that erodes the throughput gain.

- **Track cost per outcome, not cost per session.** A $4 agent session that produces a working feature is cheaper than a $0.40 session that produces noise. Most teams track the wrong number.

- **Agent context is the expensive resource, not tokens.** Spend more time on writing good prompts and feeding the right files. Don't optimize for shorter prompts; optimize for clearer ones.

One closing note on team adoption. The workflow above is what I run as an individual. Scaling it to a team adds a coordination problem — multiple engineers spawning agent sessions in shared codebases, occasionally producing conflicting changes if not careful. The pattern that's worked: agents work on isolated branches, never main, and the human engineer is the one merging up. Treat agent sessions like junior engineers' branches, with the same review discipline.

## Read this next

- [**AI-Assisted Engineering Isn't Faster Coding**](https://sublimecoding.com/blog/ai-assisted-engineering-is-a-new-workflow) — The team-level framing of the same workflow.

- [**When to Trust an Agent and When to Step In**](https://sublimecoding.com/blog/when-to-trust-an-agent-and-when-to-step-in) — The discernment side of the discipline above.

- [**How I'd Hire a Staff Engineer at an AI Startup**](https://sublimecoding.com/blog/how-id-hire-a-staff-engineer-at-an-ai-startup) — The AI-fluency bar I'd hire against.

- [**Prompt Skills, Not Claude: Four Rules from Anthropic's Engineers**](https://sublimecoding.com/blog/prompt-skills-not-claude) — Why prompt-engineering moved from the chat to the folder.

---

## 4,154 Commits in Six Months With AI Agents

URL: https://sublimecoding.com/blog/4154-commits-six-months-ai-agents
Published: 2026-07-06
Tags: AI, agents, productivity

>
**TL;DR:** In six months of 2026, I’ve committed 4,154
times to a single repo. My previous average was 500–1,500 commits per
year, across everything. The project — [EnergyConnect](https://energyconnecter.ai), a bulk liquid and
chemical marketplace — went from zero to production in sixty days. I’ve
written almost none of the code by hand. I guided every line of it,
reviewed every line of it, and broke things I didn’t catch at a pace
I’ve never broken things before. Lines of code is a terrible metric.
Commit count is a terrible metric. But zero to production in sixty days
with a team this small is not a metric — it’s a receipt.

## The screenshot that made me write this

I pulled up my GitHub contribution graph a few days ago and genuinely
didn’t believe it.

4,154 contributions in 2026. We’re barely past the halfway mark. My
entire 2024 was maybe 1,200. My 2023 was somewhere around 800. And those
were years I’d have called productive — I was shipping features, closing
tickets, doing the work.

This year I’ve shipped more code than the previous four years
combined, and I have written almost none of it by hand.

The commits are real. They compile, they pass tests, they run in
production, and customers are using the software behind them. But what
they represent is fundamentally different from what a commit meant to me
two years ago, and I think it’s worth being honest about what changed,
what got better, what got worse, and what it means if you’re watching
your own commit graph spike and wondering whether any of it is real.

## What 4,154 commits actually are

Let me kill the first objection before it forms: no, this is not
commit noise.

There’s a reasonable person’s version of that argument — that agents
commit more often, in smaller chunks, and inflate the count relative to
how a human would batch work into larger, less frequent commits. That’s
true. But I’d argue that’s a feature, not an artifact.

When I was writing code by hand, a typical commit was a bundle.
“Implement carrier onboarding flow” — a commit message covering 400
lines across eight files. If something broke in that bundle three weeks
later, good luck bisecting it. The commit message told you nothing about
which of those eight files was the problem.

The agent commits differently. It commits after each logical change —
a function, a migration, a test, a style fix — with a message that
describes exactly what changed and why. We’ve instructed it to do this
deliberately, and we’ve instructed it to attach Linear ticket IDs to
every commit. The result is that our git history is, for the first time
in my career, actually searchable. It’s documentation. It’s the
changelog we always said we’d write and never did.

So yes, the commit count is higher. It’s higher because each commit
is more precise, more traceable, and more useful than the bloated bundle
commits I used to write by hand. The 4,154 isn’t noise. It’s granularity
we never had before.

## Zero to production in sixty days

The project those commits live in is [EnergyConnect](https://energyconnecter.ai) — a SaaS marketplace
for bulk liquids and chemicals. It sits as middleware between carriers,
brokers, suppliers, and end customers. The kind of platform that has a
dozen user roles, complex multi-path forms, regulatory requirements, and
integration points with external systems.

We went from an empty repo to production in sixty days.

We’re now at roughly 120 days, and the project is more mature than
some codebases I’ve worked on for years at previous companies. We’re
close to shipping a mobile app — the first mobile app I’ve ever built. I
would not have had the confidence to say “yes, I can do that” twelve
months ago. Not because I couldn’t learn it, but because the calendar
math didn’t work. Learning a new platform, a new set of conventions, a
new build system, and actually shipping something production-grade —
that’s a quarter of focused work for an experienced mobile developer. I
am not an experienced mobile developer. But I can describe what I want,
review what comes back, and iterate until it’s right. That loop is fast
enough now that the calendar math works.

The codebase has 1.5 million lines across it. I need to be clear
about what that means: it’s not 1.5 million lines of hand-crafted,
artisan code. It includes generated code, vendor code, scaffolding,
configuration, and all the other things a real project accumulates.
Lines of code is a terrible metric for measuring value — always has
been. But it is a useful metric for measuring *surface area*, and
surface area is where the real story gets complicated.

## More code, more blast radius

Here is the thing nobody talks about when they celebrate AI-assisted
shipping speed: every line of code you ship is a line of code you have
to maintain, debug, and answer for when it breaks at 2am. I’ve [written about this at the org
level](https://sublimecoding.com/blog/ai-wont-shrink-your-team) — velocity creates surface area. This is what it looks like
from inside one codebase.

EnergyConnect has a carrier onboarding flow with six completion paths
— different carrier types, different document requirements, different
approval chains. An agent will nail the path you described in your
prompt and silently break two others. We had a week where every fix to
the hazmat carrier flow introduced a regression in the general freight
flow. The agent had no persistent memory of the other paths unless we
explicitly described them or it read the whole file. [Memory helps](https://sublimecoding.com/blog/ai-memory-outlives-any-agent) — once you
correct a mistake, a well-configured agent won’t make the same one
twice. But you have to catch it the first time, and catching it the
first time in a codebase growing this fast is genuinely hard.

This is the part of the AI productivity story that the conference
talks skip. Yes, you can ship faster. But “faster” includes faster bugs,
faster regressions, and faster accumulation of code that nobody has
deeply read. The velocity is real. The risk scales with it.

## The snake eating its own tail

Our review workflow is the part that sounds absurd until you realize
there’s no alternative.

We use adversarial review agents — agents specifically configured to
be skeptical, to look for the exact failure patterns we’ve learned to
expect, to challenge assumptions and check edge cases. Then we do a
manual pass — actually load it up, click through it, look at it with
human eyes. Then a final review before production.

There is no version of this where a single human reviews 4,154
commits by hand. The math doesn’t work. At five minutes per commit — and
that’s fast for a real review — you’re looking at 345 hours of review.
That’s two months of full-time work just reading diffs, doing nothing
else.

So the AI reviews the AI’s work. And I review the AI’s review. And
yes, this is the snake eating its own tail. I’m aware. The question is
whether the alternative — shipping less, or shipping without review — is
better. It isn’t. The adversarial review catches real bugs. Not all of
them, but enough that the process has earned its place. The [specific patterns worth
naming](https://sublimecoding.com/blog/things-ai-coding-agents-get-wrong) — the confident wrong answer, the silent regression, the
context-window cliff — repeat predictably enough that you can build
review agents around them. The bugs that get through are the ones where
both the writing agent and the reviewing agent share the same blind spot
— usually an implicit assumption about business logic that neither one
has enough context to question.

That’s where the human still earns their keep. Not in reading every
line, but in knowing which lines to look at — the ones where the
business context is subtle enough that an agent’s confident “looks good”
means nothing.

## What I gained

I can ship things I’ve never built before. That’s the headline gain,
and it’s not small.

A mobile app. A complex multi-tenant marketplace. Integration
patterns I’d have spent a week researching and another week implementing
— done in an afternoon with an agent that’s read the docs I haven’t.

I write JavaScript as confidently as I write Elixir as confidently as
I write Go. Not because I learned three more languages, but because
syntax stopped being the bottleneck. I can pick up where someone left
off in a codebase I’ve never seen, in a language I’ve barely used,
because the skill that matters now is problem decomposition and review —
not remembering whether the function signature takes the error first or
last.

The agents are better at certain kinds of work than I am.
Boilerplate, scaffolding, the tedious mechanical work that used to eat
the first two hours of any feature — they do it instantly and correctly
and I never have to think about it again. That’s real time back. That’s
afternoons I spent on architecture instead of plumbing.

And the ticket hygiene — I almost can’t believe I’m saying this — is
genuinely better. The agents attach ticket IDs to commits, update
statuses, close tickets with references to the commits that resolved
them. The project management layer that every team says they’ll maintain
and nobody does? It’s maintained. Because the thing doing the work
doesn’t get tired of process.

## What I lost

There was a simplicity in writing code by hand that I miss more than
I expected to.

Not the syntax — I don’t miss semicolons or arguing about bracket
placement. I miss the *solving*. The specific feeling of staring
at a problem, tracing through the logic in your head, and arriving at a
solution that you built from nothing. The agent gives you the solution
in four seconds, and it’s usually right, and you saved an hour, and you
lost something you can’t quite name.

Syntax used to be interesting to me. It isn’t anymore, because we
don’t need to care about it. That’s probably good — it’s probably the
same as how accountants stopped caring about long division when
spreadsheets arrived. But it was a thing I enjoyed, and it’s gone, and I
notice its absence.

The role shift is real. I am not a software engineer in the same way
I was eighteen months ago. I am a director and reviewer of software
engineering, performed at high speed by something that is very good at
the mechanical parts and genuinely bad at the judgment parts. The
judgment is what I provide. The judgment is, I think, all I provide now.
And the uncomfortable question underneath all of it is whether judgment
alone — without the reps of actually writing code — degrades over time.
I don’t have an answer yet. Ask me in a year.

## The math changed

At the end of the day, business is dollars and cents. Money in, money
out. A solo developer — or a very small team — shipping a production
marketplace in sixty days is a different economic equation than the same
project taking six developers nine months. The math has changed.

But the math has changed in both directions. You can ship more,
faster, cheaper. You also have to review more, catch more, maintain
more. The blast radius of your output has grown, and if you don’t invest
in the review layer — adversarial agents, manual verification, real QA —
you’re not shipping faster, you’re *failing faster*. The commit
count goes up either way. What determines whether those commits are an
asset or a liability is whether you respected the blast radius.

4,154 commits. More than the previous four years combined. People are
using the software. The bugs are real, the features are real, and the
pace is real.

I just didn’t write most of it.

---

**If you’re building with agents too:** I wrote about [what agents get wrong](https://sublimecoding.com/blog/things-ai-coding-agents-get-wrong)
most often, [why AI won’t shrink
your team](https://sublimecoding.com/blog/ai-wont-shrink-your-team) the way you think, and [what my daily agentic
workflow](https://sublimecoding.com/blog/my-daily-agentic-ai-workflow) actually looks like — the boring, repeatable version of
what this post describes.

---

## Your Coding Agent Has No Reason to Write Good Code

URL: https://sublimecoding.com/blog/why-ai-agents-write-unmaintainable-code
Published: 2026-08-01
Tags: AI, agents, engineering

>
**TL;DR:** The pitch of the moment is that if you
engineer enough harness — better orchestration, more review bots,
tighter sandboxes — you can delete code review and let agents run the
factory unattended. I don’t think that works, and I don’t think it’s a
skill issue. Coding models are trained against a reward that fires on
one question: did the tests go from red to green without breaking
anything else. Nothing in that signal grades design. A test verdict
costs seconds; bad architecture bills you in months, which means there’s
no fast oracle for maintainability, which means it can’t be rewarded
during training, which means it was never learned. Harness engineering
raises the floor. It does not move the ceiling, because the ceiling was
set during training.

## The pitch you’ve been hearing all year

The argument runs like this, and it’s seductive because most of it is
true.

You are the bottleneck. The models are good enough. Code is
effectively free now. So stop hand-crafting software and start running a
factory: specs go in one side, tested and deployed software comes out
the other, and the humans move up a level to writing specifications and
curating test scenarios.

The most committed version of this shipped in February 2026, when
StrongDM’s AI team published a charter for what they call a software
factory. Two rules, stated plainly: [code must
not be written by humans, and code must not be reviewed by humans](https://simonwillison.net/2026/Feb/7/software-factory/).
Agents write it, agents test it, agents ship it. The lights stay off
because nobody’s on the floor.

I want to be fair to that team, because they are not naive. They ran
directly into the obvious failure — that agents grading their own work
will cheat, hardcoding the value a test checks for — and they built
serious infrastructure to close it, including [behavioral
clones of every third-party service their software touches](https://simonwillison.net/2026/Feb/7/software-factory/) so the
tests run against something with teeth. That is not a blog post. That’s
real engineering.

But the pitch that’s circulating downstream of it has been flattened
into something much dumber: that the last human step, the one where
somebody reads the code, is a legacy ritual you can automate away with
enough linters and a prompt containing the word “adversarial.” And I’ve
watched enough agent output go by to be confident that’s wrong in a
specific, mechanical way that no amount of tooling fixes.

## The receipts are in, and they’re not good

We now have data instead of vibes, and the data is uncomfortable for
everyone selling velocity.

Faros AI published a study in April 2026 — [22,000
developers across more than 4,000 teams](https://www.faros.ai/blog/ai-acceleration-whiplash-takeaways), two years of telemetry
pulled from trackers, IDEs, static analysis, CI, version control, and
incident management. Output went up, exactly as promised: task
throughput per developer up 33.7%, epics completed per developer up 66%.
So did everything else:

- Bugs per developer up **54%**

- Incidents-to-PR ratio up **242.7%**

- Median time in review up **441.5%**

- Pull requests merged with **no review at all** up
**31.3%**

- Code churn up **861%**

Read those last two together, because that’s the whole story in two
lines. Review time exploded, so review got skipped. Not because anyone
decided to skip it — because a review queue that grows faster than the
humans reading it has exactly one release valve, and that valve is the
merge button.

This is correlation, and I’ll say so plainly. It doesn’t prove the
agents caused the incidents. But it matches what I see week to week, and
it matches the shape of the failures I’ve catalogued from my own
sessions in [the agent
bugs I catch every week](https://sublimecoding.com/blog/things-ai-coding-agents-get-wrong). Directionally, I believe it.

## “You’re holding it wrong” — you’re not

The standard response to all of this is that you’re doing it wrong.
Not enough context engineering. Not enough tokens. You’re still reading
the diffs like it’s 2023, grandpa. Let go.

I want to be careful here, because I’m not the guy who tried agents
for a weekend and wrote a thinkpiece. I’ve [committed 4,154 times in
six months](https://sublimecoding.com/blog/4154-commits-six-months-ai-agents) with agents doing nearly all the typing, on a product
that went zero-to-production in sixty days and has customers on it. I
like these tools. I’m not going back.

So this isn’t a warning from the sidelines. It’s the opposite: the
more time I spend inside the loop, the more convinced I am that the
thing people are trying to prompt their way around isn’t in the prompt
at all. It’s in the weights. And you cannot fix a training problem with
a harness.

## What actually gets graded

To see why, you have to look at what “make the model better at
coding” means concretely.

The loop is roughly: generate a bunch of agent traces attempting a
coding task, score each trace against a verifier, then update the
weights so high-scoring traces get more likely and low-scoring ones get
less likely. Repeat several million times. That’s reinforcement learning
with a verifiable reward, and it’s the machinery behind essentially
every serious coding model shipping today.

Everything hinges on the verifier. So what does it check?

Take the canonical setup, the one most of this work descends from —
[SWE-bench](https://www.swebench.com/) and the training
harnesses built on top of it. A task is scraped out of a real repository
— an actual bug, an actual fix. The model gets the repo checked out to
the commit right before the fix landed, plus the issue text. It writes a
patch. Then the grader throws away any edits the model made to test
files (yes, that’s necessary), applies the benchmark’s own test patch,
and runs the suite against two lists:

- **FAIL_TO_PASS** — the tests that were broken. Do they
pass now?

- **PASS_TO_PASS** — the tests that already worked. Are
they still fine?

Both green, reward is 1. Anything else, reward is 0. That’s it.
That’s the entire signal, and the research literature is direct about
the consequence: execution-based feedback [cannot distinguish between
trajectories that yield the same outcome](https://arxiv.org/pdf/2512.21919). Two patches, one clean and
one a disaster, both turning the suite green, are literally the same
number to the optimizer.

So ask what the model learns from millions of iterations of that.

It learns to make tests pass. It learns it extremely well — that’s
why these tools work at all. What it never learns, because it is never
once asked, is whether the codebase is better or worse for what it just
did. There is no penalty for a fix that works by wrapping the call site
in a try/rescue and swallowing the error. There’s no penalty for adding
the eleventh special case to a function that needed refactoring at the
third. There’s no penalty for solving the ticket in a way that makes the
next ticket harder.

**There is no penalty for eroding maintainability.** Not
a small one. Zero. The lived version of that, at full length and with the receipts, is [Peter Welch's essay refiled by someone who now has robots](https://sublimecoding.com/blog/programming-sucks-now-with-agents).

## The asymmetry that makes this hard to fix

The obvious response is: fine, add quality to the reward. People are
trying. It’s harder than it sounds, and the reason is structural, which
means it won’t age out with the next model release.

Running a test suite gives you a clean verdict in seconds. That speed
is the entire reason RL can afford millions of iterations — the oracle
is cheap, so you can consult it constantly.

Now price the oracle for architecture. The cost of a bad design
decision doesn’t materialize in seconds. It shows up the first time
somebody opens that file for a one-line change and discovers the one
line lives in eleven places. It shows up in the review that takes three
days because nobody can hold the blast radius in their head. Eventually
it shows up as the problem I laid out in [the onboarding playbook for
an agent-heavy codebase](https://sublimecoding.com/blog/onboarding-agent-heavy-codebase): a new person, or a new session, unable to
form an accurate model of how the system fits together.

That bill arrives in weeks, months, sometimes years. **Tests
answer in seconds; architecture answers in quarters.** You cannot
put a quarters-long feedback loop inside a training loop that needs to
run millions of times, and no clever engineering collapses that gap,
because the gap is what maintainability *means*. Maintainability
is defined by future change. It is unobservable in the present. That’s
not a tooling limitation — it’s the definition.

Which lands us somewhere uncomfortable: the single property that
determines whether your codebase survives contact with an agent for a
year is the one property the training process structurally cannot
grade.

## Why more review bots raise the floor and not the ceiling

Here’s where the harness argument comes back, and it deserves a real
answer.

More review agents do help. I run them. They catch the dumb stuff —
the unused import, the missing nil check, the obvious injection, the
test that asserts nothing. That’s genuine value and I’d keep them even
if the rest of this post is wrong.

But look at what a review agent *is*. It’s the same class of
model, running the same weights, asked to grade instead of generate. Its
ability to recognize bad design is bounded by whatever design sense
survived training — which is the exact thing we just established was
never rewarded.

And that leads to the trap sitting under every “just add an AI
reviewer” plan: **a model that could reliably tell good code from
bad would have written the good version in the first place.** The
generator and the judge are drinking from the same well. Stacking more
judges gets you more coverage of the failures the model already
recognizes. It does not get you a new capability neither model has.

So the floor comes up. The ceiling doesn’t move. And on a
long-running codebase, the ceiling is what you’re actually betting. There is a second way to reach that same ceiling, and it long predates the models: [the theory a program is built on never lived in the code](https://sublimecoding.com/blog/code-was-never-the-hard-part-theory-building), so an agent has no way to read it back out.

## The frontier is moving, slowly, and honestly

I want to be careful not to argue this is impossible. It isn’t. It’s
unsolved, and the hype is running well ahead of the discipline — those
are different claims.

The most serious attempt I’ve seen is Cognition’s [FrontierCode](https://cognition.com/blog/frontier-code), and
it’s interesting precisely because it refuses the pass/fail frame.
Instead of asking “did the tests go green,” it asks whether a maintainer
would actually *merge* the patch. Tasks were built with
open-source maintainers at 40-plus hours of input each, and the rubric
grades regression safety, scope, cleanliness, test correctness, and
maintainability as separate dimensions, each scored on its own, instead
of collapsing everything into one bit. They report an 81% lower
false-positive rate than the previous generation of benchmarks.

That is the right direction. It’s the first evaluation I’ve seen that
takes seriously the idea that a patch can be correct and still be
bad.

It’s also still a rubric applied by models and humans, with all the
cost and subjectivity that implies, and it doesn’t escape the well
problem above — it just puts more careful humans upstream of it. I’d
call it real progress and I would not yet bet a production codebase on
the models it produces. Both of those can be true.

The other honest position is: maybe a future model just gets this and
the whole problem dissolves. Maybe. If you want to hold your codebase’s
fate against that bet, that’s a choice you’re allowed to make. I’ve got
systems in production this quarter.

## What I actually do instead

None of this argues for using agents less. It argues for putting the
human effort in the one place the training process left empty. Four
things, all of which I run daily.

**Front-load the alignment, don’t back-load the
cleanup.** The oldest trick in the trade still works and works
better than ever: time spent agreeing on the approach before generation
is worth many multiples of time spent untangling afterward. The
generation step compressed hard; deciding *what to build and how it
should be shaped* didn’t compress at all, and that step is now the
highest-leverage hour in the week. A well-shaped change reviews fast. A
sprawling one doesn’t review at all — see the 441% number.

**Keep architecture human-owned, explicitly.** The agent
proposes the module boundary; a person decides it. This is the same
principle behind [never letting
the agent be the only thing between a change and production](https://sublimecoding.com/blog/amazon-let-the-ai-drive), applied
one level earlier — at the design decision instead of the deploy.
Structural decisions are precisely the ones with a feedback loop too
slow for the model to have ever learned from.

**Write the constraints down where the agent reads
them.** Most of what a senior engineer knows about a codebase is
unwritten, which means the agent doesn’t have it and can’t infer it. The
fix is a file it reads every session — [the CLAUDE.md entries that
earned their place after 50 commits](https://sublimecoding.com/blog/claude-md-after-50-commits): the boundaries, the patterns
this repo actually uses, the things that look reasonable and are
forbidden here. It doesn’t give the model design taste. It gives it
*this* codebase’s design decisions, already made, which is the
part it was never going to derive.

**Spend your reading budget asymmetrically.** You cannot
read everything at agent throughput, and pretending otherwise is how
teams end up merging without review. So don’t distribute attention
evenly — concentrate it where an error is expensive and irreversible,
and let the mechanical stuff ride on tests and bots. Which is also why
[TDD holds up unusually well with
agents](https://sublimecoding.com/blog/tdd-claude-code-elixir): a test the agent has to satisfy is a constraint that
survives the session, and it converts some of your review budget into
something that runs automatically forever.

## The step you can’t delete

The lights-off factory is a good story with one bad assumption inside
it: that code review is a bottleneck to be automated rather than the
only stage in the pipeline where anyone asks whether the system is
getting better or worse.

Every other step has a fast, cheap, automatable verdict. Does it
compile, do the tests pass, did it deploy, is it paging anyone. Review
is the odd one out precisely *because* it’s slow and human — it’s
where the quarters-long question gets asked in the present tense, by the
one participant who’ll still be maintaining this thing next year.

Delete it and you don’t get a faster factory. You get the Faros
numbers: more output, more incidents, more churn, and a codebase
drifting toward the state where the only remaining option is the
expensive one — throw it out and write it again by hand.

You can move fast. I do, and the receipts are public. You just can’t
skip the one step that nobody trained the model to care about.

## Read this next

-
[**The AI Coding
Agent Bugs I Catch Every Week**](https://sublimecoding.com/blog/things-ai-coding-agents-get-wrong) — The close-up: eight failure
patterns and the reflex for each.

-
[**When
to Trust an Agent and When to Step In**](https://sublimecoding.com/blog/when-to-trust-an-agent-and-when-to-step-in) — Where to spend the
review budget you actually have.

-
[**We’re About to Stop
Making Senior Engineers**](https://sublimecoding.com/blog/stop-making-senior-engineers) — Who’s left to own the
architecture if nobody reads the code.

---

## The Hard Part Was Never the Code. It Was the Theory.

URL: https://sublimecoding.com/blog/code-was-never-the-hard-part-theory-building
Published: 2026-08-11
Tags: AI, agents, engineering

>
**TL;DR:** Senko Rašić’s post calling “code was never
the hard part” an insult to programmers hit the front page and drew a
400-plus-comment argument, and both sides of that argument were right,
which is the tell that everyone was measuring the wrong thing. Peter
Naur settled it in 1985: programming is building a *theory* of
how some part of the world is handled by a program, and the source code
is a lossy artifact of that theory, not the theory itself. That reframe
explains why the phrase feels obviously true to some engineers and
obviously insulting to others — they hold different amounts of theory.
It also explains the thing 2026 actually broke. An agent produces text
without theory. Naur’s word for a program whose theory is gone is
*dead*, and the new failure mode is a program that was born that
way.

## A phrase that started a fight

You have heard some version of it, probably this month. *LLMs may
be good at coding, but code was never the hard part.* Sometimes it’s
dressed up — the hard part is figuring out what to build, the hard part
is the requirements, the hard part is the people.

Senko Rašić [pushed
back on it hard](https://blog.senko.net/code-was-never-the-hard-part-is-an-insult-to-all-programmers), and his framing is the reason the post traveled: he
read the phrase as a status claim rather than a technical one. “I
believe that’s a gross insult to all programmers everywhere.” His
arguments are the obvious ones and they land: if coding is easy, why
were programmers in high demand at high salaries for years? Why is so
much software so buggy? Why do *Clean Code* and SICP exist? And
the turn back on the other camp — if deciding what to build is the
genuinely hard part, why do so many product managers seem clueless? He
refuses the craft-versus-customer binary entirely: *¿Por qué no los
dos?* He ends where I’d end. Don’t outsource your understanding,
judgment, empathy, and taste to a model.

Then [the
Hacker News thread](https://news.ycombinator.com/item?id=49222189) did something more interesting than agree or
disagree. It split, cleanly, and both halves were credible.

One camp said the phrase is being read too literally. Nobody means
coding is *easy* — they mean coding was never the
*bottleneck*. The thing that killed their last three projects was
misalignment, a stakeholder who couldn’t prioritize, a requirement that
changed after the build. Enterprise work in particular is mostly
validation, SQL, and scheduling; the difficulty is pinning down rigid
requirements with the edge cases considered, not expressing them.

The other camp brought receipts in the opposite direction. Products
die of bad code all the time. Memory leaks that make a paid tool
unusable. Performance collapse that leaks users quietly for a year.
Feature velocity that decays until nothing ships. That camp read “code
was never the hard part” as a claim written by people who have never
shipped anything where the code could actually lose.

Both descriptions are accurate. Not “both have a point” in the
diplomatic sense — both are literally, verifiably true about the work
the person describing them does. So the disagreement isn’t about the
profession. It’s about something the phrase can’t see.

## Naur’s answer, from 1985

Peter Naur — the N in BNF — wrote a paper called [“Programming as
Theory Building”](https://gwern.net/doc/cs/algorithm/1985-naur.pdf) in *Microprocessing and Microprogramming* in
1985. Alistair Cockburn thought enough of it to reprint the whole thing
as an appendix in *Agile Software Development*. It is nine pages,
it is forty-one years old, and it is a better description of what just
happened to our jobs than anything published this year.

Naur’s claim is that we have the object of the work wrong. We think
we produce programs. He argues we produce *theories*:

>
programming properly should be regarded as an activity by which the
programmers form or achieve a certain kind of insight, a theory, of the
matters at hand

He is using “theory” in Gilbert Ryle’s sense — not a formal model,
but the thing a person has when they can do something *and*
explain, justify, and answer questions about the doing. You have a
theory of your commute. You have a theory of how to arrange furniture in
a room. It’s tacit, it’s held by a person, and it is not a document.

And then the line that does the damage:

The theory built by the programmers has primacy over such other
products as program texts, user documentation, and additional
documentation such as specifications.

Primacy. The code is downstream. Documentation, in Naur’s words, is
“an auxiliary, secondary product.” The real artifact is in someone’s
head, and everything on disk is a partial, lossy projection of it.

He gives three tests for whether someone actually holds the theory of
a program. They’re worth memorizing, because they’re the most useful
review checklist I’ve encountered and they predate every process
framework you’ve been made to sit through:

- They can **explain how the solution relates to the affairs of
the world** it helps handle — which real-world aspect each part
of the program text is matched to, and conversely, where in the text any
given real-world activity lives.

- They can **explain why each part of the program is what it
is** — support the code with a justification, which finally rests
on their own direct, intuitive judgment about what was relevant
here.

- They can **respond constructively to a demand for
modification** — take a change the world now requires and know
where it goes and what shape it should take.

Notice what fails all three. A person who can read the code
perfectly. A person with complete documentation. A person who wrote it
three years ago and has forgotten.

## Naur’s two case studies are about you

The paper’s evidence is two field reports, and if you’ve inherited a
codebase you already know how they end.

**Case 1.** Group A builds a compiler for language L.
Group B needs a compiler for L + M, a modest extension. Reasonable plan:
start from A’s compiler. B gets full documentation, annotated program
text, extensive written design discussion, *and* personal advice
from group A. Genuinely good conditions — better than yours.

It doesn’t work the way you’d expect. In several major cases, B
proposes changes that make no use of facilities already inherent in the
compiler’s structure — facilities that were discussed at length in the
documentation B had read. B’s instinct is to bolt on patches, and Naur
says those patches would have destroyed the design’s power and
simplicity. A’s members spot it instantly and propose a simple fix
framed entirely inside the existing structure. Naur:

>
This is an example of how the full program text and additional
documentation is insufficient in conveying to even the highly motivated
group B the deeper insight into the design, that theory which is
immediately present to the members of group A.

The postscript is worse. Ten years on, after passing through more
hands with no contact with group A, the compiler still had its original
powerful structure visible — “made entirely ineffective by amorphous
additions of many different kinds.”

**Case 2.** A 200,000-line real-time industrial
monitoring system, adapted per installation. The people who diagnose
faults on it were involved from design and have worked on it full time
for years. They diagnose almost exclusively from their ready knowledge
of the system plus the annotated program text — and here’s the part that
should stop you — they are *unable to conceive of any kind of
additional documentation that would be useful to them.* Meanwhile
the outside groups who have full documentation and full guidance keep
filing problems that trace back to inadequate understanding, which the
insiders clear up in a conversation.

Not “the docs were bad.” The insiders couldn’t imagine a document
that would help, because what they have isn’t the kind of thing a
document holds.

## This is what the Hacker News thread was actually arguing about

Now the split makes sense, and it isn’t a disagreement about
programming. It’s two groups honestly reporting different theory-to-text
ratios.

If you build forms over a database — most enterprise work, and I’m
not being snide, I’ve done years of it — the theory is genuinely thin.
The requirements *are* most of the theory. Once you know the
rules, the mapping to code is close to mechanical, and any competent
person can pick it up from the text. For that work, “code was never the
hard part” is just true. Getting eleven stakeholders to agree what the
rule is was the hard part, and it took four months.

If you build something with real invariants — a scheduler, a
distributed state machine, a query planner, anything with a performance
budget or a consistency guarantee — the theory is thick, most of it
never got written down anywhere, and the code is its only durable trace.
Tell that person coding is the easy part and you’ve told them the part
of their job that took a decade to develop doesn’t exist. That’s the
insult Senko named, and he’s right to name it.

“Code was never the hard part” is not a claim about the profession.
It’s an accurate report of how much theory your own work carries.

Which is why the argument won’t resolve. Both sides are describing
their jobs correctly and hearing a claim about everyone’s.

## The part Senko didn’t take, and 1985 did

Here’s where this stops being a semantics fight and starts costing
money.

Senko’s conclusion is don’t outsource your judgment to AI. Correct,
and too general to act on. Naur gives you the mechanism, in a sentence
he wrote when the state of the art was punched-card-adjacent:

>
if viewed merely as a change of the program text and of the external
behaviour of the execution, a given desired modification may usually be
realized in many different ways, all correct. At the same time, if
viewed in relation to the theory of the program these ways may look very
different, some of them perhaps conforming to that theory or extending
it in a natural way, while others will be wholly foreign

Read that again with an agent in mind, because it is the cleanest
description of the actual problem I have found anywhere.

A coding agent samples from *all correct*. That set is large.
Every member of it passes your tests, satisfies the ticket, and survives
review-for-style. Only a fraction of that set conforms to the theory of
your program, and nothing in the model’s inputs distinguishes them,
because the distinguishing information was never in the repository. It
was in group A’s heads.

This is the same mechanism I’ve written about from the training side
— [your agent has
no reason to write good code](https://sublimecoding.com/blog/why-ai-agents-write-unmaintainable-code) because maintainability was never in
its reward signal. Naur gets you there from the other direction, and I
find his version more useful: it isn’t that the model is careless. It’s
that the criterion is genuinely absent from the artifact. He says so
explicitly — the judgment “cannot be reduced to any limited set of
criteria or rules.” You cannot prompt your way to it, because if it
could be written as a rule, it would already be in the docs, and case 1
says the docs weren’t enough.

And Naur names the failure mode this produces. He calls it decay:
“modifications made by programmers without a proper grasp of the
underlying theory.” That’s group B’s amorphous additions. He wrote
*programmers*, and I’m not going to pretend he was predicting
anything — but swap that one word and the sentence describes a merged
agent diff on a system nobody currently understands, which is the same
failure arriving by a faster route.

## Born dead

Naur’s conclusion is the one people quote, and the reason it stings
is that he means it literally:

The death of a program happens when the programmer team possessing
its theory is dissolved.

A dead program keeps running. It keeps producing useful results, and
it keeps making money — death is not an outage. Death becomes visible
only when a demand for modification arrives and cannot be answered
intelligently. Everyone reading this has worked on a dead program. It’s
the one where a two-line change takes three weeks and the estimate is a
coin flip.

Revival, Naur says, is not reading the code and not restoring the
docs. It’s a new team rebuilding the theory. He doesn’t put a price on
that, and I won’t invent one — but nothing in his account suggests it’s
cheaper than the first build, and case 1 is a group with documentation,
source, and the original authors on call still failing to get there.

Here’s the 2026 version, and it’s new. Naur assumed every program was
alive once — a team held the theory, then dispersed. An agent-built
system can arrive without ever having been alive. Nobody dissolved.
Nobody left. The theory was never constructed, because construction is
what happens in a person while they work the problem, and the working
was delegated.

That inverts the economics everyone is currently quoting at you.
Cheap to produce, expensive from then on, permanently — because you
skipped the step that makes future changes cheap, and the bill starts
arriving at change #2. It’s why [taking over a vibe-coded
app](https://sublimecoding.com/blog/inheriting-a-vibe-coded-app) feels different from taking over a bad human codebase. A bad
human codebase has a theory you can partially recover by archaeology,
since [every insane
codebase was a sane response to constraints you can’t see](https://sublimecoding.com/blog/insane-codebases-were-once-rational). The other
kind has no theory to recover. There’s nothing behind the text.

## What this changes on Monday

I use agents for most of my working day and none of this is an
argument against them. It’s an argument about which step you are not
allowed to skip.

**Review for *why*, not for style.** Style review
has lost most of its value in my own work — the output comes back tidier
than what I’d have typed by hand, and editing it is cheap either way.
Naur’s test 2 is the question that still has teeth: *why is this part
what it is?* Ask it of the diff. If the answer is “the agent chose
it” or “it passes,” you’ve reviewed the text and not the change. This is
the same bar as verification, one level up: verification asks whether it
does what we said; theory asks whether it does it in a way this system
can survive.

**Use the modification test as your real readiness
gate.** Naur’s test 3 is a better signal than any interview or
checklist. Hand someone a change request against a surface they claim to
own, and watch whether they respond constructively or start
pattern-matching. That’s the difference between someone who read the
code and someone who holds the program.

**Keep one accountable human per surface, and mean it.**
Not a name in a CODEOWNERS file. A person who can pass all three of
Naur’s tests on that surface, who is given enough time on it to have
built the theory, and who reviews what the agents do there. This is the
real constraint on how many agents one person can supervise — not
tokens, not review throughput, but how many programs one human can hold
a theory of. It’s fewer than you want it to be.

**Record constraints, not decisions.** Documentation is
a secondary product and it will never carry the theory, but it can carry
the *inputs* — which is what group B was missing. “Single service
because two people and a March deadline.” “This queue is FIFO because
the client’s reconciliation assumes it.” Constraints are the recoverable
part.

**Stop treating prompt transcripts as documentation.** A
session log is a secondary product of a secondary product. It records
what was asked and what was emitted. It does not record why this shape
and not the other four correct ones, because in most cases nobody
decided that — the sampler did.

**Let juniors build theory on purpose.** The pile of
unglamorous work was never busywork; it was the mechanism by which
theory got built, which is [why removing it quietly stops
producing senior engineers](https://sublimecoding.com/blog/stop-making-senior-engineers). Onboarding is the same problem in
miniature, which is why [onboarding into an
agent-heavy codebase](https://sublimecoding.com/blog/onboarding-agent-heavy-codebase) needs a different shape than the one you used
in 2019.

And the honest limit, because I’d be selling you something otherwise:
you cannot fully write a theory down. That’s Naur’s entire point and no
practice above defeats it. Every one of these is damage control on an
unavoidable loss. The only real defense is continuity of people who have
spent real time on the real problem — which makes headcount decisions,
retention, and how much you delegate to agents the same decision,
whether or not anyone frames it that way in the meeting.

## Where I land

Senko is right that the phrase is an insult, and I think he’s right
for a reason he doesn’t quite state. It isn’t insulting because code is
difficult. It’s insulting because it mistakes the artifact for the work.
Typing was never the hard part — no serious person ever claimed it was.
Building the theory was, it still is, and it is the one part of this job
that has not gotten one bit cheaper since 1985.

What changed is that we can now buy the artifact without doing the
work. That’s a real capability and I use it daily. But you’re buying the
projection, not the thing it was projected from, and the difference
doesn’t show up in the demo. It shows up at change #2, when someone asks
why it’s built this way and every person in the room, including the one
who shipped it, has to guess.

If you’ve got a system nobody can confidently answer questions about
anymore — inherited, agent-built, or just old — that’s a theory problem
before it’s a refactoring problem, and the sequencing matters. [Happy to talk it through](https://sublimecoding.com/consulting).

---

## Your Junior Dev Leans on the Agent. Good.

URL: https://sublimecoding.com/blog/junior-dev-leaning-on-the-agent
Published: 2026-08-12
Tags: AI, agents, engineering leadership, engineering

>
**TL;DR:** A first-time lead asked me what to do about
the developer on his team — still in college — who builds almost
entirely with an agent. The code works, but it’s rigid, unabstracted,
and hard to read, and he’s worried the junior isn’t learning. The
instinct is to make him use AI less. That’s advice with an expiry date,
and it doesn’t fix the actual problem. The readability bar existed
because editing code was expensive; it isn’t anymore. The bar that still
holds is verification — does this do what we said it does, and does it
survive being wrong. Aim him there. Then be honest about what he
genuinely loses, because something real is lost and pretending otherwise
doesn’t help him.

## The question

Same new grad I wrote about in [Every Insane Codebase
Was Once Rational](https://sublimecoding.com/blog/insane-codebases-were-once-rational) — a year into the industry, running a project and,
for the first time, another person. His second question was about that
person:

One of his developers is still in college and leans heavily on AI for
his work. It works. But the lead is seeing a lack of flexibility when
functions need to change, a lack of abstraction, and code that’s hard to
read. He’s been trying to curb it and it isn’t landing. He’s worried the
junior isn’t learning what he needs to — bad for the junior’s future,
bad for the team, because the lead wants to hand him larger components
without worrying about quality.

Every part of that observation is correct. The diagnosis that follows
from it is the part I’d push back on.

## The advice with an expiry date

The instinct is to tell him to use AI less and write more of it
himself. I understand the instinct and I think it’s a losing position,
for a reason that has nothing to do with whether agents are good.

It’s how he’s going to build. It’s how you’re going to build. It’s
how I build — I’ve barely written a line of code by hand in the last six
months, and I’ve been doing this for thirteen years. That’s not a
prediction about 2030, it’s a description of a workflow that already
shipped [4,154 commits
in six months](https://sublimecoding.com/blog/4154-commits-six-months-ai-agents).

You haven’t hand-written an HTTP request in years either. Almost
nobody does, outside of debugging or implementing the protocol itself,
which is a different job. The abstraction moved and the skill under it
stopped being load-bearing. That’s happened repeatedly — assembly,
memory management, DOM manipulation — and the engineers who dug in each
time didn’t preserve the craft, they just arrived late.

So “use it less” spends your authority on a rule that reality is
going to overturn, in front of a junior who can see it being overturned.
You get one or two of those before your feedback stops carrying weight.
Spend it on something that survives.

## The readability bar was a proxy for edit cost

Here’s the part worth thinking through, because it’s the actual shift
and it’s easy to mistake for lowering standards.

Why did we demand readable, well-abstracted code? Not for aesthetics.
Because **changing code was expensive and the expense was human
attention.** Every future modification required a person to load
the whole thing into their head, find the right place, and change it
without breaking three other things. Readability was the tax we paid up
front to make that cheaper later. Abstraction was insurance against
having to make the same edit in nine places by hand.

The unit cost of that edit collapsed. Need to change the function?
The agent changes it. Needs a refactor to accommodate the change? It
does the refactor, faster than the junior would have, and faster than
you would have. The rigid unabstracted code the lead is worried about is
genuinely worse to hand-edit — and hand-editing is the thing that
stopped happening.

That doesn’t mean structure is worthless. It means **structure
is no longer where the risk lives**, and continuing to spend your
review attention there is spending it on the cheap problem. The
expensive problem moved.

## Where the risk actually went

If you’re not grading style, what are you grading? Behavior.
Specifically: does this do what we claimed, does it still do it after
the next change, and what happens when it’s handed something we didn’t
anticipate.

That’s where quality lives now, and it’s not a soft standard — it’s a
harder one, because it’s falsifiable in a way “is this readable?” never
was.

Concretely, what I’d ask of a junior building with an agent:

**Tests the agent didn’t write from the same context as the
code.** An agent that writes the implementation and the test in
one pass will happily write a test that passes against its own
misunderstanding. Make the test come from the requirement, not from the
code. Different session, or written first — [TDD holds up unusually well with
agents](https://sublimecoding.com/blog/tdd-claude-code-elixir) for exactly this reason.

**Mutate the test and confirm it fails.** This is the
cheapest quality check in existence and almost nobody runs it. Break the
implementation on purpose. If the suite stays green, the test is
decoration. A junior who does this once will never fully trust a green
checkmark again, and that skepticism is worth more than any style rule
you could teach him.

**Test the boundaries, not the happy path.** Empty
input, huge input, wrong type, the network call that hangs instead of
failing, the duplicate submit. The agent’s default is the path it just
imagined; the failures are always in the paths it didn’t.

**Non-deterministic output needs its own approach.** If
any part of the product calls a model, normal assertions don’t hold — [testing agent outputs is
its own discipline](https://sublimecoding.com/blog/testing-ai-agent-outputs-exunit) and it’s a good, concrete thing to hand a junior
who wants harder problems.

**Resilience over elegance.** What happens when the
dependency is down, the input is malformed, the job runs twice? Those
questions have real answers and reviewing them teaches systems thinking,
which is the thing you actually want him to have.

Note what this does to the mentoring problem. “Make your code more
readable” is taste, it’s arguable, and a junior can’t tell whether
you’re right or just particular. “Break the implementation and show me
the test catches it” is a task with a binary outcome. He can run it
himself, and it’s either green or it isn’t. That’s a standard that
teaches without requiring him to defer to you.

## The honest tension

I’ve argued before that [coding agents have
no structural reason to write maintainable code](https://sublimecoding.com/blog/why-ai-agents-write-unmaintainable-code) — the training
signal fires on tests going green, nothing in it grades design, and no
harness moves that ceiling. That post’s prescription was to spend your
reading budget asymmetrically and keep architecture human-owned. I still
hold that. This post adds the other half, which is where the budget
comes *from*: the attention you’re currently spending on
readability is the attention that should be spending on verification.
Not “review less.” Review the other thing.

And I have to argue against my own numbers here, because that same
post cites the Faros AI study of 22,000 developers across 4,000+ teams —
incidents-to-PR up 242.7%, code churn up 861%, median review time up
441.5%. That is not a picture of quality problems that stopped billing
anyone. It’s a picture of them billing harder.

But look at what’s actually in that bill. Churn is code rewritten
shortly after it’s merged — the cost of the *edit*, which is
exactly the cost that collapsed. Incidents are behavior wrong in
production. Review time is humans reading. Two of those three are the
thing I’m telling you to aim at, and the third is the queue that broke
because review time exploded until people started skipping it. None of
those three lines is “the abstraction was inelegant.” The bill went up
on verification and human attention, which is the argument for moving
your bar there, not for holding the old one harder.

The caveat that survives all of it: the ceiling still bites where
structure and correctness are the same thing. Data model, module
boundaries, auth flow, anything with a public interface. There, review
the design — a bad call there isn’t a style problem, it’s a one-way
door.

So the rule isn’t “never review structure.” It’s: review structure
where structure is load-bearing, and review behavior everywhere else.
Most code is everywhere else.

## What compounds: context, not keystrokes

The biggest single factor I’ve seen in whether AI-heavy development
gets better or worse over time isn’t the model. It’s whether the team is
accumulating anything.

At EnergyConnect we maintain what amounts to a second brain for the
agents — thousands of lines of markdown the agents read and write. Not
documentation for humans that agents happen to parse. Working knowledge:
a file describing how our bidding process runs, a file on how the
carrier-supplier relationship works, and a log. When an agent gets
something wrong because it didn’t know two of those things were related,
we correct it and write the correction down as an entry. Next time, it
looks up “carrier,” finds the file, loads that context on demand, and
doesn’t make the mistake. The learnings compound instead of evaporating
at the end of every session.

That’s the difference between a team that gets faster with agents and
one that just generates more code. I’ve written up [how to build the wiki your
agent actually reads](https://sublimecoding.com/blog/build-an-llm-wiki-for-your-agent) and [what belongs in CLAUDE.md after
fifty commits with it](https://sublimecoding.com/blog/claude-md-after-50-commits), so I won’t repeat the mechanics here.

For a lead with one junior, this is the highest-leverage thing you
can hand him, and it’s the answer to “he isn’t learning”: have him own
the knowledge base. Every time the agent gets something wrong about your
domain, he writes the entry that prevents it. To write that entry he has
to actually understand the domain — which is the learning you were
worried he was skipping, arriving through a door he’ll walk through
voluntarily because it makes his own work easier.

## The part where he’s right, and what it costs

I don’t want to wave away the worry, because underneath it is
something real.

The pile of unglamorous work that used to build judgment is exactly
the pile agents are best at, and [that’s a problem I’ve written
about at length](https://sublimecoding.com/blog/stop-making-senior-engineers). It hits juniors first and hardest.

The mitigation isn’t taking the tool away. It’s making him
responsible for the judgment the tool can’t hold:

- **He owns the domain knowledge base.** Every agent
mistake about your business becomes an entry he researches and writes.
He can’t write it without understanding the domain, and he’ll do it
willingly because it makes his own next prompt work better.

- He owns the test strategy, so he has to know what could break.

- He owns the incident when his component fails, so he learns what
production does to assumptions.

None of that requires him to type the implementation. All of it
requires him to understand the system, which is what “learning” actually
meant the whole time.

The typing was never the point. It was just the only delivery
mechanism we had.

## The standard you’re enforcing might not be yours

One more thing he raised, and it’s the same problem wearing a
different hat: he’s finding it hard to hold a standard he isn’t fully
sure of himself, worried about whether he’s doing it the way the
industry does it, while both he and his dev are new to this.

Your job as the lead is to fit the standard to *your* team,
not to import it. A practice can be a real industry standard and still
be wrong here — if it costs you more than it protects you, it’s a cost,
and you’re allowed to change it. Same instinct as with the code: fix the
pain point, and when something breaks, fix that. For a two-person team,
less process is usually the right first guess. And a fair amount of the
job is just listening — if you’ve hired well, the person doing the work
will tell you what they need and how they want to build; you mostly have
to ask and then actually hear it.

Read [*Peopleware*](https://www.oreilly.com/library/view/peopleware-productive-projects/9780133440706/).
First edition 1987. The third, in 2013, added six chapters on
leadership, meetings, and distributed teams, and otherwise left the
thing alone — and there hasn’t been a fourth in the thirteen years
since. You’re managing people. That part hasn’t moved nearly as much as
the tooling has.

## What to do this week

- **Stop asking for less AI. Ask for a broken test.**
Have him break the implementation and prove the suite catches it. One
exercise, immediate result, teaches more about quality than a month of
style review.

- **Move your review attention.** Structure gets reviewed
where it’s load-bearing — data model, boundaries, auth, public
interfaces. Everything else gets reviewed on behavior.

- **Separate the test’s context from the code’s.**
Requirement first, or a different session. Same-pass tests validate the
misunderstanding along with the code.

- **Hand him the knowledge base.** Every agent mistake
about your domain becomes an entry he writes. That’s how he learns the
domain and how the team stops repeating errors.

- **Cut one process you imported because it was
standard.** If it’s costing more than it protects on a two-person
team, it wasn’t a standard, it was a costume.

If you’re leading a small team through this shift and trying to
figure out what to actually hold people to now, [that’s a conversation I have often](https://sublimecoding.com/consulting).

---

## The 'Smallest Possible Slice' Heuristic for Shipping Complex Features

URL: https://sublimecoding.com/blog/smallest-possible-slice-shipping-complex-features
Published: 2025-11-10
Tags: engineering, productivity

**Most "we delivered late" stories trace to one decision: the team scoped the first slice too big.**

The team breaks down the feature into "frontend, backend, database changes, the new microservice." They estimate two weeks. Six weeks later, the database changes are merged but the backend isn't, the frontend is half-built, the microservice is on a feature branch that's now 800 commits behind main, and nobody can demo anything because nothing actually works end to end yet.

This pattern is so common it's worth giving it a name. Most teams break work down by *layer* when they should break it down by *slice*. The corrective: a heuristic I've leaned on for the better part of a decade — the smallest possible slice that touches every layer.

## Why "MVP" is too vague to fix this

"Just ship the MVP" is good advice in spirit and useless in practice. MVP gets defined by stakeholder negotiation: marketing wants this, product wants that, engineering says we can have either of two things by date X. The result is a feature that's sized to "what fits in a sprint" rather than to "what actually works end to end."

The smallest-possible-slice heuristic is more specific. The first version of any non-trivial feature should be defined by these three constraints, in order:

- It touches every layer the final feature will touch — frontend, backend, persistence, infrastructure.

- It works end to end for one trivially specific case.

- It can be deployed to production by Friday.

Notice what's not on the list: completeness, polish, edge cases, performance. Those come later. The first slice exists to prove the wiring is correct.

## The vertical-cut rule

Most teams default to *horizontal* decomposition. Sprint 1: schema changes and migrations. Sprint 2: backend API. Sprint 3: frontend. Sprint 4: integration. Each sprint produces something, but nothing is testable in production until sprint 4.

The vertical cut runs the other way. Sprint 1: the simplest possible end-to-end version. The schema has one table with three columns. The backend has one endpoint that returns a hard-coded response if the input matches. The frontend has one input and one output. It works for exactly one case. Ship it.

The reason the vertical cut wins: every sprint after the first is now *additive*. You're adding cases, polishing UI, expanding scope — but the core wiring is already proven. When the inevitable scope cut comes (and it always comes), you have shipping software to ship instead of four feature branches to merge.

## The deploy-by-Friday filter

The cleanest forcing function for the smallest-possible-slice mindset is a question I ask the team every time we scope a new feature: *can we get the first version into production by Friday?*

If the answer is "yes, but only the schema changes," the slice is too horizontal. Throw it out. Re-scope.

If the answer is "yes, with one customer behind a feature flag, working for the simplest case," that's the right slice. Ship that, see what breaks in production, then iterate.

If the answer is "no, even the first slice will take three weeks," the work is genuinely large and you need to break it down into smaller features, not smaller slices of one feature. Bigger problem, different conversation.

The reason the Friday filter works is that it forces the team to find the small case. Most engineers are uncomfortable shipping something obviously incomplete. The Friday deadline overrides that instinct just enough to get the first slice out the door, and once it's out, the team's attitude shifts from "what should we build" to "what should we add next."

## A concrete example

At Lavender, we shipped a new AI feature that recommended improvements to user-written sales emails. The first scoping pass from the team came back as a four-week project: prompt engineering work, a new evaluation pipeline, a UI component for inline suggestions, an analytics dashboard, an A/B testing harness.

I rejected it. Re-scoped to: one user, one email, one suggestion type, one model call, one button. The button shows up on a hardcoded email body for one specific user account. Click it, get a suggestion, render it. No analytics, no A/B, no eval pipeline.

That version shipped on the fourth day. It was visibly thin. It also *worked*, which we hadn't yet proven the original four-week design would. Over the next two weeks we extended it to all users, added the suggestion-type variations, layered in the eval pipeline, and turned on the A/B harness. The full feature was in production at three weeks instead of four, and we caught two architectural issues during the first week — issues that would have been brutal to fix at the four-week mark with everything already integrated.

Total elapsed: roughly the same. Total risk: dramatically lower. The "smaller" first slice was paradoxically the faster path.

## When the heuristic breaks

The smallest-possible-slice rule fails for one specific category of work: *research-shaped problems*. Things where you don't yet know what the right answer is, only that there's a question.

Examples: training a custom model from scratch, designing a novel cryptographic protocol, exploring whether a certain optimization is even possible. You can't ship a thin vertical slice of a question. You have to do the research first.

The way I handle these: time-box the research as a separate phase, with a defined exit condition ("by week three we will have a written go/no-go decision on training a custom embedding model"). Once the research phase exits, the implementation phase reverts to vertical slicing.

The mistake to avoid: dressing up the research phase as a feature build. If you're scoping "build an MVP of the new model" when the real question is "is custom training viable for our use case at all," the team is going to drift. Be honest about which kind of work you're doing.

## The tactical checklist

When a team comes to me with a feature breakdown, I run it through five questions:

- Does the first slice touch every layer the final feature will touch?

- Can it be deployed to production by Friday?

- Is there at least one user (or test account) who will see something different on Monday?

- If we shipped only this slice and nothing else, would it be embarrassing but functional, or non-functional?

- What's the explicit list of what's *not* in the first slice?

The fifth question is the one most teams skip and the one that prevents scope creep most reliably. Writing down what's not in the first slice locks the team into the discipline of shipping something thin. Without it, "just one more thing" creep extends the slice by 50% before code is written.

## The takeaway

The most reliable way to ship complex features fast is to ship a thin one first and grow it. Most teams know this in principle. Most teams violate it in practice because the first thin slice always feels embarrassingly incomplete.

The discipline is sitting with the embarrassment. Ship the thin slice. Watch it work in production. Then add the next slice. Repeat for as long as the work is generative.

The teams that internalize this ship faster than the teams that try to scope the whole feature up front, every time. The teams that try to predict everything at sprint planning and then deliver in one big bang ship slower and ship buggier. The slice heuristic is what separates these two patterns. Use it.

## Convincing a skeptical team

The slice heuristic is intellectually obvious and culturally hard. Engineers who've spent years on teams that scope full features upfront will resist shipping the embarrassingly thin first version. The objections are predictable.

"*It's not ready for users.*" Right — that's why it's behind a feature flag with one allowlisted account. Nobody will see it except the team.

"*We're going to have to throw away this code when we build the real version.*" Maybe. But the discarded code is rarely the expensive part. The expensive part is the architectural learning, and that's preserved regardless of whether you keep the code.

"*The product team will think we're shipping garbage.*" Get product in the room when you scope the slice. Show them that the next slice ships next week. The thin version stops being garbage when it's framed as a milestone, not a shipped feature.

The way I get past this with new teams is to show, not tell. Pick a feature, scope it the team's preferred way, then re-scope it as a thin slice. Walk through both timelines on a whiteboard. The thin slice almost always wins on calendar time, even when the team initially said it would take longer. After one or two demos of this, the team converts itself.

## The retro question that locks the habit in

One question, asked in every retrospective for a quarter: *could we have shipped a smaller first slice of the work we did this sprint?*

The answer is yes more often than the team initially thinks. Asking the question consistently trains the muscle. After a quarter of asking, the team starts asking it during sprint planning instead of in retro — and that's when the heuristic has actually been internalized.

One additional pattern: the slice heuristic compounds with continuous deployment. If your team can ship to production multiple times a day, slicing becomes the default mode of operation rather than a discipline you have to remember. If your deploys are weekly or batched, slicing requires constant re-justification because each "small slice" feels like a wasted deploy slot. The teams that internalize the heuristic fastest are usually the teams that already have the deploy infrastructure to support it. If your team doesn't, fix that first — the slice discipline rests on it.

## Read this next

- [**AI-Assisted Engineering Isn't Faster Coding**](https://sublimecoding.com/blog/ai-assisted-engineering-is-a-new-workflow) — How modern engineering tooling shortens the slice cycle even further.

- [**How I Triage a New Codebase in 90 Minutes**](https://sublimecoding.com/blog/triage-a-new-codebase-90-minutes) — The pragmatic discipline applied to onboarding instead of feature work.

- [**From One Engineer to Fifteen**](https://sublimecoding.com/blog/from-one-engineer-to-fifteen-engineering-leadership) — Why slice discipline is partly a leadership problem, not just an engineering one.

---

## How to Manage a 4-Person Engineering Team Without Becoming a Manager

URL: https://sublimecoding.com/blog/managing-a-four-person-engineering-team
Published: 2025-10-27
Tags: engineering leadership, founders

**A 4-person engineering team is the most overlooked unit of management in startups.**

Big enough that the tech lead can't write all the production code. Small enough that hiring an EM kills velocity and adds a layer of communication overhead the team can feel within a week. Most startup engineering org charts skip from "founding engineer" straight to "Director of Engineering at twenty people" — and pretend the territory in between doesn't have its own playbook.

It does. I've run 3–5 person engineering teams at PopSocial, at InsideTrack during a phase transition, and most recently across two fractional engagements with AI-native startups. The patterns hold. Five rituals that work at this size, three traps to avoid, and the signal that tells you it's time to evolve.

## Why 4 is the awkward number

One engineer is a co-founder. Two is a duo. Three is a tight team where everyone communicates by osmosis. By four, osmosis breaks. Engineer A doesn't know what engineer C is working on, two of them ship overlapping changes that have to be reconciled in code review, and the lead starts feeling like they're spending half the day on coordination that didn't exist last quarter.

This is the moment most founders panic and either hire an EM or default to the worst option: announce themselves as the EM and stop coding. Both miss the point. At four engineers, you don't need a manager. You need *cadence*. Four people running on shared rhythms perform like five-and-a-half. Four people without rhythm perform like three.

## The five rituals that work at this size

### 1. Weekly 1-on-1s, 30 minutes

The single highest-leverage 30 minutes on your calendar. Three sections, in this order: what's blocking you, what's bothering you, what are you working on. The first two are non-negotiable; the third is often obvious from context and can be skipped.

Why this works at 4 engineers specifically: the alternative is finding out your strongest engineer is unhappy via their resignation email. At 20 engineers, you need a layer of management to surface this. At 4, the layer is you, weekly, in 30 minutes.

### 2. Async daily check-in, in Slack, text only

Three lines per engineer per day, in a single thread. What I shipped yesterday. What I'm shipping today. What's blocking me. No video. No meeting. Twenty seconds to write, two minutes for the team to read.

The crucial constraint: *text only*. Standups by video at small scale are a tax. The same information conveyed in text takes a tenth of the time and creates a written record you can search later. The team I most recently ran did standups this way for a year and never once felt the lack of synchronous time.

### 3. Friday demos, 15 minutes

Once a week, on Friday, the team gathers for fifteen minutes. Each engineer shows the most interesting thing they shipped that week. Could be a feature, a refactor, a bug fix with a great post-mortem, anything they're proud of.

The point is not status reporting. The point is *visibility* — engineers seeing each other's work, picking up patterns, noticing where someone has built something useful that another engineer didn't know existed. At 4 engineers, this is also the highest-bandwidth moment of pure team identity in the week.

### 4. Quarterly written goal docs, one per engineer

Twice a year is too infrequent at startup pace; weekly is theatre. Quarterly hits the right rhythm. One page per engineer: three goals, written by them, reviewed with you, signed by both at the end of the meeting.

Three goals, not five. They span the three buckets of work: ship something hard, learn something specific, contribute to the team in a defined way. At the end of the quarter you sit back down, look at the doc, and have a real conversation about what happened. This is how you compound performance at small scale without bureaucracy.

### 5. Monthly retros: what's broken

One hour, last Friday of the month, no calendar invite needed beyond the recurring slot. Single question: *what's broken about how we work?* Not what's broken in the code. What's broken in the process, the tools, the meetings, the deploys, the comms.

You take notes. The next week, you fix the one thing the team most wanted fixed. The team feels heard, the process actually improves, and the next month's retro builds on a smaller list of complaints. After six months of this, the team's operating practices are visibly better than every other 4-person team you'll ever see.

## Three traps to avoid

**Trap 1: Hiring an engineering manager at four engineers.** The marginal cost is enormous and the marginal value is small. The EM at this scale will inevitably either become a tech lead in disguise (in which case you should have just promoted internally) or spend their day creating process to justify their own existence. Wait until six engineers. Possibly eight.

**Trap 2: The lead going full-time managing.** This is the most common pre-Series-A founder mistake. You stop shipping code, the team's velocity drops 30% within a month because the senior IC just left the keyboard, and you spend the dropped time on meetings the team didn't want anyway. The right load at this stage is 70/30 IC/management. Below 50/50 management, you're failing the team.

**Trap 3: Rituals that drift into status meetings.** Every ritual on the list above can collapse into "tell me what you're working on so I know" if you're not careful. The signal a ritual has drifted: the engineers stop volunteering things, you start asking direct questions, and the meeting feels heavy. When that happens, kill the meeting that week. Bring it back next week with explicit re-framing.

## The transition signal at six engineers

The model in this post breaks somewhere between five and seven engineers. The exact number depends on team-shape and product-shape but the symptoms are consistent: 1-on-1s start eating your whole Tuesday, the async standup thread is too long to read in two minutes, and the Friday demo runs over because four-out-of-six demos already feels rushed.

The transition is to a sub-team structure: two ICs, two ICs, with a tech lead per group. You're now managing the leads, not the ICs. Different rituals, different cadence, different challenges — and a whole different post.

The good news: getting the 4-person ritual stack right is the foundation everything that follows builds on. The teams that hit Series A with healthy engineering culture are the teams that ran disciplined rituals at four. The ones who skipped the discipline at small scale are the ones spending the year after Series A re-installing it under pressure.

## Hiring at the 4-person team size

One topic the rituals don't cover: every hire at this size is a culture-level decision, not just a skill match. At twenty engineers a single wrong hire is a 5% problem. At four, the wrong hire is a 25% problem and you'll feel the consequences inside six weeks.

The implication: the bar for hires 3, 4, and 5 should be unreasonably high. Not "great engineer." Not "really senior." It needs to be: "this person makes the team meaningfully better the day they start, in ways the existing engineers couldn't have produced themselves." Anything less and you're hiring someone who needs to be brought up to the existing team's level, which the existing team will resent at this scale.

Concretely, what's worked for me at this size:

- **Every existing engineer is a hard veto.** No exceptions. If any of the three engineers on the team has a strong reservation about a candidate, the candidate doesn't get hired, regardless of what the founder thinks. This kills political pressure and keeps the team's culture in their own hands.

- **Take-home → trial week → offer.** A 3-day paid trial week between final interview and offer is the most predictive single signal at this scale. You're not hiring for resume; you're hiring for whether they fit the team's flow.

- **The fifth hire is the one to slow down on.** Going from four to five is when team dynamics noticeably shift. Most founders rush this hire. The team performs better at four for an extra month than at five with the wrong addition.

## Remote vs in-person at this size

Brief note because the rituals work differently in each setup. Remote 4-person teams need *more* structure, not less — the async daily check-in becomes load-bearing instead of nice-to-have, and the Friday demo is the single thing that holds team cohesion together. Skip the demo for two weeks in a row and you'll feel the drift.

In-person 4-person teams can run looser. The async standup can be optional because the team is already overhearing each other's progress. The Friday demo is still worth doing but it has less work to carry.

Hybrid is the worst of both worlds at this size. If you can choose one or the other, choose. If you can't, default to remote-first rituals — they degrade gracefully when half the team is in the room and the rest are remote. In-person-first rituals don't.

One last note on cadence. Founders sometimes ask whether all five rituals run from week one with a new hire. The answer is yes — the new engineer joins the existing rhythm rather than the team adapting to them. New hires actually onboard *faster* when they're plugged into a working ritual stack on day one, because the rituals make the team's expectations legible. Skip the rituals during onboarding and the new hire spends three weeks figuring out norms that should have been transmitted in week one.

## Read this next

- [**From One Engineer to Fifteen**](https://sublimecoding.com/blog/from-one-engineer-to-fifteen-engineering-leadership) — Where the 4-person rituals fit in the broader leadership arc.

- [**The Pre-Series-A AI Startup Hiring Plan**](https://sublimecoding.com/blog/pre-series-a-ai-startup-hiring-plan) — Who you hire to build the 4-person team in the first place.

- [**How I'd Hire a Staff Engineer at an AI Startup**](https://sublimecoding.com/blog/how-id-hire-a-staff-engineer-at-an-ai-startup) — The single hire that levels up the team you already have.

---

## How I'd Hire a Staff Engineer at an AI Startup

URL: https://sublimecoding.com/blog/how-id-hire-a-staff-engineer-at-an-ai-startup
Published: 2026-02-23
Tags: hiring, staff engineer, AI startups, engineering leadership

**The title "Staff Engineer" means three different things at three different companies. At an AI startup pre-Series-A, only one of those three is what you actually need.**

I've been on both sides of the staff-engineer interview, hiring for the role at Lavender and BlockFi, and being interviewed for it more times than I want to count. The pattern I see most consistently in misfires: the company hires a staff engineer who's calibrated for a different flavor of "staff" than what the company actually needs, and either the engineer leaves within twelve months or the team works around them.

If you're hiring a staff engineer at an AI startup pre-Series-A, here's the interview process I'd run, and the calibration I'd hold to.

## The three flavors of staff engineer

The title is overloaded. The three distinct shapes:

- **The systems architect.** Designs platforms, sets technical direction across teams, owns the architectural roadmap. Often doesn't write much code. Strongest at large companies with multi-team coordination problems.

- **The principal IC.** Writes the hardest code on the team, owns the most-load-bearing parts of the codebase, mentors senior engineers. Hands-on. The "tech lead, but better."

- **The deep specialist.** Single-domain expert — distributed systems, ML infra, cryptography, real-time graphics. The team needs them when the problem requires their specific expertise; otherwise they're slotted into general work and underperform.

The flavor an AI startup pre-Series-A needs is **flavor 2: the principal IC.** You don't have multi-team coordination problems yet. You don't have a single domain so deep that a specialist is required. What you have is a small team that needs someone who can hold the entire codebase in their head, take the hardest features, and pull the senior engineers up.

If a candidate's resume reads like flavor 1 (systems architect) or flavor 3 (deep specialist), they're not wrong as engineers. They're wrong for this role. Calibrate the funnel for principal-IC type and reject hard against the others, even if their pedigree is impressive.

## The screen: not LeetCode

The first 30 minutes with a staff candidate should not be a coding question. By the time someone has reached staff level, you can confirm they can code through their work history, their code samples, and the take-home. The 30-minute screen is for two questions:

- **Walk me through the most technically difficult thing you've shipped.** Listen for: depth of ownership, awareness of tradeoffs, ability to talk about failure modes, presence or absence of grandiosity. A staff engineer should have at least one or two "I owned this end-to-end and here's where it nearly went wrong" stories.

- **What would you change about how engineering operates at the last company you worked at?** Listen for: opinion, not complaint. A staff engineer should have a clear, articulated point of view about engineering practice. If they don't, they're senior, not staff.

If both answers are strong, move them to the take-home. If either is weak, decline immediately. Do not waste your team's interview hours on a candidate who can't pass these two.

## The take-home: should it exist? Yes. With caveats.

Take-homes are controversial. The argument against: they're disrespectful of senior candidates' time, the signal is noisy, and the strongest candidates won't do them.

I've heard all the arguments and I still believe in take-homes for staff hires, with three constraints:

- **Two hours, hard cap.** If a candidate puts in eight hours, you're getting eight-hour signal — useless for calibrating against the actual job, where they'll have less time.

- **Realistic problem, not algorithmic puzzle.** Build a small CLI tool that solves a real product problem. Wire two real APIs together. Implement a small retry policy with backoff. Things you'd actually ask them to do in week one.

- **Pay for the candidate's time.** Not a lot — $200 for two hours. Sends the message that you respect their time, and it filters out candidates who have so many options they'd rather not bother.

What I evaluate in the take-home: code clarity, naming, error handling, tests if they wrote any, and the README. The README is half the signal. A staff candidate's README should explain what they built, what they considered, what they cut, and why. If the README is missing or one paragraph, you've learned something.

## The interview loop: four rounds

After the take-home, four rounds. Each one tests something specific.

### Round 1: Take-home walk-through (60 minutes)

The candidate explains their take-home submission. You ask: "Why this approach?" "What did you not do, and why?" "What would you do if you had a full day instead of two hours?" "How would you test this in production?"

You're looking for: tradeoff awareness, the difference between "I shipped it" and "I shipped it and here's the production-readiness gap."

### Round 2: Open-ended system design (60 minutes)

Pick a system close to what the company actually builds. "Design the AI evaluation pipeline for a chat product." "Design the agentic tool-use authorization layer." Give the candidate 60 minutes; they drive, you ask follow-ups.

You're looking for: ability to ask clarifying questions before designing, awareness of what they don't know, ability to scope the design to the actual problem (not the perfect-world version), and willingness to push back when your hypothetical doesn't make sense.

The single biggest red flag in this round: a candidate who immediately sketches the "right" architecture without asking who the users are, what the load looks like, what the failure mode is. Senior engineers do that. Staff engineers don't.

### Round 3: Live code review (60 minutes)

You hand the candidate a 200-line PR from your real codebase (or a contrived equivalent). They review it. You watch.

You're looking for: what they comment on, what they miss, how they phrase the comments, whether they catch the bugs you planted, whether they suggest stylistic changes that don't matter and ignore the substantive ones.

This round is the single most predictive signal I've found for staff-level performance. The way an engineer reviews other engineers' code is the way they'll show up to your engineering culture. If the review is sharp, kind, and substantive, you've found a staff engineer. If it's nitpicky, vague, or ego-driven, decline.

### Round 4: Judgment / leadership (60 minutes)

This is the round most companies skip. They shouldn't.

You walk the candidate through three to four scenarios pulled from real situations at your company:

- "A junior engineer pushes a hot fix to production at 11pm Friday without code review. Walk me through what you do."

- "The CTO wants to migrate from Postgres to a new vector database. You think it's premature. How do you handle the conversation?"

- "Production is down. The on-call is in over their head. You're not on call. What do you do in the next 30 minutes?"

- "A peer engineer is consistently producing low-quality work, and the eng manager isn't acting on it. Your move."

You're looking for: temperament, judgment under pressure, willingness to disagree professionally, awareness that the technical decision is rarely the only decision in the room.

## The reference call that matters more than the interview

If a candidate gets through all four rounds, you talk to references. Not the references they list on the resume — those are filtered. Talk to the people who reported to them, and to the people they reported to.

Three questions, in this order:

- "What's the kind of work this person is best at?" — Calibrates strengths.

- "What kind of work do they struggle with?" — Calibrates limits. If the reference can't name a single weakness, they didn't know the candidate well enough; the call is useless.

- "Would you hire them again?" — The most predictive single question in any reference check. The pause, the tone, the qualifications they put on the answer matter more than the literal yes or no.

Three reference calls is enough. Five is overkill. One is too few.

## AI-fluency calibration: the new bar

Here's where the AI-startup version of staff hiring diverges from generic staff hiring. In 2026, an engineer who can't fluently use Claude Code, Copilot, or Codex is not a staff engineer. They might be a great senior engineer, but they're not operating at the leverage a staff engineer should have.

The bar I'd hold to:

- **They use AI tooling daily.** Not "I've tried Copilot." Not "I'm a skeptic." Daily, with opinions about which tool for which job.

- **They know what NOT to use AI for.** Auth code, real-money flows, performance-critical paths. If a candidate says "I use AI for everything," that's a flag.

- **They have a take on team adoption.** A staff engineer should have thought about how AI tooling changes engineering practice at the team level, not just personal productivity.

How to test it: a question in round 4. "Walk me through how you used AI in the last feature you shipped." Listen for specificity. Vague answers signal lip-service usage. Specific answers — "I used Claude to enumerate the edge cases on the state machine before I wrote the code, then I had it review my PR before I opened it" — signal real fluency.

If a candidate is otherwise excellent but lacks AI fluency, hire them anyway and budget two months for them to develop it. If they're average plus AI-skeptical, decline. The bar has moved.

## Comp, equity, and selling them

A staff engineer at an AI startup pre-Series-A in 2026 is looking at:

- **Cash:** $220–280K base. Bay Area / NYC / remote-but-competitive.

- **Equity:** 0.4–0.8% over four years. Higher end if they're early; lower end if they're hire 8.

- **Sign-on:** $25–50K to make up for unvested equity at their previous company.

The candidates you want at this level have options. Selling them is half the job. The pitch that lands: a clear, articulate vision of what the engineering org is going to look like in 18 months, what their role in shaping it is, who they'll be working alongside, and what the realistic path to Series A and beyond looks like for the company.

The pitch that doesn't land: "we're hiring fast, lots to do, hope you like ambiguity." That's not a pitch, that's a confession.

## The first 30 days after they say yes

Hiring a staff engineer is half the work. The other half is onboarding them so they're operational at the level you hired them for.

Three commitments to make in writing during the offer stage:

- A clear "first 90 days" set of expectations. What good looks like at day 30, day 60, day 90.

- Direct access to the founder for the first 30 days. Weekly 1-on-1s. They are part of how the company is run, not three layers below it.

- Ownership of one tangible piece of the platform within 30 days. Not "shadowing." Not "learning the codebase." Something they own with their name on the GitHub commits.

Without these three, the staff engineer ramp drifts and they end up doing senior-engineer-level work for two quarters before someone notices. With them, they're contributing at staff level by month two, and the existing team is leveling up against them.

## The takeaway

Hiring a staff engineer is one of the most consequential hires a pre-Series-A AI startup makes. Get it right and the entire engineering bench levels up. Get it wrong and you've spent $300K+ all-in on a hire who either underperforms or leaves.

The interview process I've described is more rigorous than what most companies run. That's the point. The cost of running this process is high. The cost of hiring the wrong staff engineer is higher.

If you're a founder who hasn't run this kind of loop before, partner with someone who has. The first staff hire is not the time to learn the process from scratch.

## Read this next

- [**The Pre-Series-A AI Startup Hiring Plan**](https://sublimecoding.com/blog/pre-series-a-ai-startup-hiring-plan) — Where the staff engineer fits in the broader hiring sequence.

- [**From One Engineer to Fifteen**](https://sublimecoding.com/blog/from-one-engineer-to-fifteen-engineering-leadership) — The leadership lessons that inform how I'd onboard a staff hire today.

- [**AI-Assisted Engineering Isn't Faster Coding**](https://sublimecoding.com/blog/ai-assisted-engineering-is-a-new-workflow) — The AI-fluency bar, in much more detail.

---

## The Perfect Hire Is Killing Your Team

URL: https://sublimecoding.com/blog/perfect-hire-killing-your-team
Published: 2026-06-26
Tags: hiring, engineering leadership, founders

>
**TL;DR:** Most engineering teams hire for the wrong
things. They screen for pedigree and the ability to perform algorithms
on a whiteboard under a running clock — two of the *weakest*
known predictors of who actually does good work. They reject the people
who’d have compounded into their best engineers: the ones who put in the
effort, own the outcome, and keep growing. The whiteboard ritual
measures a skill nobody uses on the job, the “perfect match” you’re
hunting is a fiction, and the search for it is what’s hollowing out your
team. Stop hiring for the intercept. Hire for the slope.

## The interview tests a skill nobody uses at work

Picture the standard loop. Reverse a linked list on a whiteboard.
Implement quicksort from memory. Explain the time complexity of an
algorithm you last touched in a CS class, with no editor, no
documentation, no internet, and a stranger watching the clock. Get it
perfect on the first try or lose points.

Now picture the actual job. Nobody writes a sorting algorithm from
scratch — they call the one in the standard library, because
re-implementing well-documented, battle-tested code by hand is how you
introduce bugs, not how you ship. The real skill on the job is knowing
*which* tool to reach for, reading the docs well, composing
existing pieces, and recognizing when the obvious approach is wrong. The
interview measures the opposite of that: recall under artificial
pressure, of things you would and should look up the moment you were
doing the work for real.

We built a hiring ritual around a performance that has almost nothing
to do with the performance we’re actually buying.

## What the whiteboard actually measures — and what the data says

It measures two things: how much syntax and trivia you’ve memorized,
and how calmly you perform while being judged. Neither is the job.

This isn’t a hot take; it’s what the people with the most hiring data
concluded years ago. Google ran the numbers on its own famously brutal
process and [Laszlo
Bock, its head of People Operations, called brainteasers “a complete
waste of time”](https://www.inc.com/adam-robinson/google-used-this-brainteaser-to-identify-top-job-candidates-heres-why-it-failed.html) — they predicted nothing except a candidate’s ability
to solve brainteasers, and mostly served to make the interviewer feel
clever. Google replaced them with structured, work-sample-style
assessment because that’s what actually correlated with performance.

The broader selection-science research says the same thing, and it’s
held up even after a [2022
reanalysis that corrected decades-old validity estimates](https://www.siop.org/tip-article/is-cognitive-ability-the-best-predictor-of-job-performance-new-research-says-its-time-to-think-again/):
work-sample tests and structured interviews sit near the top of the
predictive-power ranking, while **years of experience, GPA, and
educational pedigree sit near the bottom.** Read that again,
because it’s the whole game. The two signals the typical broken
interview leans on hardest — an impressive résumé and a flawless
on-the-spot puzzle solve — are among the *least* predictive of
whether someone will be good at the job. You are optimizing your filter
for noise.

## “I don’t know — I’ll look it up” is a senior answer

Here’s a thing the best engineers I know have in common: they don’t
have the docs memorized, and they’re completely unbothered by that. They
know where the materials live, they know how to evaluate what they find,
and they know how to apply it. Ask them something outside their working
memory and they’ll say, plainly, *“I don’t know that off the top of
my head — I’d look it up and figure it out.”*

In most interview formats, that honest, accurate, senior answer loses
points. We’ve built a process that rewards the candidate who confidently
recites and penalizes the one who tells the truth about how knowledge
actually works. But there are always gaps. There are always limits to
what any one person carries in their head. Pretending otherwise —
treating recall as competence — selects for confident memorizers over
honest problem-solvers, which is exactly backwards.

And in 2026 this is no longer even debatable. The memorization
premium has collapsed. [AI-assisted
engineering isn’t faster typing — it’s a different workflow](https://sublimecoding.com/blog/ai-assisted-engineering-is-a-new-workflow), one
where the durable skill is knowing what to ask, how to verify the
answer, and how to integrate it safely. The engineer who reaches for the
right tool and validates the output is demonstrating the *actual*
job. The whiteboard interview was an outdated philosophy long before
LLMs arrived; the tools just made it impossible to keep pretending
otherwise.

## Everyone knows this, and we keep doing it anyway

I have yet to meet an engineer who loves the algorithmic-whiteboard
gauntlet and thinks it’s a wonderful way to find talent. The only people
who enjoy it tend to be the ones who enjoy competitive programming and
grinding challenge sites for their own sake — a real and fine hobby, and
also not the same thing as the job.

So why does it survive? Because it’s easy to administer, it
*feels* rigorous and objective, it lets the interviewer feel
smart, and it’s what the big-name companies do, so copying it feels
safe. None of those reasons is “it works.” It’s cargo-cult hiring:
imitating the visible ritual of a process whose actual results you never
measured. The cost is that you filter your entire pipeline down to one
narrow profile — the person who recently drilled LeetCode — and quietly
discard everyone else.

## Hire for the slope, not the intercept

Here’s the reframe. Stop trying to measure where a candidate is
*today* with a trivia exam, and start trying to measure which
direction they’re moving and how fast. Hire for the slope.

The traits that compound are effort, ownership, and coachability. The
willingness to put in the work, the time, and the deliberate practice to
keep getting better is worth more over two years than any amount of
memorized syntax — because the memorized syntax is a depreciating asset
and the growth habit is an appreciating one. The trait I’d weight
highest is the one I’ve written a whole essay about: [owning the whole outcome](https://sublimecoding.com/blog/professional-owns-the-outcome),
not just the assigned task. That predicts more than any credential.

And the part most hiring managers miss entirely: **good
teammates are forged, not found.** A large fraction of how well
someone performs is a function of the team around them — the standards
they’re held to, the review they get, the patterns they absorb. [The rituals that
make a small team good](https://sublimecoding.com/blog/managing-a-four-person-engineering-team) are also what turn a promising hire into a
great engineer. If you’re only willing to hire someone who’s already
perfect, you’ve outsourced your own most important job: building the
environment that makes people better. The juniors you skip because they
“aren’t there yet” are precisely the [engineers you’ll wish you’d
grown](https://sublimecoding.com/blog/stop-making-senior-engineers) two years from now.

This is why the unicorn hunt is so corrosive. The candidate who
checks every box, matches every keyword, and clears the whiteboard
flawlessly is mostly a fiction — and chasing that fiction leaves seats
empty for months, homogenizes your team into one profile, and rejects
the people who would have become your strongest contributors. The fix
isn’t to stop assessing skill. It’s to assess the *real* skill:
give candidates a realistic task close to the actual work, let them use
their tools and their references the way they would on the job, and
watch how they think, how they recover from not knowing, and how they
own the result. That’s the constructive version — [the actual
process I run](https://sublimecoding.com/blog/how-id-hire-a-staff-engineer-at-an-ai-startup) — and it has nothing to do with reversing a linked
list under fluorescent lights.

The question was never whether someone can recite an algorithm on
command. It’s whether they’ll own the outcome, do the work, and keep
growing. Stop searching for the perfect match. Build the team that
forges great engineers — and hire the people who want to be forged.

---

## The Pre-Series-A AI Startup Hiring Plan: Who to Hire, in What Order, and Why Most Get It Wrong

URL: https://sublimecoding.com/blog/pre-series-a-ai-startup-hiring-plan
Published: 2025-12-29
Tags: hiring, founders, AI startups, engineering leadership, business

**Most pre-Series-A AI founders hire in panic order, not strategic order. The result is a team that can't ship the product the company actually needs to build.**

 **Key takeaways**



 - **Hire in strategic order, not panic order.** Founding engineer → second backend generalist → first frontend specialist → infra/platform → first PM → second backend cluster → first eng manager around hire 8–10.

 - **Don't hire a Director of Engineering before there are at least two ICs to manage.** Don't hire a CISO before Series A. Don't hire a Head of AI when an IC with a clear mandate works.

 - **Equity-heavy through hire 5; base-heavy after.** Comp calibrated to stage and risk, not to what FAANG would pay.



The pattern I see, repeatedly: a founder closes a seed round, gets pressure from the board to "scale the team," and posts five senior backend engineering openings on a Monday morning. Six months later they've hired four backend engineers, the product still doesn't have a designer, the AI features they're shipping look like internal tools, and the BD pipeline that was supposed to fund the next round is empty because no one has been working it.

The right framing is not "scale the team." It's **each hire should either unblock the product or unblock the customer**. If the hire doesn't do one of those, it's an expensive bet you didn't need to make at this stage.

Here's the plan I'd run if I were starting an AI-native company today and going from two co-founders through to a Series A.

## The six hires before Series A

For an AI-native company with two technical co-founders raising a $3–5M seed round, this is the order I'd hire in. The total span is roughly 14 to 18 months from first close to Series A.

### Hire 1: Founding engineer

The first hire is not a "great engineer." The first hire is a third co-founder who didn't get the title.

What you're looking for: full-stack capability, willingness to own a feature end to end, the temperamental capacity to be the only one in the codebase besides you for the first six months. Someone who's been a senior IC at one or two real companies and has decided they want startup risk now.

Comp: heavy equity (1.0–2.5%), market-rate-or-below cash. If they're asking for FAANG cash plus founding-engineer equity, they're not the right hire. The math doesn't work and they're going to bail at month 9.

What this hire should NOT be: a specialist. The first hire is the second pair of hands across the entire stack. The specialists come later.

### Hire 2: Product designer

The single most counter-intuitive hire on this list is also the most important. AI products that look like engineering tools die. Almost without exception.

Your customers cannot tell whether your model is good. They *can* tell whether your product feels considered. A great designer in seat from month four will reshape every feature you ship — for the better — and meaningfully change what an enterprise prospect sees in your demo.

What you're looking for: someone who's shipped product design at a venture-backed startup, ideally one with a complex underlying technology. Senior level. Comfortable with no full-time PM in seat (you're the PM, the founder, until much later).

Comp: market-rate cash, 0.5–1.0% equity. Fewer designers than engineers in the candidate pool, so you'll pay closer to senior-PM rates.

### Hire 3: ML or Applied AI specialist

By month six or seven, your AI features have moved past "wrap an LLM in a UI" and into territory where someone needs to think hard about prompt engineering, retrieval, fine-tuning, evals, and the rest of the AI engineering stack. This is not the founding engineer's job. This is a specialist.

What you're looking for: someone who's shipped AI features in production at another company. *Not* a research scientist. Not a PhD straight out of grad school. The hire is "applied" — they know how to ship, they know how to handle the messiness of LLMs in production, and they have opinions about evals.

Comp: market-rate cash, 0.4–0.8% equity. Hot market — be ready to move quickly when you find the right one.

### Hire 4: The GTM hire

This is where most founders get the order wrong. They hire engineer 3, then engineer 4, then engineer 5, then somewhere around month twelve realize they have no one running the customer side and they're still doing all the BD calls themselves.

By the time you're at four engineers, you should have one person whose job is owning customer development end to end. What flavor of GTM hire depends on your product:

- **Founder-led sales motion still working?** Hire a founding BDR / sales associate to handle the top of funnel and let the founder close.

- **Self-serve / PLG product?** Hire a growth engineer who's also done marketing.

- **Enterprise contracts already pulling?** Hire a founding AE — yes, even at $200k base + variable + equity. The math works if they close one deal.

This hire pays back the seed round in pipeline within their first year if you've hired the right person. Skipping it for "one more engineer" is the most common pre-A mistake.

### Hire 5: Senior product engineer

Now, finally, you hire engineer #3 (after the founding engineer and the AI specialist). This is the engineer who builds product features against the backlog the designer has shaped.

What you're looking for: someone who's shipped product features at scale at a previous startup. Less senior than the founding engineer, but with enough taste to make the right tradeoffs without supervision. Strong frontend or strong full-stack — depends on where the gap is at this point.

Comp: market-rate cash, 0.3–0.5% equity.

### Hire 6: Security / ops person

By month 14, your customer pipeline is asking for SOC 2, vendor questionnaires, and a security trust page. Your [vCISO](https://sublimecoding.com/tools/vciso-cost) has been doing the strategy work, but you need someone in seat for the day-to-day execution. This hire is part security engineer, part DevOps, part compliance ops.

What you're looking for: someone with cloud security and compliance ops experience at a startup of similar stage. Not a full CISO yet — you're not ready for that role. Senior IC with leadership trajectory.

Comp: market-rate cash, 0.3–0.5% equity. The vCISO transitions to advisor; the in-house person owns execution.

## The hires NOT to make pre-A

For every hire on the list above, there's a tempting wrong-stage hire that founders make instead. The list of *don'ts*:

- **Don't hire a full-time PM yet.** Founder is PM. The day you hire a PM is the day product velocity drops 30% as the PM "gets up to speed" and adds a layer between engineering and customers. Wait until post-A.

- **Don't hire an EM yet.** Same reason. You're managing six engineers; you don't need an engineering manager. The founding engineer is the de-facto tech lead.

- **Don't hire a CISO.** Hire a vCISO (covered in [vCISO Math](https://sublimecoding.com/blog/vciso-math-for-ai-founders)). Save the full-time hire for $20M ARR or after a regulatory event.

- **Don't hire a research scientist.** Almost all AI startups don't need one. The applied AI specialist (hire 3) is sufficient until you're shipping novel research as the product.

- **Don't hire a full-time recruiter.** Founder is recruiter. If you can't recruit your own first six hires, you don't yet know what you're hiring for.

- **Don't hire a head of marketing.** Wait until you have a repeatable GTM motion the head of marketing can scale. Until then, the founder owns positioning.

## The compensation framework

The biggest reason founders blow this plan is bad comp framework. They either underpay and lose candidates to bigger checks, or overpay and burn the runway they need for the next 18 months of progress.

The framework that's worked for me:

- **Cash:** roughly 80–90% of market median for a senior at a similar-stage startup. Pull market data from Carta, Pave, or Levels.fyi. Pay slightly below median because you're paying in equity.

- **Equity:** heavy for early hires (founding engineer 1.0–2.5%), tapering down (hire 6 at 0.3–0.5%). Use a tool like Carta to manage option pool dilution carefully.

- **Refresh grants:** commit in writing to a refresh grant at the 24-month mark. This is how you keep early hires from leaving when their original grant gets eclipsed by new joiners' grants.

- **Cash-vs-equity flexibility:** some great candidates need more cash because of life circumstances. Have a documented sliding scale (e.g., "+$20k base = -0.2% equity") so you're not negotiating each one from scratch.

## The post-Series-A inflection

This plan stops at six hires. After Series A, the discipline changes.

You'll go from six to roughly thirty in the year following the A. The hire-by-hire framing breaks down at that velocity; you start hiring against role profiles and team needs. The right framing at that scale is "how many engineers do we need to ship the product roadmap" — but you only earn the right to ask that question after you've shipped pre-A with a tight team that proves the product works.

The single biggest predictor of which AI startups successfully transition pre-A to post-A is whether the team they assembled before the Series A could actually ship. The roster matters more than the headcount. Get the first six right and the rest of the company is downstream of that decision.

## What it actually costs to get this wrong

Founders skip past hiring sequencing because the cost of getting it wrong feels abstract. It isn't. Here's what hiring two extra engineers in months 4–6 instead of a designer + a GTM hire actually costs.

- **Two engineers fully loaded:** ~$500K cash + 1.0% equity over 18 months.

- **Lost product quality from no designer:** hard to quantify directly, but typically manifests as enterprise demos that don't convert. Three lost enterprise deals at $80K ACV each = $240K in lost first-year revenue.

- **Lost pipeline from no GTM hire:** a competent founding BDR generates $300–500K in qualified pipeline in their first six months. Not having one means the founder is doing top-of-funnel work instead of product or fundraising.

- **Compounding delay:** the Series A pitch eighteen months later is "we have great product, weak distribution" — a much harder pitch than "we have great product and a working GTM motion." Down-round risk goes up materially.

Total expected cost of the wrong sequencing in real dollars and equity: somewhere between $800K and $1.5M over two years, plus the fundraising delta. The right sequencing has a better expected value *even if the product takes one more month to ship*, because the customer-side work compounds in parallel with the engineering work.

## Read this next

- [**How I'd Hire a Staff Engineer at an AI Startup**](https://sublimecoding.com/blog/how-id-hire-a-staff-engineer-at-an-ai-startup) — A deep dive on the screen, take-home, and interview loop for one of the most consequential roles on this list.

- [**From One Engineer to Fifteen**](https://sublimecoding.com/blog/from-one-engineer-to-fifteen-engineering-leadership) — What I learned scaling an engineering team during my own founding stretch.

- [**vCISO Math for AI Founders**](https://sublimecoding.com/blog/vciso-math-for-ai-founders) — Why hire #6 starts as a vCISO, not a full-time CISO.

- [**How to Manage a 4-Person Engineering Team Without Becoming a Manager**](https://sublimecoding.com/blog/managing-a-four-person-engineering-team) — The 5 rituals that work at 3–5 engineers, before you need real process.

---

## The Ruby to Elixir Migration That Cut Our Service Footprint From Ten to Six

URL: https://sublimecoding.com/blog/ruby-to-elixir-migration-ten-to-six-services
Published: 2026-02-09
Tags: Elixir, Ruby, Phoenix, migration, engineering, OTP

**We had ten microservices that were 60% Ruby and 40% Elixir. Two years later we had six, fully Elixir, and our on-call alert volume had halved.**

The migration was less about the language and more about what running real-time messaging for 450,000 active students across 900 partner universities forced us to think about. Memory pressure. Long-running connections. Concurrency that didn't tip over. Operational ergonomics that made on-call survivable. Ruby could do all of these things, but every solution required a layer of accidental complexity that Elixir's runtime gave us for free.

What follows is the migration playbook from InsideTrack — the order that worked, the patterns we leaned on, the unexpected wins, and the parts I'd do differently with what I know now.

## The stack we started with

The platform served two-way messaging between coaches and students. Mostly SMS, some email, increasing volume of in-app chat. The architecture, when I joined:

- Three Rails monolith services (web, API, admin)

- Two Sidekiq workers (one for messaging dispatch, one for analytics ingestion)

- Three small Sinatra services (one webhook receiver, one cron scheduler, one feature-flag service)

- Two early Phoenix services (a real-time inbox and a notification dispatcher) — both written by the previous team in a "let's try Elixir" experiment

Total: 10 services, 6 Ruby, 4 Elixir. Combined the team operated 60+ background workers and a Postgres cluster handling several thousand writes per second at peak.

The motivation to consolidate wasn't ideological. It was operational. The Ruby services were memory-hungry, the Sidekiq workers had to be horizontally scaled aggressively to keep up with peak load, and the on-call rotation was getting paged 8–12 times per night during exam season because of the cumulative weight of running too many services.

## The trigger to start moving

Two specific events forced the decision.

First, we lost a contract with a large university because our messaging dispatch latency P99 spiked above the contractual threshold during an exam-season peak. The latency wasn't a code bug — it was Sidekiq queue depth backing up because the worker fleet couldn't scale fast enough. We could have thrown more Sidekiq workers at it, but the marginal cost was high enough that we'd have eaten the contract margin.

Second, our on-call engineer quit. The exit interview was honest: too many services, too much ambient alert noise, no clear ownership boundaries. The team morale knock was as expensive as the lost contract.

The combined message — both customer-facing and internal — was that the architecture was the bottleneck. Not the team's effort, not their skill, not the underlying tech of any single service. The number of services was the problem, and the runtime characteristics of Ruby + Sidekiq made consolidation in Ruby genuinely hard. Elixir's BEAM gave us a runtime that handled the same workload with one or two services instead of seven.

## What's the right order to migrate Ruby services to Elixir?

The first lesson I learned was that migrations work backwards. You don't migrate the easy thing first; you migrate the thing that's most painful to keep on the old stack.

Our order, in retrospect:

- **The messaging dispatcher.** The most painful service. The one driving the on-call alerts. Migrating it first meant on-call ergonomics improved within the first quarter and the team had visceral evidence the migration was paying off.

- **The analytics ingestion worker.** Second-most painful. Sidekiq queue depth here was a chronic capacity issue. Re-implementing as a GenStage pipeline in Elixir collapsed memory usage by ~70%.

- **The webhook receiver and cron scheduler.** Smaller services we consolidated into a single Phoenix app with multiple endpoints and a Quantum scheduler. Saved two services in one move.

- **The feature-flag service.** Replaced wholesale with a managed service (LaunchDarkly). Not strictly an Elixir migration — but the Ruby-to-Elixir framing forced us to evaluate "is this our problem to host at all?" and the answer was no.

- **The Rails admin service.** Migrated to Phoenix LiveView. Surprised us by being one of the easier moves once we got over the learning curve.

- **The Rails API service.** Migrated last and most carefully. This was the customer-facing surface; we ran a dual-deploy period for two months with traffic mirrored to both stacks for parity testing.

- **The Rails web monolith.** Stayed Ruby. We never migrated it. Too much business logic, too low a marginal benefit. Lesson: not everything needs to move.

Final state: six services, all Elixir except the Rails web monolith. One major Phoenix app handling messaging dispatch, ingestion, webhooks, scheduling, and admin. Three smaller Phoenix apps for the inbox, notifications, and a public API. Plus the Rails web monolith. Down from ten.

## The wrong order I tried first

My initial plan, before reality course-corrected it, was to start with the API service. Reasoning: it's the most visible, it has the most code, getting it migrated first proves the platform.

That plan was wrong. The API service was the riskiest single move and had the lowest operational pain associated with it. We would have spent six months on a high-risk migration that wouldn't have meaningfully reduced on-call burden, while the messaging dispatcher kept paging us. The team would have lost faith in the migration before we got to the actually painful services.

The corrected ordering — pain first, value-prove second, polish last — is the framework I'd use again. **Migrate the service that's hurting you most, even if it's not the most strategic one.** The early operational win pays for the political capital you'll spend later on the harder migrations.

## The Elixir patterns we leaned on

Three OTP primitives did the bulk of the work.

**GenServer for stateful work.** The messaging dispatcher's previous architecture was Sidekiq + Postgres rows for state. Re-implementing as GenServers per active conversation eliminated the database churn for state machine transitions and let us hold conversation state in memory cheaply. The supervision tree handled crashes per-conversation without taking down the whole dispatcher.

**Registry for routing.** Looking up "which GenServer handles conversation 42" is a few microseconds with Registry. We used it everywhere — for active conversations, for active user sessions, for active webhook subscriptions. Dead simple, fast, and it eliminated a class of "where does this message go" problems that had been complex in the Ruby version.

**Supervision trees for failure isolation.** The single most important property of the Elixir runtime is that one bad message can't take down the service. A ten-thousand-conversation dispatcher might have one or two crashing GenServers at any given moment; they get restarted in milliseconds and the other 9,998 conversations don't notice. Sidekiq could not give us this without significant infrastructure investment.

The fourth pattern, less universal but useful: **GenStage for backpressure-aware pipelines.** The analytics ingestion worker was a GenStage pipeline with explicit demand-driven flow control. Made the queue-depth-spike pattern that had been killing us in Sidekiq simply not exist as a category.

## The unexpected wins

**Halved on-call alerts.** By far the biggest morale and retention win. The team that had been getting paged 8–12 times a night dropped to 3–4. Not because the services were doing less work, but because they handled load shedding, partial failures, and self-healing without paging humans.

**Better dev ergonomics for the kind of work we did.** Pattern matching against incoming messages made the dispatcher code dramatically clearer than the Ruby case statements it replaced. `iex` with remote shell into a running production node was an operational superpower.

**Hiring quality went up.** This surprised me. The Elixir candidate pool is smaller, but the candidates who self-select into Elixir tend to be more curious and more rigorous than the Ruby candidate average. We hired better engineers per interview hour after the migration than before.

## The unexpected losses

**Gem ecosystem.** I missed Devise. I missed ActiveAdmin. I missed Sidekiq Pro's UI. There were Elixir-equivalent libraries for most of these, but the Elixir ecosystem in 2018-2019 was visibly less mature, and rolling our own auth or admin UI cost more time than the migration math accounted for.

**Hiring pool narrower.** Yes, the candidates who came through were better. But the funnel was smaller. We'd see 30 Ruby applicants for every 5 Elixir applicants. For a small team this didn't matter. For a team scaling fast, it would have been a constraint.

**Internal training cost.** Engineers coming from Ruby need 2–3 months to be productive in Elixir. We absorbed that cost but it was real and it slowed the migration. Account for it explicitly in your timeline.

## When should you not migrate to Elixir?

The math has shifted somewhat since 2019. I would not unconditionally recommend a Ruby-to-Elixir migration today. The cases where I'd push back:

- **You're not running real-time, long-lived connections.** The killer features of the BEAM are concurrency and supervision. If your workload is short, request-response, and stateless, Ruby/Rails on a modern hosting platform is genuinely fine.

- **Your team has zero Elixir experience and you're already understaffed.** The 2-3 month productivity dip per engineer is real. If you can't afford it, don't start.

- **Your product is dominated by AI features, not real-time messaging.** The AI ecosystem in Python is significantly stronger than in Elixir. Most AI startups today should be in Python or Go for the AI portion, regardless of what the rest of the stack runs.

- **Ruby 3 + YJIT is meeting your needs.** The performance gap between modern Ruby and Elixir narrowed considerably with YJIT. If your Ruby services aren't hurting you, leave them alone.

The right reason to migrate is operational pain that's expensive to solve in your current runtime. The wrong reason is novelty.

## What I'd do differently

If I were running this migration again today:

- **I'd budget the per-engineer onboarding cost explicitly.** 60 days off the keyboard for the first migration project, then ramp. We crashed into this; it should have been planned.

- **I'd build the dual-stack observability layer first.** Migrating with consistent metrics across both stacks would have made the parity testing meaningfully easier. We bolted this on.

- **I'd skip the LiveView migration of admin and use a managed admin tool.** LiveView is great. The admin we built was fine. But the time we spent on it was better spent on the API migration.

- **I'd not migrate the Rails web monolith. Same conclusion. We made the right call there.**

If you're scoping a migration like this — language consolidation, service-count reduction, or a Ruby/Rails monolith that's hit its limits — I do this kind of architectural work as a [fractional engineering lead](https://sublimecoding.com/consulting). The interesting decisions aren't language choice; they're sequencing and what you preserve in the existing data layer.

## The takeaway

Migrations are paid for by operational pain reduction, not by language preferences. The Ruby-to-Elixir move at InsideTrack worked because real-time messaging is exactly the workload BEAM is built for, and the operational pain we were running into was specifically the kind that BEAM eliminates.

For other workloads, the calculation may go the other way. The disciplined version of the question — "what's hurting us today, would moving runtimes solve it cheaply, and can we afford the transition cost" — is a much better framing than "what's the right tech stack for our company in 2026." The right answer to that latter question is almost always "the one you already have, optimized harder."

## Read this next

- [**Migrating 225K Users from AWS Cognito to Auth0**](https://sublimecoding.com/blog/aws-cognito-to-auth0-migration-without-forcing-logout) — A different migration war story, same disciplined pattern: pain first, value-prove second.

- [**How We Cut $350K From Cloud Spend**](https://sublimecoding.com/blog/cut-350k-cloud-spend-six-months) — When the platform you migrate to also rewrites the cost structure.

- [**AI-Assisted Engineering Isn't Faster Coding**](https://sublimecoding.com/blog/ai-assisted-engineering-is-a-new-workflow) — How modern teams approach migrations with AI tooling in the loop.

---

## Your AI Product Needs a Telemetry Layer Before It Needs a Better Model

URL: https://sublimecoding.com/blog/your-ai-product-needs-telemetry-before-better-model
Published: 2026-01-26
Tags: AI, engineering

**I've watched three AI startups burn months trying to "improve the model" when they couldn't even tell which prompts produced which outputs at scale.**

Every team had the same instinct: hallucination rate too high, response quality inconsistent, costs creeping — must be a model problem, let's tune the prompts, let's swap to GPT-5, let's fine-tune. None of them stopped to ask the more useful question first: *what's actually happening inside the model calls we're already making?*

The answer, almost always: nobody really knew. There was no production logging of prompts. No structured capture of model outputs. No correlation between which user did what and what the model returned. The team was making decisions about model improvement based on cherry-picked screenshots and vibes.

That's not a model problem. That's an instrumentation problem. And it's solvable in two weeks of disciplined engineering, which buys you the visibility to know whether the model problem is even real.

## What AI telemetry actually means

Classic application telemetry — request rate, latency, error rate — does not tell you anything useful about an AI feature. A successful 200 response from your LLM endpoint tells you nothing about whether the response was correct, helpful, or hallucinated. You need a different layer of observability that's specific to how AI features fail.

The four things you must capture for every model call:

- **The full prompt.** Every variable interpolation. Every system prompt. Every retrieval-augmented context. Stored as structured data, not a stringified blob.

- **The full response.** Including any tool calls, function calls, or structured outputs. Stored verbatim.

- **The cost and latency.** Tokens in, tokens out, dollar cost, wall-clock time. These compose into your unit economics.

- **The user context.** Who triggered this call, in what feature, against what state. Anonymized if you must, but linkable to the user session.

Without those four, you cannot reason about model performance at any scale beyond "let me copy this prompt into the playground and see what happens." That's not engineering, it's gambling.

## The four-layer telemetry stack

Once the basics are captured, the actual decisions you make benefit from layered aggregation.

### Layer 1: Request-level telemetry

Every model call gets logged with the four-tuple above plus a request ID. This is the source of truth. Every other layer aggregates from this layer.

Storage decisions matter here. The volume can be large — for a product making 100k model calls a day, this is 100k structured rows daily. We chose Postgres with JSONB columns at Lavender, with a 90-day retention policy. Worked fine for our scale; would not scale to 10M calls/day. Use what fits.

### Layer 2: Feature-level aggregation

Each model call belongs to a feature: "summarize," "draft email," "suggest reply," etc. Aggregate the request-level data by feature to answer questions like:

- What's the median response time of the "draft email" feature this week?

- What's the daily cost of "summarize" over the past 30 days?

- Which feature has seen the biggest cost spike since the last release?

This is the layer where you start making product decisions: "the suggest-reply feature costs 4x what summarize does and gets used 1/10 as much — we should kill it or rebuild it."

### Layer 3: User-level signal

Each user has interactions across multiple features. Aggregate at the user level to answer:

- Are heavy users seeing more or fewer hallucinations than light users?

- Is there a cohort of users for whom the feature consistently fails?

- What's our cost per active user per week?

The user-level layer is where you discover that your model is fine for 90% of users but catastrophically bad for the specific use case 10% of users have. Without this layer, that 10% is invisible.

### Layer 4: Aggregate trends and regression detection

Daily / weekly rollups across the whole product. The metrics that go on a dashboard the founder reads every Monday morning:

- Total cost trend

- Cost per active user trend

- P95 latency trend

- Hallucination signal trend (more on this below)

- Feature-level usage distribution

The point of layer 4 is regression detection. When something breaks, you want to know within 24 hours, not 21 days into the quarter when finance asks why the OpenAI bill tripled.

## The hallucination signal

Hallucination is the hardest thing to measure because there's no ground truth label at runtime. Real-world signals that approximate it:

- **User regenerates the response.** One of the strongest negative signals. If a user immediately clicks "regenerate," they didn't like what they got.

- **User edits the response heavily before using it.** If you have a copy-and-edit flow, measure the edit distance.

- **User abandons the feature mid-flow.** Strong signal something went wrong.

- **Explicit thumbs-up / thumbs-down.** Lowest-volume signal but the cleanest. Add it everywhere it's not annoying.

- **Response contains markers of uncertainty.** "I don't have information about" or "I cannot determine" — sometimes useful, sometimes a euphemism for hallucination.

None of these is a clean ground-truth label. Combined, they give you a directional indicator that's good enough for relative comparisons over time. The goal isn't "what's our true hallucination rate" — that's unanswerable. The goal is "is hallucination getting better or worse this week, and which features are driving the change."

## Tooling I'd reach for

The build-vs-buy decision for AI telemetry has shifted in the last 18 months. There are now real options.

- **[LangSmith](https://www.langchain.com/langsmith)** — strong if you're already using LangChain. Decent if you're not. Captures request/response/cost out of the box.

- **[Helicone](https://www.helicone.ai/)** — proxy-based capture. Lowest integration cost — point your LLM SDK at Helicone's URL, get telemetry for free. Best for early-stage teams that want zero-config.

- **[Langfuse](https://langfuse.com/)** — open source, self-hostable. Good for teams with security/data residency concerns.

- **Custom OpenTelemetry instrumentation.** If you already have a strong observability stack (Datadog, Honeycomb, etc.), wrapping your model calls in OTel spans is sometimes the right answer because it integrates with existing dashboards.

For pre-Series-A AI startups I'd start with Helicone and graduate later. The integration cost is one afternoon. The telemetry you get back is enough to make the next dozen product decisions correctly.

## Model problem or instrumentation problem?

The most useful framing I've found, when an AI feature is underperforming:

**Can you, right now, answer these five questions in under five minutes?**

- What was the prompt and response of the last 10 calls to this feature?

- What's the median latency for this feature over the past 7 days?

- What's the daily cost for this feature, broken out by model?

- Which users had the worst experiences this week, by hallucination signal?

- How does any of this compare to two weeks ago?

If the answer to any of these is "I don't know" or "let me write a query," you have an instrumentation problem, not a model problem. Fix instrumentation first. Then look at the data, and the model problem either becomes obvious — or evaporates because what looked like a model problem was actually a prompt regression in last week's deploy.

## A concrete example

At Lavender, we shipped a new prompt template for one of our AI features early in 2025. The hallucination signal — measured via the regenerate-rate — climbed about 60% over the next two weeks. The instinct was "the new prompt is worse, let's rewrite it."

Telemetry told a different story. The regenerate-rate climbed for users on a specific email template that one of our customer-success team had recommended internally. The new prompt was fine. The customer template was triggering an edge case we hadn't anticipated, and the regenerate-rate spike was an artifact of that template being used 4x more than usual.

The fix was a 20-line guardrail in the prompt that handled the edge case. Hallucination signal dropped by 40% within 72 hours. We didn't tune the model. We didn't change LLMs. We did not run a single eval. We instrumented, looked at the data, found the actual cause, fixed it.

That story is impossible to tell without telemetry. Without it, the team would have spent two weeks rewriting the prompt, regressing on something else, and ending up worse than where they started. With it, the cause was obvious within 90 minutes of looking at the data.

## The takeaway

Most AI startups will eventually need to think hard about the model. None of them should think about the model first. The order is:

- **Instrument.** Capture every model call, structured and queryable.

- **Aggregate.** Build feature-, user-, and trend-level views.

- **Look.** Stare at the data for a week. Most "model problems" reveal themselves as something else.

- **Then, if needed, tune the model.** But you'll be tuning against actual data, not vibes.

The two weeks of disciplined engineering this requires is the highest-leverage AI work most startups aren't doing. It's also boring. Which is exactly why doing it is an edge over teams that go straight to fine-tuning.

## The team discipline this requires

Telemetry is a code problem for half a sprint and an organizational problem forever after. The engineering team has to keep instrumentation current as new features ship, or the system rots within a quarter.

The disciplines that worked at Lavender:

- **No model call ships without telemetry.** Code review checklist item, enforced. New AI feature PRs get rejected if they don't wire up the four-tuple capture.

- **One engineer owns the telemetry layer.** Not full-time, but they're the named point of contact. Schema evolution, dashboard updates, retention policies — they own it. Without an owner, the layer drifts.

- **Weekly review of the dashboards.** 15 minutes at the top of an engineering meeting. Just looking at the trends. Catches regressions while they're small and trains the team to think in terms of these metrics.

- **Cost alerts before user complaints.** If the daily AI spend deviates from the rolling 7-day median by more than 30%, it pages the on-call. Most product issues show up here before they show up in support tickets.

The instrumentation work is one or two weeks. The discipline of keeping it useful is forever. Build the muscle early — adding it later, against an existing AI product with no telemetry, is meaningfully harder.

## Read this next

- [**How I'd Run Security at an AI-Native Company in 2026**](https://sublimecoding.com/blog/running-security-at-an-ai-native-company-2026) — The audit-and-security layer of AI observability — what to log for incident response and customer trust.

- [**AI-Assisted Engineering Isn't Faster Coding**](https://sublimecoding.com/blog/ai-assisted-engineering-is-a-new-workflow) — Engineering discipline applied to the team using AI; this post applies it to the AI itself.

- [**How We Cut $350K From Cloud Spend**](https://sublimecoding.com/blog/cut-350k-cloud-spend-six-months) — The same instrumentation discipline applied to infrastructure, with bigger dollar consequences.

---

## Surviving Technical Due Diligence as an AI Founder

URL: https://sublimecoding.com/blog/surviving-technical-due-diligence-ai-founder
Published: 2026-06-30
Tags: AI startups, founders, engineering leadership

>
**TL;DR:** Every technical due diligence guide on the
internet is written for the investor’s associate — a checklist of what
*they* should poke at. You’re on the other side of that table,
and nobody writes for you. So here’s the inverse. The reviewer opens
your commit history before they open your README, and they’re reading it
for *who actually built the core IP*, not how clever it is. In
2026 the AI-specific layer is where rounds get re-priced: model
provenance, training-data rights, whether your eval is reproducible, and
how badly you’re locked into one LLM vendor. And the part nobody will
tell you because it doesn’t sell a checklist — there are gaps you should
**not** scramble to fix, because a competent reviewer
*expects* them at your stage. Panic-fixing them looks worse than
the gap. The skill isn’t having no holes; it’s knowing which holes to
name out loud with a plan.

## The checklists are all pointed the wrong way

Search “technical due diligence” right now. Every result on the first
page is written for the person doing the diligence — frameworks for VC
associates, “30 questions to ask a startup’s CTO,” scorecards for
evaluating an engineering team you’re about to wire money into. All of
it is the *reviewer’s* manual.

That’s useful if you’re the reviewer. It’s almost useless if you’re
the founder about to be reviewed, because it tells you what gets checked
but not what any of it *means*, what’s load-bearing versus
theater, or what a “finding” actually does to your terms.

I’ve sat on both sides of that table. I’ve been the technical
reviewer an investor brings in to spend a week inside a startup’s repo
and data room and write the memo that decides whether the round closes
at the agreed price, closes lower, or quietly dies. And I’ve been the
person sitting next to a founder for the two weeks *before* that,
getting the house in order so the memo comes back clean. The view from
those two chairs is very different, and the gap between them is exactly
where founders lose leverage they didn’t have to lose.

This is the founder’s-seat version. What the reviewer actually opens
first, what they’re really reading it for, the AI-specific things that
are new in 2026, the red flags that genuinely tank or re-price a round —
and, honestly, the gaps you should leave alone.

## What the reviewer opens first (and it isn’t your pitch)

Founders assume diligence starts with the architecture diagram. It
doesn’t. A good reviewer goes to the evidence that’s hard to fake, and
reads the polished stuff last. Here’s the actual opening sequence,
roughly in order.

**Commit history and the contribution graph.** This is
the first thing I open, every time. Not the code — the *history*.
`git log --author`, contribution-by-author over time, when
the core files were written and by whom. It answers the one question
diligence exists to answer: *does the team in the room actually own
the IP they’re claiming?* A repo where 90% of the foundational
commits came from one contractor who left, or from a co-founder who’s
now “advising,” tells a story the pitch deck won’t. So does a history
that starts three weeks ago with one giant “initial commit” of 40,000
lines — that’s either a migrated repo (fine, but I’ll want the old one)
or a story I’m not being told.

**Who wrote the core IP.** Related but distinct. I’m
mapping the handful of files that *are* the company — the
model-serving layer, the eval harness, the retrieval logic, whatever the
moat actually lives in — to specific humans, and then checking those
humans are full-time and on the cap table. The scary version: the
genuinely novel part was written by someone who isn’t staying.

**The README and onboarding path.** I will try to run
your project. Not to judge your code style — to measure *how long
until a new engineer is productive*. A README that gets me to a
running local environment in under an hour signals a team that can hire
and scale. A README that’s three stale commands and a Slack handle to
ping signals key-person risk, because the system only runs in one
person’s head.

**Test coverage reality versus the claim.** Nobody
expects 90% coverage at seed. But I’m checking whether tests exist
*where the money is* — the billing path, the auth boundary, the
core inference logic — versus 80% coverage concentrated on trivial
utility functions to make a number look good. The gap between “we have
good test coverage” in the data room and what’s actually covered is one
of the most common credibility leaks I find.

**Infra and secrets hygiene.** A thirty-second
`git log -p` grep for high-entropy strings. Hardcoded API
keys in the history, a `.env` committed in 2024, prod
credentials in a Slack export — these aren’t just security findings,
they’re *judgment* findings. They tell the reviewer how the team
operates under deadline pressure, and the round is happening precisely
because the team is under pressure.

None of this is the impressive part of your company. That’s the
point. Diligence is calibrated to find the gap between what you
*say* and what’s *true*, and the boring artifacts are
where that gap shows.

## The 2026 AI layer: where rounds actually get re-priced

Everything above applies to any software company. What’s new — and
what most founders are least prepared for — is the AI-specific diligence
layer that’s become standard in 2026. This is where I’ve watched
valuations move, because these are the questions a generalist reviewer
didn’t know to ask three years ago and every competent one asks now.

**Model and weights provenance.** If you fine-tuned or
trained anything, where did the base model come from and what’s its
license? An astonishing number of startups built a product on a base
model whose license prohibits commercial use, or stacked a fine-tune on
weights with a non-compete clause baked into the terms. I want a
provenance chain: base model → license → your modifications → the
license you’re shipping under. If that chain has a gap, your core asset
has a legal cloud over it, and that’s a re-pricing event, not a
footnote.

**Training-data rights and the paper trail.** Where did
your training and eval data come from, and can you prove you had the
right to use it? Scraped data, data from a previous employer, customer
data used for training without the contractual right to do so — this is
the single fastest-growing category of finding I see, and it’s the one
that scares acquirers most because it follows the company. “We scraped
it and everyone does” is not a paper trail. You want documented sources,
licenses or contracts, and a clean answer to “could a customer demand
you delete their data from your model?” If the buyer is really after the team rather than the company, the exam is a different one entirely — [what reverse acqui-hires actually diligence](https://sublimecoding.com/blog/reverse-acqui-hire-technical-diligence).

**Eval reproducibility.** Your deck says you beat the
incumbent by 12 points on some benchmark. Can I *re-run* that
eval? A reviewer who knows the space will ask for the eval harness and
try to reproduce the number. If the benchmark only lives in a
screenshot, or the harness has the test set leaking into context, or the
number was cherry-picked from the best of twenty runs — that’s not just
a credibility hit, it calls the entire technical claim into question.
Reproducible evals are quietly becoming the strongest *positive*
signal an AI startup can offer in diligence. Almost nobody has them
tidy. Having them is a moat in the room.

**Third-party LLM dependency and vendor lock-in.** How
much of your product *is* a prompt to someone else’s model?
That’s not automatically bad — most great AI products are wrappers
around a foundation model plus real proprietary scaffolding. But the
reviewer is sizing the risk: if your single provider 3x’s their price,
deprecates the model you depend on, or ships your exact feature
natively, what happens to your margins and your moat? The answer they
want isn’t “we’d be fine” (nobody believes it). It’s evidence you’ve
*thought about it* — an abstraction layer over the provider, a
fallback model you’ve actually tested, an honest read on which parts of
your value are yours versus rented.

**Prompt-injection, output validation, and data
isolation.** For anything agentic or anything touching customer
data, the security posture is now a first-class diligence item, not an
afterthought. Can a malicious input make your agent exfiltrate another
tenant’s data or call a tool it shouldn’t? Is model output validated
before it hits a database or an API, or do you trust the model’s text?
Is tenant data isolated, or does everything share one context window and
one vector store? You don’t need to be airtight at seed, but you need to
demonstrate you understand the threat model. I’ve written more on what
that posture looks like in practice in [running
security at an AI-native company](https://sublimecoding.com/blog/running-security-at-an-ai-native-company-2026).

**Inference cost and unit economics.** What does one
unit of usage actually cost you in inference, and what’s the trend? A
product that’s magical because it burns four dollars of tokens per
user-session per day has a unit-economics problem dressed up as a
product. The reviewer is checking whether your gross margin survives
contact with scale, or whether growth makes the burn worse. Have the
per-request cost, the trajectory as you’ve optimized, and an honest read
on margin at 10x volume.

## The red flags that actually tank or re-price a round

Not every finding is equal. Some get a note in the memo; some change
the number. Here are the ones that genuinely move terms, why they scare
a reviewer, and the cheap fix if you have time before the process
starts.

Red flag
Why it scares the reviewer
The cheap fix

Core IP written by someone who’s gone
Key-person risk on the actual moat; can you even maintain it?
Document the system; get the departed contributor to sign a clean IP
assignment if not already done

Secrets in git history
Judgment + security failure; what *else* is sloppy under
pressure?
Rotate every exposed credential now; don’t claim it was “never real”
— own it and show the rotation

Training data with no provenance
Legal cloud over the core asset that follows the company
Write the honest data-source memo *before* you’re asked; flag
the gaps yourself

Benchmark you can’t reproduce
Calls every technical claim in the deck into question
Build a clean, runnable eval harness; restate numbers
conservatively

“Test coverage is great” + tests only on trivia
Credibility leak — what else is overstated?
Just be precise about what’s covered; don’t inflate the claim

100% dependency on one LLM with no abstraction
Margin and moat both controlled by a third party
A thin provider abstraction + one tested fallback is enough to
change the story

No environment/tenant isolation in a multi-tenant AI product
One injection away from a breach that ends the company
Isolate tenant context and data stores; document the boundary

The pattern across that whole table: the *technical* severity
matters less than what the finding implies about the team. A reviewer
extrapolates. One overstated claim makes them re-read every other claim
with suspicion, and that suspicion is what re-prices a round — not any
single bug.

## The contrarian part: gaps you should NOT panic-fix

This is the section no investor-side checklist will ever write,
because the entire genre exists to find problems. But it’s the most
valuable thing I tell a founder before diligence, and it’s the part that
builds trust precisely because it’s not trying to sell you a frantic
remediation sprint.

A competent reviewer is not looking for a startup with no gaps. A
seed-stage startup with no technical debt, full test coverage, SOC 2,
and a hardened internal admin tool is *suspicious* — either
you’re lying, or you spent your runway on the wrong things instead of
finding product-market fit. The reviewer is calibrating against your
stage. So some gaps are not findings; they’re *expected*, and
scrambling to paper over them right before the process makes you look
like you don’t understand your own stage.

Here’s the calibration I use:

The “gap”
What a reviewer actually expects at your stage
What to do

Some tech debt, a few TODOs, a known-ugly module
Expected at seed and Series A. Its absence is the red flag.
Leave it. Have a one-line answer for *why* it’s fine for
now.

No SOC 2 at seed
Standard. SOC 2 is a revenue tool, not a security one, and
pre-revenue it’s premature.
Don’t start an audit to impress a reviewer. Know *when*
you’ll need it.

An internal tool that isn’t hardened
Internal tooling is supposed to be scrappy.
Note it’s internal-only and access-controlled; move on.

Monolith instead of microservices
A monolith at your stage is *correct*, not naive.
Defend it as a deliberate choice, because it is one.

Manual steps in deploy
Acceptable early; reviewers care that you *know* which are
manual.
List them honestly. The list is the maturity signal.

I’ve written separately about why [SOC 2 is a
revenue tool, not a security tool](https://sublimecoding.com/blog/soc-2-is-a-revenue-tool-not-a-security-tool) — chasing it pre-revenue to look
good in diligence is one of the most common ways founders burn runway on
the wrong thing.

The real skill isn’t having zero gaps. It’s **framing a known
gap with a credible plan instead of hiding it.** Compare two
answers to “your test coverage is thin on the inference path”:

- *Hiding it:* “Oh, coverage is solid, we’re in good shape
there.” — Now when the reviewer finds the gap (they will), every other
claim you made is suspect.

- *Framing it:* “Coverage on the inference path is
intentionally light right now — we’ve been moving the model interface
weekly and locking it down with tests would’ve slowed iteration. Now
that the interface is stabilizing, here’s the two-week plan to get it
covered, and here’s the one critical path that *is* tested today
because a regression there bills customers wrong.” — Now the gap is
*evidence of judgment.*

Same gap. Opposite outcomes. The reviewer isn’t grading the codebase;
they’re grading whether they can trust the founder’s self-assessment for
the next several years. A founder who names their own weaknesses
accurately is worth more than one with a slightly cleaner repo and a
tendency to round up.

## A short pre-diligence prep checklist

If you’ve got a process starting in the next month, this is what
actually moves the needle — in priority order, not “best practices”
order:

- **Read your own commit history as a stranger.** Who
wrote the core files? Is everyone who matters full-time and on the cap
table? Any “initial commit” mysteries? Fix the *story*, not the
code.

- **Grep your git history for secrets.**
`git log -p | grep`-style sweep for keys and
`.env` files. Rotate anything you find. This is an hour and
it’s pure downside protection.

- **Write the data-provenance memo before you’re asked.**
Every training/eval data source, the right you have to use it, and the
honest gaps. Bringing this unprompted is a massive trust signal.

- **Make your headline benchmark reproducible.** A clean
eval harness a reviewer can run, with no test-set leakage. Restate any
number you can’t reproduce.

- **Write a one-page LLM-dependency honest take.** Which
provider(s), what’s abstracted, what your fallback is, your per-request
cost and margin trajectory.

- **List your known gaps with plans.** The tech debt, the
manual deploy steps, the un-hardened internal tool — written down, with
a one-line “why it’s fine now / when we fix it” for each. This
*is* the framing move, done in advance.

- **Confirm the human story.** IP assignments signed by
everyone who touched the core, including contractors and departed
founders. This is the gap most likely to actually kill a deal.

Notice what’s not on that list: a six-week refactor, a rushed SOC 2,
a microservices migration. None of that survives the cost-benefit math
right before a raise, and most of it makes you look like you’re
optimizing for the wrong audience. If you’re staffing for the months
*after* the round instead, that’s a different exercise — I’ve
laid out how I think about it in the [pre-Series-A AI startup
hiring plan](https://sublimecoding.com/blog/pre-series-a-ai-startup-hiring-plan).

## You shouldn’t do this alone

Diligence is one of the few moments where a founder is being judged
on a dimension — engineering depth, security posture, technical honesty
— that most founders, even technical ones, haven’t had to perform under
an adversarial expert before. The cost of getting it wrong is measured
in valuation, and it’s concentrated into a single short window.

This is exactly the kind of thing a fractional CTO or technical
advisor does *with* you: reading your data room the way the
reviewer will, finding the findings before they do, and helping you
frame the real gaps with credible plans instead of hiding them. If
you’ve got a process coming up, [that’s a
conversation worth having early](https://sublimecoding.com/consulting) — before the data room opens, not
after the memo comes back.

---

## Elixir Is the Language AI Codes Best

URL: https://sublimecoding.com/blog/elixir-ai-codes-best
Published: 2026-05-27
Tags: Elixir, AI tools, engineering

>
**TL;DR:** A Tencent benchmark across 20 languages found Elixir had the **highest LLM code-completion rate of any mainstream language** — 97.5% of problems solved by at least one model, with Claude Opus 4 hitting **80.3% on Elixir** vs **74.9% on C#** and **72.5% on Kotlin**. [Dashbit broke down why](https://dashbit.co/blog/why-elixir-best-language-for-ai). The reasons aren't a coincidence — they're the same boring properties that have always made Elixir pleasant to work in, now compounded by the fact that AI agents are writing more of your code every quarter. The strategic takeaway isn't "rewrite everything in Elixir." It's that the cost of choosing the trendy stack now includes "your AI tools will be measurably worse at it."

## The benchmark nobody saw coming

I have a small reflex when a benchmark says my preferred stack wins: assume someone cooked the books. So when [Dashbit pointed at a Tencent study](https://dashbit.co/blog/why-elixir-best-language-for-ai) showing Elixir at the top of an LLM coding benchmark, my first move was to look for the catch.

I didn't find one. The result is real. Across 20 languages, **97.5% of Elixir problems were solved by at least one model — the highest of any language tested.** Claude Opus 4 scored **80.3%** on Elixir, against **74.9%** on C# and **72.5%** on Kotlin. Those are not borderline gaps; in a benchmark where a few points decides the order, eight points is a moat.

The puzzle is *why*. Elixir is not the most popular language. It is not in the top ten for Stack Overflow answers or GitHub commits. Models should have a lot less Elixir to learn from than, say, Python or JavaScript. And yet here we are.

Dashbit's answer is the one I find convincing: the same language design choices that make Elixir nice to work in for humans turn out to be a *force multiplier* for the next decoder predicting the next token. None of it is about how much training data exists. It's about what the data looks like when there is some.

## What Dashbit's argument actually says

The full piece is worth reading at the source, but the load-bearing claims are four.

**Immutability gives models local reasoning.** In mutable languages, a function can quietly mutate an object passed in from a caller, and now reasoning about what the function does requires understanding everyone who might have touched the object first. Dashbit calls this "spooky action at a distance." Elixir doesn't allow it: anything a function needs is given as input, anything a function changes is given as output. A model — or a human — can predict the next few lines from the function signature alone. The pipe operator (`|>`) then makes the *flow* of transformations literal in the source. Local reasoning is cheap, and cheap is what models are good at.

**Documentation is a first-class language feature.** Elixir distinguishes `@moduledoc` and `@doc` from inline comments. The doc strings are part of the language; they ship to HexDocs; they support `iex>` examples that are *also* executed as tests via doctests. The training corpus for Elixir is therefore unusually clean: a function's docs include a literal demonstration of its inputs and outputs, verified by CI to still be true. That is high-signal data per token. The recent addition of TypeSense-backed `mix hex.search` makes those docs version-aware, which matters more than it sounds — models trained on stale docs are wrong in confident, hard-to-detect ways.

**Stability means training data ages well.** Elixir 1.0 shipped in 2014. It is still on 1.x. Phoenix is on 1.8; Ecto is on 3. The community treats deprecation warnings as the upgrade path, not breaking changes. Compare against any JavaScript framework you can name. Every blog post and tutorial about Elixir from the last decade is still mostly correct, which means the training data isn't polluted with contradictions between v2 and v15 advice for the same library. A model trained on "how to do X in Elixir" has one answer to learn, not five.

**Tooling closes the agent feedback loop.** Compiled language, parallel compilation and tests, type *inference* (not full annotation) that catches the usual class of bugs without forcing ceremony, warnings rather than errors so iteration isn't blocked, and — critically — runtime introspection. You can inspect a live BEAM process's state, mailbox, and ancestry from a shell or programmatically. Tidewave's MCP server exposes that introspection to coding agents directly. The runtime is *legible* to an AI in a way Python or Node aren't, because it was built legible for humans first.

That last property is the one nobody else is replicating.

## Why this compounds

Pretend for a second the benchmark is right and accept the explanation. What does it mean for a team picking a stack in 2026?

It means a tax that didn't used to exist now does. Every language choice already had a hiring cost, a library cost, a hosting cost. Now it has an *agent-effectiveness* cost, and that cost is not flat across stacks.

If a model is eight points better at completing Elixir tasks than the runner-up — and your engineers now spend a meaningful chunk of their day reviewing model output, asking for changes, and watching agents do work autonomously — that delta walks straight into your team's throughput. It compounds because every PR your humans don't have to babysit becomes time spent on the next one. It compounds *again* because the AI getting things right means less context-switching back into "I'd better write this from scratch, the agent's confused." Friction tax dropped twice.

This is the angle that's mostly missing from "which language should we use" debates. People still talk about ergonomics and ecosystems and hiring. Those matter. They no longer fully describe the cost function. The boring, stable, well-documented language that *AI models reason about cleanly* is now strictly more valuable than it used to be, and the trendy framework with twelve breaking changes a year is strictly more expensive.

The kind of code Elixir produces — small, isolated, well-typed by signature, documented in a format that's part of the language — is what AI is good at:

`defmodule Inventory do
 @moduledoc "Stock-level operations for product SKUs."

 @doc """
 Decrements stock for a SKU by `qty`. Returns `{:ok, new_stock}` or
 `{:error, :insufficient_stock}` when the request would go negative.

 ## Examples

 iex> Inventory.decrement(%{"WIDGET-1" => 5}, "WIDGET-1", 2)
 {:ok, 3}

 iex> Inventory.decrement(%{"WIDGET-1" => 1}, "WIDGET-1", 5)
 {:error, :insufficient_stock}
 """
 def decrement(stock, sku, qty) when qty > 0 do
 case Map.get(stock, sku, 0) do
 current when current >= qty -> {:ok, current - qty}
 _ -> {:error, :insufficient_stock}
 end
 end
end
```

Ten lines of executable spec — the docstring *is* the test, the signature is the contract, the function is pure, the failure case has a tagged tuple instead of an exception. A model reading this sees a complete, verifiable unit. There is nothing offstage. That is the shape of Elixir at rest, and the shape AI handles best.

## The honest costs

If Elixir-for-AI were free, every team would already be on it. It isn't. We ship Elixir in production for clients, and the costs are real.

**The hiring pool is still smaller.** This was the leading objection in 2018 and it's still the leading objection in 2026. You will interview fewer senior Elixir engineers than senior Go or Node engineers, full stop. The mitigation — "strong devs pick it up in two weeks" — is true and we've watched it work, but you cannot run a hiring strategy on it if you need three engineers next month.

**Ecosystem corners still gap.** Phoenix, LiveView, and Ecto are excellent. Outside that core: certain cloud SDKs are thinner than the Python or Go equivalents, niche protocol clients sometimes don't exist, and you'll occasionally write a NIF or shell out to another runtime. None of that is fatal; all of it is work you wouldn't have on a mainstream stack.

**The BEAM isn't a number-crunching runtime.** This is the one the AI conversation specifically muddles. Elixir is *excellent* for orchestrating AI workloads — calling models, streaming tokens to a LiveView UI, supervising long-running inference jobs, fanning out across providers. It is *not* where you run your matrix math. Nx` and `EXLA` give you a real answer for numerical work via XLA, and they're impressive, but they're an escape hatch into compiled native code — not a claim that pure Elixir is fast at tensor ops. Get the framing right or your benchmarks will lie to you.

**Erlang-isms leak through.** Stack traces drop into Erlang term syntax. Docs split across Elixir *and* Erlang/OTP. Eventually you read Erlang source to understand a library. That tax is paid by every engineer, forever — not just at onboarding.

## Verdict

The Tencent number is real, the Dashbit explanation is the right one, and the strategic implication is bigger than the benchmark headline suggests. You are not choosing a language for humans anymore. You are choosing a language for humans *and* the increasingly autonomous tools they work alongside. The properties that have always made Elixir pleasant — immutability, doc culture, version stability, runtime introspection — turn out to be the exact properties that make AI good at it, too. That is not a coincidence; it is a design philosophy paying off twice.

Pick Elixir for the reasons you would have picked it anyway: a stateful backend where things must fail independently, real-time UI without three layers of glue, a team that wants to ship boring code that ages well. Then notice that your AI assistants are also measurably better at it. Then pay the hiring tax with your eyes open, keep it away from your matrix math, and let it compound.

*Source: ["Why Elixir is the best language for AI" — Dashbit](https://dashbit.co/blog/why-elixir-best-language-for-ai).*

---

## Elixir's BEAM Is the Runtime AI Agents Want

URL: https://sublimecoding.com/blog/elixir-beam-ai-agents
Published: 2026-05-31
Tags: Elixir, agents, AI

>
**TL;DR:** The AI-agent language argument has two camps
and both are right about the wrong layer. Go won the *transport*
— the MCP server, the daemon, the thing that ships as one binary. Python
owns the *reasoning* — the prompts, the evals, the model glue.
But there’s a third layer nobody is fighting over, and it’s the one that
actually hurts in production: keeping thousands of long-lived, stateful
agents alive while they crash, retry, and hold conversation state for
hours. That’s not a daemon problem and it’s not a prompt problem. It’s a
*supervised-process* problem, and the BEAM — Erlang’s virtual
machine, the one Elixir runs on — has been the best tool on earth for it
since before the word “agent” meant this. Here’s the honest version,
including where the BEAM is the wrong call.

## The layer nobody named

A while back I [pulled apart
why every AI agent framework is written in Go](https://sublimecoding.com/blog/go-ai-agent-frameworks). The argument held up:
an agent *tool* — an MCP server, a CLI, an orchestrator — is
mechanically a concurrent network daemon that has to ship as one file
into someone else’s machine, and Go is unreasonably good at exactly
that. The reasoning content stays in Python. The conclusion was a split:
**Go owns the daemon, Python/TS owns the reasoning, talk over a
wire between them.**

That post named two layers and quietly walked past a third. I even
flagged the crack at the time — that a panic in one goroutine “is not
isolated the way people assume,” and pointed at the [failure mode I went deep on in
Elixir’s concurrency model](https://sublimecoding.com/blog/elixir-concurrency-model). That crack is the whole subject of this
post.

Because here is the thing an MCP server is *not*: it is not
the agent. The MCP server is plumbing — it answers
`tools/call` and goes back to sleep. The agent is the thing
on the *other* side of the model call: the long-lived entity that
holds a conversation, remembers what it was doing, calls six flaky tools
in sequence, gets a malformed response from one of them, and has to
either recover or die without taking its eleven thousand siblings down
with it. That entity isn’t a daemon and isn’t a prompt. It’s a
*process with a lifecycle*, and which language’s runtime owns
that layer is a question both camps skipped.

## What an agent actually is, mechanically

Strip the word “AI” off an agent the same way we stripped it off the
tool, and describe what’s left to a backend engineer.

It’s an entity that starts up when a user begins a session. It holds
state — the conversation, the scratchpad, the half-finished plan — for
the *entire* duration of that session, which might be seconds or
might be hours. While it’s alive it makes a sequence of unreliable
calls: model APIs that rate-limit and time out, tools that throw,
subprocesses that hang. Some fraction of those calls will fail in ways
you did not anticipate, because the failure is coming from a stochastic
model deciding to emit malformed JSON, or a third-party API having a bad
afternoon. When one of those failures happens, the *correct*
behavior is almost never “take down the server.” It’s “this one agent’s
current step failed; retry it, reset it to its last good state, or let
this one agent die — and leave the other ten thousand completely
untouched.”

Now describe *that* to someone who wrote telecom software in
the nineties and they will name the primitive before you finish the
sentence: it’s a supervised process. One process per agent, holding its
own isolated state, linked to a supervisor that knows what to do when it
dies. You did not invent a new architecture for AI. You rediscovered
OTP.

This is the tell. The shape of “many independent, stateful,
long-lived, failure-prone things that must be isolated from each other”
is not new and was never about AI. It’s the shape of phone calls, of
chat sessions, of multiplayer game state — the workloads the BEAM was
*purpose-built* for. Agents just happen to have that exact shape,
and most teams are discovering it the hard way by reimplementing
supervision badly in a runtime that doesn’t have it.

## Why the BEAM fits the shape

Four properties, each of which is a direct answer to a problem the
agent workload creates. None of them is an AI feature — that’s the
point, the same way Go didn’t win the daemon by being good at AI.

**Real process isolation, not the goroutine kind.** A
BEAM process has its own heap, its own stack, and its own garbage
collector. Nothing is shared by default; processes communicate only by
copying messages. The practical consequence is the one that matters for
agents: when a process crashes, the blast radius is *that
process*. Its memory is reclaimed, its siblings never notice.
Contrast the goroutine, which I was careful about in the [Go post](https://sublimecoding.com/blog/go-ai-agent-frameworks): goroutines share an
address space and an unrecovered panic in one goroutine takes down the
entire OS process — every other in-flight agent with it. You can paper
over that with `recover()` at every boundary, but you are
hand-rolling, imperfectly, the isolation the BEAM gives you for free.
For a system whose defining characteristic is “individual units fail
constantly and unpredictably,” shared-fate concurrency is the wrong
default and isolated-fate concurrency is the right one.

**Supervision trees turn “let it crash” into a retry
strategy.** OTP’s supervision model — a tree of supervisor
processes whose only job is to start, monitor, and restart their
children according to a declared policy — came out of Ericsson’s work on
systems that were not allowed to go down. Joe Armstrong’s 2003 thesis
was literally titled *Making reliable distributed systems in the
presence of software errors*, and the AXD301 switch built on these
ideas famously reached availability figures quoted in the nine-nines
range. The philosophy is “let it crash”: don’t litter defensive
`try/catch` through your business logic trying to anticipate
every failure; let the process die cleanly and let a supervisor restart
it from a known-good state. Read that sentence again with an agent in
mind. An agent step that fails on a bad model response *should*
crash and restart from its last checkpoint — that’s not a workaround,
it’s the designed-for case. The retry-with-backoff, reset-to-checkpoint,
escalate-after-N-failures logic that agent frameworks in other languages
write by hand is a `Supervisor` strategy you *declare*
on the BEAM.

**One process per agent is genuinely free.** The
objection to “a process per agent” in most runtimes is cost — OS threads
are expensive, so you pool and multiplex and lose the isolation. On the
BEAM the objection evaporates. A freshly spawned process starts at
roughly 2–3 KB of memory and is created in microseconds; a single node
sustains hundreds of thousands to millions of concurrent processes
without breaking a sweat. So you don’t pool agents onto shared workers
and reintroduce shared fate. You give every agent — every conversation,
every sub-task, every tool invocation if you want — its own process,
with its own state and its own crash domain, and you do it by the
hundred thousand. The architecture you’d *want* on paper (total
isolation) is also the cheap one, which is not a tradeoff you usually
get to make.

**Preemptive scheduling means one wedged agent can’t starve the
rest.** The BEAM scheduler is preemptive: it counts reductions
(roughly, work units) and forcibly yields a process after a small
budget, so no single process can monopolize a scheduler thread. For
agents this matters more than it first looks. A long-running agent that
does something CPU-heavy — parsing a huge document, a tight retry loop,
a runaway tool — cannot wedge the runtime and freeze every other agent’s
progress. Go’s scheduler is cooperative at the edges and a tight CPU
loop can misbehave; Python’s GIL serializes CPU-bound work outright
(free-threading is coming, but it isn’t the world most agent code runs
in today). The BEAM’s “everyone gets a fair slice, always” is the
property you want when you’re running a noisy crowd of independent
agents of wildly varying behavior on one box.

Put those four together and you have a runtime whose native unit
*is* the long-lived, isolated, supervised, fairly-scheduled
stateful process. That is the agent, described exactly.

## It’s not vaporware

The fair pushback is “great theory, but is anyone actually building
agents on this, or is it a forum argument?” The ecosystem is real and
getting realer fast — I watched the [SERPs fill with it](https://sublimecoding.com/blog/elixir-ai-codes-best) while
researching this.

[**Jido**](https://github.com/agentjido/jido)
is an OTP-native autonomous-agent framework: agents are supervised
processes with an immutable functional state model, and the AI layer is
optional — the core gives you the agent *architecture* (planning,
actions, lifecycle) and you bolt the LLM on. That factoring is the whole
thesis of this post shipped as a library.

**LangChain for Elixir** — the `langchain`
Hex package maintained by Mark Ericksen at Fly.io — is the pragmatic
model-integration layer: a clean client for OpenAI, Anthropic, and the
rest, with tool-calling and structured output, so the “reasoning” wire
from the Go post terminates somewhere sane in Elixir. And because
Phoenix is right there, streaming a model’s tokens to a live UI is not a
separate websocket stack you stand up — [LiveView already holds the
connection](https://sublimecoding.com/blog/elixir-concurrency-model), and async assigns push tokens as they arrive. The
“simplest real-time AI UI” is close to free when the agent and the UI
live in the same supervised tree. Which structured-output library to reach for, and why the obvious one is not the one shipping, is [its own comparison](https://sublimecoding.com/blog/structured-llm-output-elixir).

I’ll keep the scope honest: this ecosystem is younger and smaller
than Python’s, and you will occasionally be the first person to hit a
rough edge. But the *primitives* — processes, supervisors,
message passing — are thirty years mature. The agent libraries are thin,
sane layers over a deep foundation, which is the opposite of the usual
situation where a slick library hides a shaky core.

## What it costs you

Here’s the part the title doesn’t say, the same way the Go post owed
you the costs of choosing Go. The BEAM is weak in exactly the place
Python is strong, and pretending otherwise is how you talk yourself into
a bad architecture.

**The model and ML layer is not the BEAM’s.** If your
agent needs to run inference *locally* — embeddings, a local
model, real tensor math — you are swimming against the current. [Nx](https://github.com/elixir-nx/nx) and [Bumblebee](https://github.com/elixir-nx/bumblebee) exist and
are genuinely impressive work, and you can run Whisper or a Llama-class
model from Elixir today. But the frontier of models, the day-one SDKs,
the research code, the eval tooling, the sheer gravity of the ecosystem
— that’s all Python, and it will be for years. If the *center of
mass* of your system is the model itself rather than the
orchestration of agents around it, you are buying the wrong runtime to
save the wrong cost.

**The talent pool is smaller.** You will hire Elixir
engineers more slowly than Python or Go engineers, full stop. For a lot
of teams that single fact outweighs every architectural elegance in this
post, and it should — the [most
pragmatic stack is frequently the boring one you can staff](https://sublimecoding.com/blog/ruby-boring-ships). Be
honest with yourself about whether you’re optimizing for the system’s
properties or for your own enjoyment of them.

**You’re calling out for the model anyway.** In the
overwhelmingly common case, your “model call” is an HTTPS request to a
hosted API. That’s true in every language, which means the BEAM’s
weakness at *local* inference is irrelevant to most production
agents — but it also means the model layer isn’t where your language
choice pays off, so don’t let “but Python has the SDKs” decide a system
whose hard problem is supervising ten thousand stateful sessions, not
calling an API.

Notice the costs all cluster in the same place — the model/reasoning
layer — exactly as Go’s costs all clustered in *its* reasoning
layer. The BEAM’s weaknesses are Python’s strengths. Which is the entire
point of the next section.

## The split that completes the trilogy

The Go post ended with a two-way split. With the third layer named,
it’s a clean three-way one, and the seams fall in obvious places:

Layer
What it is
Right tool
Why

**Transport / daemon**
MCP server, CLI, the thing users install
**Go**
One binary, near-zero deps, cheap concurrency — its home turf

**Reasoning / model**
Prompts, evals, inference, ML glue
**Python / hosted API**
REPL loop, the SDKs, the entire ML ecosystem

**Stateful supervision**
Long-lived agents, session state, crash recovery, fan-out
**BEAM (Elixir)**
Isolated supervised processes are the *native unit*

The decision rule, not the language-war version: pick your runtime by
**where your system’s hard problem actually lives.** If the
hard problem is “ship a tool into a thousand machines,” that’s Go. If
it’s “iterate on prompts and run a model,” that’s Python. If it’s “keep
an enormous number of independent, stateful, failure-prone agents alive
and isolated for hours,” that’s the BEAM, and it’s not close. Most real
systems are more than one of these, and the mature answer is the same as
it was for daemon-versus-reasoning: don’t force one language across a
seam it doesn’t belong on. Let the BEAM supervise, let Python think, let
Go ship the binaries, and put wires between them.

## The part everyone’s been arguing past

The whole “what language for AI agents” debate has been a fight about
the two visible layers — the binary you install and the prompt you tune.
Both matter and both have clear winners. But the layer that actually
decides whether your agent platform survives contact with production
isn’t either of those. It’s what happens at 3 a.m. when four hundred
agents are mid-task and the model API starts returning garbage: do four
hundred sessions die, or does one runtime quietly crash-and-restart each
failed step from its last checkpoint while everything else keeps
running?

That problem — many isolated stateful things failing independently
and recovering without a global blast radius — was solved, productized,
and battle-hardened for telephone switches before most of us were
writing code. The agent era didn’t create a new hard problem at the
orchestration layer. It walked straight into an old one that already has
a famously good answer. The only surprising thing is how few people
building agents have noticed that the runtime they want already exists,
has for decades, and is sitting one `mix new` away.

Use the BEAM for the thing the BEAM is for. Let it hold the agents.
Just don’t ask it to run your model — keep that where it belongs, on the
other side of a wire, in the language built for it.

---

## Elixir's Concurrency Model Is the One You Actually Want

URL: https://sublimecoding.com/blog/elixir-concurrency-model
Published: 2026-05-16
Tags: elixir, otp, engineering

>
**TL;DR:** `async/await` and goroutines solve *scheduling* — how to interleave a lot of work on a few OS threads without blocking. The BEAM solves *failure* — what happens when one of those units of work blows up at 3am. Most of the "concurrency" pain backend devs feel is actually failure-isolation pain wearing a concurrency costume, and only the actor model plus supervision trees address it head-on. Elixir's model is the one you actually want; it's also the one with the smaller hiring pool, the Erlang-isms, and no business doing your matrix multiplication. Here's the honest version, with a working GenServer and Supervisor you can paste into a fresh `mix` project.

## The 3am page async/await can't prevent

Here's a bug I've shipped, in some form, in three different languages.

A request handler does five things. It validates input, hits the database, calls a third-party API, transforms the result, and writes to a cache. The third-party API starts returning a malformed payload — not an HTTP error, just JSON with a field that's now `null` where it used to be a string. Your transform step does `payload.token.toUpperCase()`. It throws. The throw is unhandled in that code path because you wrote the happy path first and the deadline was Friday.

In Node, depending on where that ran, you either crash the process or — worse — you reject a promise nobody's awaiting and the runtime prints `UnhandledPromiseRejection` and, in modern Node, exits anyway. In Python with `asyncio`, an exception in a task that nobody `await`s gets logged when the task is garbage-collected and silently swallowed until then. In Go, if that transform ran in a bare `go func()` with no `recover()`, the panic walks up that goroutine's stack and takes the entire process with it. One bad upstream payload, one unguarded line, whole service down. The pager goes off at 3am.

Notice what the bug *isn't*. It isn't a scheduling problem. `async/await` scheduled that work perfectly. Goroutines would have scheduled it perfectly. The event loop did its job. The bug is a *failure-isolation* problem: there was no boundary between "this one request's transform step exploded" and "the process serving every other request is now dead."

This is the thing I want to convince you of: the concurrency model you reach for should be judged less on how elegantly it schedules work and more on what it does when one unit of that work fails. By that measure, the models most backend devs use every day are weak, and the one they keep getting told to try — "just use Elixir" — is strong for reasons nobody bothers to explain past the slogan.

Let me explain past the slogan.

## The three models, honestly

Strip the marketing off and there are three concurrency models a backend dev is likely to touch. They are not competing implementations of the same idea. They guarantee different things.

**Threads and async (Node, Python, the JVM's default style).** You have one or a few OS threads. You multiplex many logical tasks onto them using an event loop (`libuv`, `asyncio`) or a thread pool. Tasks share the same heap. The model's core guarantee is *throughput*: you can have ten thousand in-flight requests without ten thousand OS threads. What it does *not* give you is isolation. Every task lives in the same memory space and, in single-threaded runtimes, the same failure domain. An unhandled exception's blast radius is "whatever shares this thread/process," which in practice is everything. You bolt safety on afterward with `try/catch` discipline, framework-level error middleware, and a process supervisor like `pm2` or systemd restarting the whole thing.

**Goroutines and channels (Go).** Go gives you cheap green-threaded units (goroutines) multiplexed by the runtime onto OS threads, plus channels for communicating between them. This is genuinely better ergonomics than callback or async-coloring soup — goroutines don't have a "color," and `go foo()` is about as low-friction as concurrency syntax gets. The guarantee is *cheap concurrency with first-class communication*. What Go does not give you is memory isolation or failure isolation. Goroutines share the same address space by design; the language even documents the data-race rules you must follow because shared memory is the default substrate. And an unrecovered panic is not goroutine-local — it terminates the program.

**Processes and supervision (the BEAM: Erlang, Elixir).** The BEAM gives you *processes* — not OS processes, not threads, but VM-level units that are extraordinarily cheap and, critically, **share no memory**. The Erlang docs are blunt about it: "Threads of execution in Erlang share no data, that is why they are called processes." ([erlang.org, Concurrent Programming](https://www.erlang.org/doc/system/conc_prog.html)) They communicate only by copying messages between isolated mailboxes. The guarantee here is different in kind: *isolation plus a structured story for failure*. A process can crash without touching any other process's memory, because there is no shared memory to touch. And the platform ships a first-class abstraction — supervision trees — whose entire job is deciding what to do when one does crash.

The first two models optimize "run lots of things without blocking." The third optimizes "contain the damage when one of those things dies." Those are different problems, and most production incidents are the second one.

## Where Go's model leaks

I'm picking on Go specifically because Go is the language people most often reach for when they've outgrown Node/Python concurrency and *think* they've solved the problem. Go's model is good. It is not the same kind of good.

**Shared memory is the default, not the exception.** The Go proverb is "Do not communicate by sharing memory; instead, share memory by communicating" ([go.dev, Share Memory By Communicating](https://go.dev/blog/codelab-share)). It's good advice precisely because the default is the opposite. Channels are opt-in; the shared heap is opt-out. The Go memory model exists to tell you the rules for the unsafe thing you can do by accident: it defines a data race as "a write to a memory location happening concurrently with another read or write to that same location" and is explicit that races are errors — its own summary of the philosophy is "Don't be clever." ([go.dev, The Go Memory Model](https://go.dev/ref/mem)) A model whose spec needs a "don't be clever" section is a model where the foot-gun is loaded by default.

**A panic in one goroutine takes down all of them.** This is the big one and it surprises people coming from "concurrency means isolation." It does not, in Go. From the official Go blog: when a panic isn't recovered, "the process continues up the stack until all functions in the current goroutine have returned, at which point the program crashes." ([go.dev, Defer, Panic, and Recover](https://go.dev/blog/defer-panic-and-recover)) And `recover` only works from a deferred function *on the same goroutine that panicked* — a sibling goroutine cannot catch it for you. So the moment you write `go handleRequest(conn)` and `handleRequest` panics on a nil dereference from that malformed upstream payload, every other in-flight request in that process dies with it. The mitigation is real but it's manual: you wrap every goroutine entry point in `defer func(){ recover() }()`. Forget one — in a library, in a callback, in code a teammate wrote on a Friday — and you're back to whole-process death. Isolation that depends on every author remembering a boilerplate incantation is not isolation; it's a convention.

**Channel deadlocks are a class of bug, not an edge case.** Unbuffered channel sends block until there's a receiver. A goroutine waiting to send on a channel nobody will ever receive from is stuck forever, holding whatever it holds. If *every* goroutine ends up blocked this way the Go runtime can detect total deadlock and crash with `fatal error: all goroutines are asleep - deadlock!` — but the far more common production case is a *partial* deadlock: a few goroutines wedged on channel operations while the rest of the program runs fine, leaking a little memory and one request's worth of progress every time it happens, invisible until you're staring at a slowly climbing goroutine count in production.

None of this makes Go bad. Go's model is a massive upgrade over callback-era Node for the *scheduling* problem. It just doesn't solve the *failure-isolation* problem, and it's frequently sold as if it does.

## The BEAM bet

The BEAM makes a specific, opinionated bet: optimize the runtime for *isolated failure and structured recovery*, and accept the costs that come with it.

**Processes are cheap and isolated.** A freshly spawned BEAM process is small. The Erlang efficiency guide states the default initial heap is 233 words and notes this is "quite conservative to support Erlang systems with hundreds of thousands or even millions of processes"; the same guide's worked example shows a newly spawned process at 327 words total ([erlang.org, Processes](https://www.erlang.org/doc/system/eff_guide_processes.html)). A word is 8 bytes on a 64-bit VM, so we're talking single-digit kilobytes per process, growing on demand. The point isn't a brag number — it's that "spawn a dedicated process per request, per connection, per job" is a normal, expected thing to do, not a resource gamble. Each one has its own heap. A crash in one cannot corrupt another's state because there is no shared state to corrupt.

**Preemptive scheduling, so one process can't starve the rest.** Go's scheduler is good but its preemption story has historically had rough edges. The BEAM is preemptive by reduction counting: a process is given a fixed budget of "reductions" (roughly, function calls) and yielded when it's spent them. The budget is the `CONTEXT_REDS` constant — 4000 reductions — defined in the VM's `erl_vm.h` and documented in *The BEAM Book* ([happi/theBeamBook, scheduling](https://github.com/happi/theBeamBook/blob/master/chapters/scheduling.asciidoc)). Practically: one process running a tight loop cannot freeze the others. The scheduler will pull the rug at 4000 reductions whether the code cooperates or not. That property is *why* one slow request doesn't degrade the latency of the other ten thousand.

**"Let it crash" and supervision trees.** This is the philosophical core, and it's the opposite of defensive programming. Instead of wrapping every operation in error handling to keep a process limping along in a corrupted state, you let the process *die* cleanly at the first sign that its assumptions are violated — and you put a supervisor above it whose job is to restart it from a known-good initial state. The OTP design principles describe this directly: "The supervision tree is a hierarchical arrangement of code into supervisors and workers, which makes it possible to design and program fault-tolerant software." ([erlang.org, OTP Design Principles](https://www.erlang.org/doc/system/design_principles.html)) A supervisor "is responsible for starting, stopping, and monitoring its child processes," with restart strategies (`:one_for_one`, `:one_for_all`, `:rest_for_one`) that declare exactly how a sibling's death affects the others ([erlang.org, Supervisor Principles](https://www.erlang.org/doc/system/sup_princ.html)).

Here's the thing that's hard to convey until you've run it: in this model, the 3am bug from the opening *is not a page*. The request process handling the malformed payload crashes. Its supervisor restarts a fresh worker. Every other request is untouched because it was a different process with a different heap. You get a log entry and a metric, not an outage. The failure didn't have to be anticipated, caught, and handled at the call site. It had to be *contained*, and containment is the runtime's job, not yours.

Here is a real, working example — a rate-limiter `GenServer` supervised by a `Supervisor`. Paste it into `lib/` of a fresh `mix new demo` project and it runs.

`defmodule Demo.RateLimiter do
 @moduledoc """
 A token-bucket rate limiter as an isolated process.
 If its state ever becomes inconsistent it is allowed to crash;
 the supervisor restarts it from a clean bucket.
 """
 use GenServer

 # --- Client API ---

 def start_link(opts) do
 name = Keyword.get(opts, :name, __MODULE__)
 GenServer.start_link(__MODULE__, opts, name: name)
 end

 @doc "Returns :ok if a token was available, :rate_limited otherwise."
 def request(server \\ __MODULE__) do
 GenServer.call(server, :request)
 end

 # --- Server callbacks ---

 @impl true
 def init(opts) do
 max = Keyword.get(opts, :max_tokens, 5)
 refill_ms = Keyword.get(opts, :refill_ms, 1_000)
 :timer.send_interval(refill_ms, :refill)
 {:ok, %{tokens: max, max: max}}
 end

 @impl true
 def handle_call(:request, _from, %{tokens: tokens} = state) when tokens > 0 do
 {:reply, :ok, %{state | tokens: tokens - 1}}
 end

 @impl true
 def handle_call(:request, _from, state) do
 {:reply, :rate_limited, state}
 end

 @impl true
 def handle_info(:refill, %{max: max} = state) do
 {:noreply, %{state | tokens: max}}
 end
end

defmodule Demo.Application do
 @moduledoc false
 use Application

 @impl true
 def start(_type, _args) do
 children = [
 # If RateLimiter crashes, ONLY RateLimiter is restarted.
 {Demo.RateLimiter, name: Demo.RateLimiter, max_tokens: 5, refill_ms: 1_000}
 ]

 opts = [strategy: :one_for_one, name: Demo.Supervisor]
 Supervisor.start_link(children, opts)
 end
end
```

Wire Demo.Application` into `mix.exs` with `mod: {Demo.Application, []}`, run `iex -S mix`, and try it:

`iex> Demo.RateLimiter.request()
:ok
# ...call it past the bucket size...
iex> Demo.RateLimiter.request()
:rate_limited

# Now kill it on purpose and watch the supervisor heal it:
iex> Process.exit(Process.whereis(Demo.RateLimiter), :kill)
true
iex> Demo.RateLimiter.request()
:ok # a brand-new process, fresh full bucket, no manual restart
```

That Process.exit(..., :kill)` is the whole argument in three lines. You deliberately destroyed the process. You did not write a single line of recovery code. The `:one_for_one` supervisor noticed the child died and started a clean replacement, and the very next call succeeds against fresh state. `GenServer` is, in the official Elixir docs' words, "a behaviour module for implementing the server of a client-server relation" that plugs directly into supervision and standard error reporting ([hexdocs.pm, GenServer](https://hexdocs.pm/elixir/GenServer.html)). You write the state transitions; OTP writes the resilience.

That is the bet: a little ceremony (`GenServer` callbacks, child specs, supervision strategy) bought up front, in exchange for failure isolation being a *structural property of the system* rather than a discipline every author must remember.

## The honest costs

If the BEAM model were free, everyone would use it. It isn't. A flagship post that doesn't say this is a brochure.

**The hiring pool is genuinely smaller.** This is the real one, and no amount of "but it's easy to learn" hand-waving makes it go away. You will interview fewer Elixir engineers than Go or Node engineers, full stop. You can mitigate it — the language is approachable and strong devs pick it up fast — but if your hiring strategy depends on a deep local market of people who already know the stack, that's a strike against, and pretending otherwise is dishonest.

**Ecosystem gaps in specific corners.** The web story (Phoenix, LiveView, Ecto) is excellent and competitive with anything. Outside that, you will hit libraries that are thinner than the Go or Python equivalent: certain cloud-vendor SDKs, some ML/data tooling, niche protocol clients. The usual escape hatch is a port/NIF or shelling out to another runtime, which is fine but it's work you wouldn't have on a more mainstream stack.

**Erlang-isms leak through.** Elixir is a lovely language, but it sits on a 1980s telecom VM, and the substrate shows. Stack traces drop into Erlang term syntax. Tooling and observability docs are split across Elixir *and* Erlang/OTP. You will, eventually, read Erlang source to understand a library. That's a tax on every engineer, paid forever, not just at onboarding.

**It is not a number-crunching runtime.** The BEAM is optimized for massive concurrency and message passing, not raw CPU throughput on tight numeric loops. Heavy computation — image processing, large-matrix math, cryptographic grinding — is not what it's for, and naively doing it in pure Elixir is slow. The community answer is real (NIFs, `Nx`/`EXLA` for numerical work, offloading to native code), but the honest framing is: the runtime's strength is concurrency-and-failure, and you pay for that focus in compute-bound work.

## When NOT to reach for Elixir

Decisions are made by knowing when *not* to use the thing.

- **CPU-bound batch work with little concurrency.** A nightly job that does heavy math on one big dataset has no failure-isolation problem worth solving and will run faster in a runtime built for throughput. Wrong tool.

- **Small teams on a tight deadline who already know Go/Node/Python well.** The right concurrency model in your hands beats the better one you're learning under deadline pressure. Familiarity is a real engineering input. Ship the thing.

- **You genuinely don't have a failure-isolation problem.** A mostly-stateless service that fans out a few HTTP calls behind a load balancer that already restarts unhealthy instances has externalized the supervision problem to your orchestrator. The BEAM's biggest advantage is partly redundant there. It's still nice; it's not decisive.

- **Hard real-time or microsecond-latency systems.** The BEAM's preemptive, garbage-collected scheduling is built for soft real-time (consistent low-ish latency under massive concurrency), not hard guarantees. If you need bounded microsecond worst-case, this is not your platform.

The pattern: reach for the BEAM when your dominant pain is *many independent things that can each fail independently and must not take each other down*. That's chat backends, telephony, IoT fleets, payment orchestration, real-time multiplayer, anything stateful-per-connection. When that's not your shape, the model's signature advantage is muted, and its costs are still fully priced in. (Network config automation is that shape too — [Cisco's NSO quietly runs its core on an Erlang VM](https://sublimecoding.com/blog/cisco-nso-secretly-erlang).)

## Verdict

The slogan "just use Elixir" is right for the wrong stated reason. People say it like it's about concurrency throughput. It isn't — Go and well-written async Node handle enormous concurrency. It's about what the runtime does at 3am when one unit of that concurrency hits a bug nobody anticipated. `async/await` and goroutines schedule beautifully and then make that failure your problem, manually, forever. The BEAM makes it the runtime's problem, structurally, by default.

Here's the decision table I actually use:

Your dominant pain
Best-fit model
Why

"I have lots of I/O-bound requests and don't want a thread per request"
Threads/async (Node, Python, JVM)
Scheduling is the whole problem; isolation isn't your bottleneck

"I want cheap concurrency with clean communication and a big hiring pool"
Goroutines + channels (Go)
Excellent scheduling + ergonomics; accept manual failure isolation

"Independent stateful things that must fail without taking each other down"
Processes + supervision (Elixir/BEAM)
Failure isolation is structural, not a convention you must remember

"Heavy CPU-bound numeric work, low concurrency"
A throughput runtime (Go, Rust, native)
None of the above models' isolation buys you anything here

"Hard real-time, bounded worst-case latency"
A real-time runtime, not the BEAM
GC + preemptive scheduling is soft real-time only

If your honest answer to "what's my dominant pain" is the third row — and for a surprising number of stateful backends it is, people just file it under "concurrency" because that's the word they have — then yes. It's the model you actually want. Pay the hiring and Erlang-ism costs with your eyes open, keep it away from your matrix math, and let it crash.

---

## Why Every AI Agent Framework Is Written in Go (And What That Costs You)

URL: https://sublimecoding.com/blog/go-ai-agent-frameworks
Published: 2026-05-16
Tags: go, agents, engineering

>
**TL;DR:** Open the repos behind the agent tooling you actually run — Ollama, the MCP SDKs, the orchestration engines — and you keep landing on the same language: Go. Not because Go is good at AI; it isn't, particularly. Because the *thing* an agent tool actually is — a long-lived concurrent network daemon that has to ship as one binary your users can run without a runtime — is exactly what Go was built for. That's a great trade for the daemon and a bad one for the reasoning layer, which is why the ecosystem is quietly splitting in two: Go for the process, Python/TS for the prompt logic. Here's the honest version, with a stdlib-only MCP server you can compile to a single file.

## The roll-call

Go look at what you're running.

If you've followed along with the [Claude Code Resource Bible](https://sublimecoding.com/blog/claude-code-resource-bible), you've already installed half of this list without noticing the pattern. Ollama, the thing you `brew install` to run a local model, is Go — a single binary, no Python environment to break ([github.com/ollama/ollama](https://github.com/ollama/ollama)). The Model Context Protocol's official Go SDK is Go, maintained in collaboration with Google ([github.com/modelcontextprotocol/go-sdk](https://github.com/modelcontextprotocol/go-sdk)). The most-used community MCP library, `mark3labs/mcp-go`, is Go ([github.com/mark3labs/mcp-go](https://github.com/mark3labs/mcp-go)). CloudWeGo's Eino — billed flatly as "the ultimate LLM/AI application development framework in Go" — is Go ([github.com/cloudwego/eino](https://github.com/cloudwego/eino)). LangChainGo is Go ([github.com/tmc/langchaingo](https://github.com/tmc/langchaingo)). Drop down a layer to the orchestration substrate agents run *on* and it's the same story: Temporal, the durable-execution engine a lot of agent workflows are built on, is Go ([github.com/temporalio/temporal](https://github.com/temporalio/temporal)); Dagger, the automation engine that grew agent modules, is Go ([github.com/dagger/dagger](https://github.com/dagger/dagger)); Docker and Kubernetes — the things the whole circus is shipped and scheduled in — are Go.

The reasoning *content* of all of this — the models, the research, the prompt engineering, the eval harnesses — is overwhelmingly Python. But the *machinery* that delivers it to you is overwhelmingly Go. That split is not an accident and it is not taste. It's an architectural tell, and once you see it you can't unsee it.

The title of this post is the argument. The rest of it earns the claim, and then tells you what believing it costs you — because the same properties that make Go the right call for the daemon make it an actively annoying place to write the part of an agent that thinks.

## Why Go fits the shape of the problem

Start from what an agent tool actually *is*, mechanically, once you strip the word "AI" off it.

It's a process that starts up, holds open some connections (stdio to a host, a socket, an HTTP listener), waits for requests, fans each one out to a few concurrent things — a model call, a tool call, a file read, a subprocess — collects the results, and writes a response. It does that for a long time without being restarted. It needs to be installed by people who do not have, and do not want, your development environment. Describe that to a backend engineer without ever saying "LLM" and they will tell you the language: it's a network daemon, and Go is a language designed, deliberately and narrowly, to write network daemons.

**It ships as one file.** This is the single biggest reason and it has nothing to do with AI. The Go FAQ states it plainly: "The linker in the `gc` toolchain creates statically-linked binaries by default. All Go binaries therefore include the Go runtime" ([go.dev/doc/faq](https://go.dev/doc/faq)). A Go MCP server is one executable. No `pip install` that resolves differently on the user's machine than yours. No "works on my Python 3.11, your 3.13 broke a transitive dep." No asking a user to manage a virtualenv to run your tool. You cross-compile it for three platforms in CI and the install instruction is "download this file." For a tool whose entire job is to be installed into *other people's* agent setups, that property is worth more than any language feature. It's why Ollama feels like a native app and most Python AI tooling feels like a science experiment you have to host.

**Goroutines are the right concurrency primitive for fan-out.** An agent step is a fan-out: call the model, and while that's in flight maybe pre-warm a tool, read a file, hit a cache. Go's answer to "do these concurrently" is `go f()` and a channel, and the runtime cost is genuinely low. From the same FAQ: goroutines "can be very cheap: they have little overhead beyond the memory for the stack, which is just a few kilobytes," the CPU overhead "averages about three cheap instructions per function call," and "it is practical to create hundreds of thousands of goroutines in the same address space" ([go.dev/doc/faq](https://go.dev/doc/faq)). You do not need an async framework, an event-loop mental model, or function coloring. A goroutine per in-flight tool call, a `context.Context` for cancellation when the model returns early, done. (It is worth knowing where this model *stops* helping you — a panic in one goroutine is not isolated the way people assume; I went deep on exactly that failure mode in [Elixir's concurrency model](https://sublimecoding.com/blog/elixir-concurrency-model), and it applies directly to long-lived agent daemons.)

**The standard library already has the daemon parts.** `net/http` is a production HTTP server and client in the stdlib. `encoding/json` is in the stdlib. `os/exec` for spawning the tool subprocesses agents love is in the stdlib. `context` for deadline and cancellation propagation is in the stdlib. The dependency footprint of a competent MCP server in Go can be *zero third-party packages*, which means the supply-chain surface of the thing you're injecting into a user's machine is the Go team plus you. Compare that to the transitive dependency tree of an equivalent Python or Node tool. For software whose threat model includes "runs with access to a developer's filesystem and shell," a near-empty `go.mod` is not austerity; it's a security property.

**Cold start and steady-state are both cheap.** A statically linked Go binary has no interpreter to boot and no JIT to warm. It starts, it serves, its memory is roughly what its working set is. This matters for the specific way agent tools get used: spawned per-session, sometimes per-invocation, by a host process that may start and stop them constantly. A tool that takes 800ms to import its dependency tree before it can answer the first request is a tool that makes the whole agent feel slow. (Treat the comparison as directional, not a benchmark — "no interpreter boot, no JIT warmup" is an architectural fact; the millisecond figure depends entirely on your dependency tree.)

None of these four is an AI capability. That's the point. Go didn't win agent infrastructure by being good at the AI part. It won by being unreasonably good at the boring 90% of an agent tool that isn't the AI part.

## The MCP angle: proof in ~30 lines

The cleanest demonstration is the Model Context Protocol, because an MCP server is the agent-tool shape distilled to its essence: read framed JSON-RPC requests on stdin, do something, write JSON-RPC responses on stdout, live as a subprocess of the host. Here is a working line-delimited MCP-style stdio server in Go that exposes one tool, with **zero imports outside the standard library**. It compiles to one binary.

`package main

import (
	"bufio"
	"encoding/json"
	"os"
	"strings"
)

type rpc struct {
	JSONRPC string `json:"jsonrpc"`
	ID json.RawMessage `json:"id,omitempty"`
	Method string `json:"method,omitempty"`
	Params json.RawMessage `json:"params,omitempty"`
	Result any `json:"result,omitempty"`
}

func main() {
	in := bufio.NewScanner(os.Stdin)
	in.Buffer(make([]byte, 1<<20), 1<<20)
	out := json.NewEncoder(os.Stdout)

	for in.Scan() {
 var req rpc
 if json.Unmarshal(in.Bytes(), &req) != nil {
 continue
 }
 resp := rpc{JSONRPC: "2.0", ID: req.ID}
 switch req.Method {
 case "initialize":
 resp.Result = map[string]any{
 "protocolVersion": "2025-06-18",
 "capabilities": map[string]any{"tools": map[string]any{}},
 "serverInfo": map[string]any{"name": "echo", "version": "0.1.0"},
 }
 case "tools/list":
 resp.Result = map[string]any{"tools": []any{map[string]any{
 "name": "shout",
 "description": "Uppercases its input.",
 "inputSchema": map[string]any{"type": "object"},
 }}}
 case "tools/call":
 var p struct{ Arguments struct{ Text string } }
 _ = json.Unmarshal(req.Params, &p)
 resp.Result = map[string]any{"content": []any{map[string]any{
 "type": "text", "text": strings.ToUpper(p.Arguments.Text),
 }}}
 default:
 continue // notifications and unknown methods: no reply
 }
 _ = out.Encode(&resp)
	}
}
```

I am being deliberately honest about what this is: a *shape* demonstration, not a spec-complete server — a production MCP server adds Content-Length framing, proper error objects, schema validation, and the rest of the lifecycle, which is precisely what an SDK like mark3labs/mcp-go` or the official `modelcontextprotocol/go-sdk` gives you so you don't write it by hand. But look at what the 40 lines already are: a long-lived process, reading framed requests, dispatching concurrently if you wanted (`go handle(req)`), zero dependencies, one `go build` away from a binary you hand someone. That is the entire job. The reason every MCP SDK has a first-class Go implementation is that this is the language where that job is a Tuesday. The same daemon written to be spec-complete in Python is more code *and* a dependency tree *and* a runtime your user has to already have.

If you want to see what this looks like when it's not a toy — many tools, real lifecycle, orchestrated — that's essentially the architecture I pulled apart in the [ruflo / claude-flow multi-agent deep-dive](https://sublimecoding.com/blog/ruflo-claude-flow-multi-agent-deep-dive): a swarm of these daemons is still, underneath, this loop.

## What it costs you

Here is the part the title doesn't say and most "Go for AI" posts skip, because it's the part that bites you three weeks in.

**The LLM-orchestration layer is verbose and joyless in Go.** The work of an agent's *brain* — assemble a prompt from fragments, call a model, parse a structured response, branch on it, maybe retry with a tweaked prompt, thread some state through — is exactly the kind of code Go is worst at. It's data-shuffling glue, and Go's error-handling model means every one of those steps is three lines (`x, err := ...; if err != nil { return ... }`) where Python is one. A prompt pipeline that is fifteen readable lines of Python becomes sixty lines of Go where the *logic* is buried under ceremony. None of those lines are wrong. They're just noise drowning the part you actually want to iterate on.

**Generics help, but the ergonomics for prompt pipelines are still weak.** Go only got generics in 1.18 — the announcement calls it "our biggest change ever to the language," released 15 March 2022 ([go.dev/blog/go1.18](https://go.dev/blog/go1.18)). They're real and they help, but they're constrained by design: no sum types, no rich pattern matching, no ergonomic "this is one of these five structured outputs" the way you'd model an LLM's response variants in a language with proper ADTs. Modeling "the model returned either a tool call, or text, or a refusal, or a malformed blob" is clean in TypeScript's discriminated unions and clumsy in Go's type switches and interface assertions. The reasoning layer is *full* of that shape of problem.

**JSON ⇄ struct friction is constant, and LLM output is the worst case for it.** Go's `encoding/json` wants to marshal into known, typed structs. LLM output is semi-structured, frequently almost-but-not-quite the schema, and routinely needs "parse what you can, tolerate the rest." Go's typed unmarshalling fights you here: you end up reaching for `map[string]any` and type-asserting your way through a blob, which is exactly the dynamically-typed code Go is trying to prevent you from writing — except now it's verbose dynamically-typed code. Python's "it's a dict, deal with it" is genuinely better for the messy boundary where model output meets program.

**There's no official REPL, so prompt iteration has no inner loop.** Iterating on a prompt is inherently interactive: tweak wording, run it, look at the output, tweak again, ten times in two minutes. Python and a notebook are *built* for that loop. Go's edit-compile-run cycle is fast by compiled-language standards but it is not a REPL, and there is no official one — your prompt-tuning loop is "edit file, `go run`, read stdout, repeat," which is enough slower per iteration that you simply iterate less, which makes your prompts worse. This is a real product cost hiding inside a developer-experience complaint.

Notice these four costs all live in the same place: the *reasoning* layer, not the daemon. Go's weaknesses are precisely Python's strengths and vice versa, which is why mature teams stop trying to pick one.

## The split that's actually emerging

The interesting thing isn't "Go won" or "Python won." It's that the production answer is increasingly *both, with a wire between them*, and the seam is falling in a consistent place.

The pattern: **Go owns the daemon, Python/TS owns the reasoning.** The long-lived process — the MCP server, the orchestrator, the thing holding connections and fanning out concurrent tool calls and shipping as one binary — is Go, for every reason in the "why Go fits" section. The model-facing logic — prompt construction, the eval harness, anything you want to iterate on interactively, anything that benefits from the ML ecosystem — is Python or TypeScript, called *across a boundary*: a subprocess the Go daemon spawns, a local HTTP call, an MCP tool that is itself implemented in Python. The Go process doesn't do the thinking; it's the supervisor and the I/O multiplexer for the things that do. You can see this directly in the ecosystem: the *infrastructure* repos are Go, and they call out to model code rather than embedding it.

This is a better factoring than "rewrite the prompt logic in Go for consistency," which teams try once and regret. The boundary is load-bearing: it's the line between the part that has to be operationally boring (one binary, cheap concurrency, tiny dependency surface) and the part that has to be iteration-friendly (interactive, dynamically typed, ML-ecosystem-adjacent). Fighting that boundary by forcing one language across it costs you either the daemon's deployability or the reasoning layer's iteration speed.

## Verdict: when to write your agent tool in Go

The honest decision rule, not the language-war version:

Your situation
Write it in
Why

MCP server, CLI tool, orchestrator, anything users *install*
Go
One binary, near-zero deps, cheap concurrency, ops-boring — its home turf

Prompt logic, eval harness, anything you iterate on interactively
Python / TS
REPL/notebook loop, ergonomic semi-structured data, ML ecosystem

A tool that is mostly daemon with a thin model call
Go
The model call is one HTTP request; don't move the daemon for it

A tool that is mostly reasoning with a thin I/O wrapper
Python / TS
Don't pay Go's orchestration tax to save a few MB of binary

A system that is meaningfully both
Both, split at the daemon/reasoning seam
Go supervises and multiplexes; Python/TS thinks; talk over a wire

"We want one language for consistency"
Pick by the *dominant* axis above
Consistency is not a strong enough reason to eat the wrong side's tax

The reason every AI agent framework is written in Go is that "agent framework" is a misleading name. The framework part — the part that's actually a framework — is a concurrent network daemon, and Go is the best mainstream language for concurrent network daemons by a wide margin. The AI part isn't in the framework. It's in the model, and in the prompt logic you should keep in a language built for iterating on it.

Use Go for the thing Go is for. Ship the binary. Just don't let "all the infra is Go" talk you into writing your prompt pipeline there too — that's how you end up with sixty joyless lines doing fifteen lines of thinking, and a prompt you tuned three times instead of thirty because the loop was too slow to bother.

---

## Ruby Isn't Dead, It Got Boring — And Boring Is Why It Ships

URL: https://sublimecoding.com/blog/ruby-boring-ships
Published: 2026-05-16
Tags: ruby, developer-workflows, engineering

>
**TL;DR:** "Ruby is dead" is a category error. It confuses *hype velocity* — how often a language trends — with *shipping velocity* — how fast a team gets a working product in front of users and keeps it running. Those are different metrics and they are not even correlated. Meanwhile Rails 8 (November 2024) quietly deleted the three things people actually left Ruby over: the Redis dependency, the Sidekiq-style job stack, and the Node build step. The reasons to leave got removed while nobody was tweeting about it. Boring is not the absence of progress. For a team trying to launch, boring *is* the progress. Here's the honest version — including the parts Ruby still loses on.

## The obituary runs every year. The shipping never stops.

There is a genre of blog post that publishes itself annually. "Is Ruby dead?" "The decline of Rails." "Why we moved off Ruby." It has run, in some form, every year for over a decade. It is one of the most reliably recurring pieces of content in our industry, which is itself the tell: you do not need to keep re-announcing the death of something that is actually dead. Nobody writes "is COBOL dead" think-pieces, because nobody's emotionally invested in the answer. The obituary cycle is a sign of *relevance*, not decline — you only keep eulogizing the thing people are still using.

And people are still using it. The companies you interact with daily — the largest code-hosting platform on earth, one of the biggest commerce platforms in the world, the company that *invented* the framework and still ships consumer products on it — run Ruby in production at enormous scale, today, in 2026, having had every resource and every reason to rewrite if Ruby were actually the liability the obituaries claim. They didn't. That's not nostalgia. Rewrites are expensive and these are companies that count. They stayed because the thing still ships.

So there's a contradiction to resolve. If Ruby is dead, why does the death keep needing to be announced, and why do the people with the most money and the most users keep shipping on the corpse? The resolution is that "dead" is measuring the wrong thing.

## Define "dead" properly before you use it

When someone says a language is dead, ask them what metric moved. They almost always mean one of these, and they almost never mean the one that matters.

**Mindshare is not usefulness.** Most "Ruby is dead" arguments are, underneath, "Ruby doesn't *trend* anymore." Fewer conference keynotes, fewer hot takes, fewer "I rewrote my side project in it" posts. All true. Also irrelevant to whether it ships software. Mindshare measures how *interesting* a technology is to talk about, and interesting-to-talk-about peaks early — when a thing is new, unsettled, and everyone's still arguing about how to use it. A mature tool stops generating novelty content precisely *because* the arguments got settled and it just works. Falling mindshare is what success looks like from the outside. It is indistinguishable, on a graph, from death — and people keep reading the graph wrong.

**Hype velocity vs. shipping velocity.** Here's the distinction the whole obituary genre misses. *Hype velocity* is how fast a technology accumulates attention, stars, and conference talks. *Shipping velocity* is how fast a team using it gets a correct product in front of real users and keeps it up at 3am. These are different axes. They are arguably *inversely* related past a point: the technologies with the highest hype velocity are usually the youngest, which means the least settled, which means you spend shipping time relearning the stack every six months instead of building the product. Ruby has near-zero hype velocity in 2026 and that says nothing — *nothing* — about its shipping velocity, which is the only number a team trying to launch should care about.

**The metrics that actually matter.** If you want to know whether a stack is alive in the only sense that pays rent, don't count tweets. Ask: Is it maintained and releasing? (Ruby 3.x and Rails 8 — yes, actively.) Can you hire for it? (Yes, and we'll get to the honest version of that.) Can a small team get to production fast and stay there? (This is the whole rest of the post.) Are serious companies betting revenue on it *now*, not in 2014? (Yes.) Every one of those is green. The only red metric is "does Hacker News find it exciting," and that has never once shipped a feature.

This is, deliberately, the same move as [the Elixir argument](https://sublimecoding.com/blog/elixir-concurrency-model): judge the runtime by what it does at 3am, not by how it demos. Elixir is the *interesting* answer to that question. Ruby is the *boring* one. Both ship. This post is about why boring is underrated.

## What Rails 8 actually changed (tested, not asserted)

Here is the part that makes the 2026 obituary not just wrong but *out of date*: the concrete, technical reasons people gave for leaving Ruby have been removed from the box. Not mitigated — removed. This is from the official Rails 8.0 release, finalized November 7, 2024 ([rubyonrails.org](https://rubyonrails.org/2024/11/7/rails-8-no-paas-required), [release notes](https://guides.rubyonrails.org/8_0_release_notes.html)), and I'm quoting it rather than characterizing it because the specifics are the argument.

**The Redis dependency is gone.** For years, "a real Rails app needs Redis" was true and was a legitimate complaint: another service to run, monitor, secure, and pay for. Rails 8 ships what the announcement calls "a trifecta of new database-backed adapters named Solid Cable, Solid Cache, and Solid Queue" — and they're database-backed, capable of running on SQLite. The release notes are explicit, per adapter: Solid Queue "replaces the need for Redis, also a separate job-running framework, like Resque, Delayed Job, or Sidekiq." Solid Cache "replaces either Redis or Memcached for storing HTML fragment caches." Solid Cable "replaces Redis to act as the pubsub server to relay WebSocket messages." That is the entire historical "Rails needs a pile of infrastructure" objection, addressed at the framework default level. The job framework *and* the cache server *and* the pubsub server collapse into "the database you already have."

**The Node build step is gone.** The other durable complaint — "I came to Rails to escape JavaScript tooling and now there's a `node_modules` and a bundler in my Ruby app" — is also addressed. Per the announcement, "Propshaft is a dividend from the mission to focus on #NOBUILD as the default path" and is "now the default for all Rails 8 applications," replacing Sprockets. The default path to ship JavaScript no longer routes through a Node build pipeline. The thing people specifically left to get away from is no longer in the default stack.

**Deployment stopped requiring a PaaS.** Rails 8 "comes preconfigured with Kamal 2 for deploying your application anywhere," and the generated Dockerfile "has been upgraded to include a new proxy called Thruster, which sits in front of the Puma web server." The implicit "...and you'll need Heroku to make any of this manageable" is gone too: the default new app is a container you can deploy to a plain server.

Sit with the timeline. The technical case for "we left Ruby because of Redis / Sidekiq / the Node step / the PaaS tax" was *answered at the framework's default level in late 2024* — and the 2026 obituaries are still citing the pre-2024 version of the stack. They're not wrong about why people left. They're wrong that the reasons still exist.

## The boring advantage is a real advantage

"Boring" gets used as an insult. It is, for a team trying to launch, a feature you should be willing to pay for. Here is the mechanism, not the platitude.

**One framework decision instead of forty.** Start a project on a high-hype stack and the framework is the first of forty decisions: router, ORM, migration tool, job queue, cache layer, auth, asset pipeline, test framework, the wiring between all of them. Each is a fork in the road, a comparison matrix, a "well it depends," and a thing your next hire has to learn *because you chose it specifically*. Rails' actual product is that it has already made those forty decisions, they cohere, and they're the same forty in every Rails app on earth. That is what "omakase" means and it is worth real money: the decisions you don't make are the schedule you don't lose, and the onboarding you don't have to write. Boring here means *every Rails app looks like every other Rails app*, which is exactly the property you want when a new engineer has to be productive in week one instead of month two.

**Fewer moving parts to debug at 3am.** This is the one that connects straight back to the Elixir post's frame. An incident's difficulty scales with the number of independent components that could be the cause. The pre-Rails-8 stack — app, database, Redis, Sidekiq, a Node-built asset bundle, a PaaS abstraction — is six things that can independently fail and three network boundaries between them where a 3am page can hide. The Rails 8 default stack collapses several of those *into the database you were already running and already monitoring*. Fewer services is fewer failure domains is fewer things to check while you're half awake. Boring infrastructure is infrastructure with a small incident surface, and a small incident surface is the single most underrated property a young company can have.

**Settled means the answers are findable.** A boring stack's other dividend: every problem you will hit, someone hit in 2017 and wrote down. The Stack Overflow answer exists. The pattern is in a book. The framework convention tells you where the code goes so you're not relitigating project structure. High-hype stacks trade this away — the answer to your problem might be in a GitHub issue from four months ago that's already out of date because the API changed. Maturity means the search results are *correct*. That is shipping velocity, directly.

None of this is exciting. That's the entire point. Excitement is a cost you pay in shipping time, and a team trying to launch should be extremely suspicious of how exciting its stack is.

## The honest limits

A flagship post that only sells the upside is a brochure, so here is where Ruby genuinely loses, with no hedging — the same discipline I held the [Elixir post](https://sublimecoding.com/blog/elixir-concurrency-model) to.

**Hiring optics are real, even when the hiring is fine.** You *can* hire Ruby engineers; the talent pool is deep and experienced precisely because the language is mature. But "we're a Rails shop" reads as less exciting to *some* candidates than a trendier stack, and perception is a real recruiting input even when it's not a real engineering one. You'll occasionally lose a candidate to vibes. The mitigation — strong engineers mostly care that the stack ships and the team is good — is true, but pretending the optics cost is zero is dishonest. There's a sorting effect worth naming, though: the engineers most repelled by "boring" are often the ones most attracted to novelty for its own sake, and novelty-seeking is not a trait you want concentrated in the people choosing your production architecture. The filter sometimes selects *for* you. That's a consolation, not a cancellation — you still pay the optics cost; you just occasionally get paid back in judgment.

**AI/ML is still Python's, full stop.** If your product's core is model training, data science, or heavy numerical work, that work is happening in Python and Ruby is not going to change that. The realistic 2026 architecture is Ruby/Rails for the product — the part that is web application, which Rails is exceptional at — calling out to Python (or a hosted model API) for the ML. Trying to do the ML *in* Ruby is fighting the ecosystem for no prize. Know which part of your system you're talking about before you pick the language for it.

**Raw concurrency is still not its game.** This is the direct handoff to the other two posts. For workloads that are fundamentally about massive concurrent connections or failure-isolated parallel work, Ruby's model is not the strong choice — that's the [Elixir/BEAM argument](https://sublimecoding.com/blog/elixir-concurrency-model) for failure isolation, and the [Go argument](https://sublimecoding.com/blog/go-ai-agent-frameworks) for concurrent network daemons. The honest framing: Rails is the best tool I know for the *web application* — the CRUD, the forms, the business logic, the admin, the 80% of most products that is not a concurrency problem. When the dominant pain genuinely *is* concurrency or failure isolation, that specific service should be Elixir or Go, and that's fine — "use Rails for the product, use the right thing for the hard concurrent edge" is a stronger architecture than forcing one runtime to do both.

Notice none of these limits is "it's dead." They're "here is the specific shape of work it's not best at," which is true of every tool and is the only useful form of criticism.

## Verdict: pick Rails in 2026 when…

The honest checklist, not the fan version. Reach for Rails in 2026 when most of these are true:

- [ ] **The product is mostly a web application** — CRUD, forms, workflows, an admin, business logic. This is the 80% case and it's exactly Rails' home turf.

- [ ] **The team is small and the deadline is real.** The forty decisions Rails already made are forty you don't lose schedule to. Boring is fastest here.

- [ ] **You want a small operational surface.** The Rails 8 Solid stack collapses Redis/Sidekiq/Node/PaaS into "the database you already run." Fewer failure domains, fewer 3am unknowns.

- [ ] **Your concurrency needs are normal web concurrency**, not "hundreds of thousands of stateful connections that must fail independently." If they're the latter, that service is Elixir or Go — by design, not defeat.

- [ ] **Your ML is an API call or a separate Python service**, not the thing Ruby itself has to do.

- [ ] **You can recruit on "we ship fast and the team is good"** rather than needing stack novelty to attract people.

If most of those boxes are checked, the 2026 "Ruby is dead" narrative is not describing your reality — it's describing a stack that stopped existing in November 2024 and a metric (hype) that never shipped anything.

"Dead" was always a confusion of hype velocity with shipping velocity. Ruby's hype velocity is near zero and its shipping velocity is excellent, Rails 8 just deleted the last good technical reasons to leave, and the companies with the most to lose are still on it because — quietly, unexcitingly, reliably — it ships. Boring isn't the eulogy. Boring is the whole product.

---

## 15 Elixir Libraries I Reach For in 2026

URL: https://sublimecoding.com/blog/elixir-libraries-i-reach-for-2026
Published: 2026-06-23
Tags: Elixir, AI tools, developer workflows

>
**TL;DR:** A curated day-one stack beats the
awesome-list dump every time. The famous [awesome-elixir](https://github.com/h4cc/awesome-elixir) page is
a thousand links sorted alphabetically — useful as a reference, useless
as a decision. (For the other end of the spectrum, the scale-infra libraries big companies released, [I audited what Discord and Supabase open-sourced](https://sublimecoding.com/blog/discord-supabase-elixir-libraries) for whether any of it is still maintained.) This is the opposite: the ~15 packages I actually
`mix deps.get` on the first morning of a new Elixir project
in 2026, the one job each earns its place doing, and the libraries I
dropped to get here (HTTPoison, Tesla, Quantum, hand-rolled mocks). The
honest tradeoff: a tight stack means you sometimes write 20 lines you
could have pulled from a dependency — and that’s the trade I take every
time, because a dependency you don’t understand is a liability you can’t
debug at 2am.

There’s a moment on every new project where the `deps`
block in `mix.exs` is empty and you have to decide who you’re
inviting to the party. Get it right and the next six months are smooth.
Get it wrong — pick the unmaintained thing, the over-abstracted thing,
the thing with no escape hatch — and you spend those months fighting
your own foundation.

I’ve made enough of those mistakes to have opinions. What follows
isn’t every good library; it’s the ones I reach for *by default*,
before I know much about the project, because they’ve earned trust
across many of them. At the end I’ll give you the rubric I use so you
can judge your own stack instead of cargo-culting mine.

A note on what this is *not*: it’s not a substitute for
`mix phx.new`. Phoenix, Plug, Telemetry, and friends come in
the box. This is about the deliberate additions on top.

## HTTP & talking to the outside world

**[Req](https://hex.pm/packages/req)** — the
single most consequential default I’ve changed in years. Req is a
batteries-included HTTP client built on top of Finch, and it is the
answer to “what do I use to make an HTTP request in Elixir?” in 2026.
Retries, redirects, JSON encode/decode, compression, and streaming all
work out of the box with sane defaults. I reach for it the instant a
project needs to call *anything* — a payment API, an LLM
endpoint, a webhook.

This is the clearest “what I dropped” story in the whole list. I used
to reach for HTTPoison, then Tesla. HTTPoison is effectively in
maintenance mode and its ergonomics show their age. Tesla is still [actively maintained](https://hex.pm/packages/tesla) and
genuinely good — its middleware model is powerful — but for most
projects Req gives me 95% of what Tesla’s middleware stack did with a
fraction of the ceremony. I keep Tesla in my back pocket for the rare
case where I need a very specific adapter or a custom middleware
pipeline shared across many clients. For everything else, Req. (José
Valim maintains it; that’s not why it’s good, but it’s a decent proxy
for “this will still be here in three years.”)

**[Finch](https://hex.pm/packages/finch)** —
I rarely write `Finch.request/2` directly anymore, because
Req sits on top of it. But Finch is what I name explicitly when I need
connection pooling tuned for a specific high-volume downstream — a
metrics sink, an LLM provider I’m hammering. Knowing it’s the layer
underneath Req is the difference between “the HTTP client is slow” and
“I need a dedicated pool with 50 connections to this one host.” Reach
for it directly when pool topology matters.

**[Jason](https://hex.pm/packages/jason)** —
JSON in and out. It’s so ubiquitous that half your other deps already
pull it in. One honest caveat worth knowing in 2026: Jason’s last
*stable* release predates much of this list — recent versions on
Hex are alpha-tagged. In practice it’s rock-solid and battle-tested at
enormous scale, so I still install it without hesitation, but it’s worth
keeping an eye on whether the wider ecosystem shifts toward stdlib JSON
(Elixir now ships a built-in `JSON` module) over the next
year. For now, Jason stays.

## Data & Ecto

**[Ecto](https://hex.pm/packages/ecto) + [ecto_sql](https://hex.pm/packages/ecto_sql)** — not
optional, not interesting to argue about. If your project touches a
database, Ecto is the data-mapping and query layer, full stop. What’s
worth saying to newer Elixir developers: resist the urge to treat
changesets as annoying boilerplate. They’re the single best place to
validate input at the boundary, and leaning into them is how you keep
garbage out of your domain. I write changesets even for embedded schemas
that never hit a database, purely for the validation pipeline.

The Ecto-adjacent thing I’ll flag: separate queries for
`has_many`, JOINs for `belongs_to`. The number of
N+1 problems and accidental Cartesian products I’ve reviewed because
someone reached for a clever single query is not small. Ecto gives you
the tools to do it right; the discipline is on you.

## Background work — the part Elixir does better than anyone

**[Oban](https://hex.pm/packages/oban)** — if
I could only install one library from this entire list, it would be
Oban. Postgres-backed background job processing with a real database
transaction guarantee, a UI, cron, uniqueness, rate limiting, and a
maturity level that’s frankly rare in any ecosystem. The day-one move on
any project with a database: add Oban *before* you need it,
because the first time you reach for “I’ll just spawn a Task” for
something that absolutely must not be lost, you’ll wish you had it.

Two non-negotiables I enforce on every Oban job, no exceptions: jobs
must be **idempotent** (they *will* run twice
eventually), and args must use **string keys** and contain
only serializable data — never a struct, never a PID. I’ve written a
deeper piece on driving AI agents through Oban queues (*coming
soon* — the durable-execution-for-LLM-calls pattern deserves its own
post), because the retry/backoff/uniqueness machinery turns out to be
exactly what flaky, expensive model calls need.

**[Broadway](https://hex.pm/packages/broadway)** — not
every project, but I reach for it the moment the shape is “ingest a
firehose and process each message with backpressure” — SQS, Kafka,
RabbitMQ, GCP Pub/Sub. Oban is for *jobs you enqueue*; Broadway
is for *streams you consume*. Knowing which problem you have
keeps you from bending one into the other. Skip it entirely if you don’t
have a real pipeline — it’s not a default, it’s a when-you-need-it.

A note on scheduling: I no longer install [Quantum](https://hex.pm/packages/quantum) for new projects.
It’s a fine cron-style scheduler, but Oban Cron covers the same ground
with the same durability guarantees as the rest of my background work,
so I’d rather have one mental model than two. If a project already has
Quantum and no Oban, that’s fine — I just don’t add it fresh.

## Web & LiveView

**[Phoenix
LiveView](https://hex.pm/packages/phoenix_live_view)** — the reason a one-person team can ship
interactive product without a separate frontend stack. I default to
LiveView for essentially all internal tools and most product UI, and
only reach for a heavier client-side approach when there’s a genuine
offline or ultra-low-latency requirement. The two disciplines that
separate good LiveView from janky LiveView: use **streams**
for any list that can grow past a hundred items (don’t hold ten thousand
rows in socket assigns), and never run an unconditional DB query in
`mount` — use `assign_async` or gate on
`connected?/1`. I go deeper on the runtime model that makes
this cheap in [Elixir’s
concurrency model](https://sublimecoding.com/blog/elixir-concurrency-model) and on why the BEAM is a natural fit for [agentic workloads](https://sublimecoding.com/blog/elixir-beam-ai-agents).

**[Bandit](https://hex.pm/packages/bandit)**
— the pure-Elixir HTTP server that’s now the default under new Phoenix
projects, replacing Cowboy. I don’t usually *add* it so much as
*keep* it — but I’ll call it out because if you’re upgrading an
older app still on Cowboy, moving to Bandit is one of the lower-risk,
higher-clarity upgrades available: a stack trace that stays in Elixir
instead of disappearing into Erlang internals is worth a lot when
something goes wrong.

## Testing & developer experience

**[Mox](https://hex.pm/packages/mox)** —
mocks based on explicit behaviours, not magic. This is the one that
enforces an architecture I’d want anyway: define a behaviour for your
boundary (the payment gateway, the LLM client), mock *that*, and
your production code is forced to depend on a contract rather than a
concrete module. I reach for it on day one specifically because
retrofitting “wrap third-party APIs behind a behaviour” later is
painful. Mox is mature and stable — infrequent releases here are a
feature, not neglect. The thing I dropped: hand-rolled mock modules and
`:meck`-style runtime patching, both of which let you mock
things that should have been refactored instead.

**[ExMachina](https://hex.pm/packages/ex_machina)** — test
factories. The job it earns its place doing: making test data setup
readable so your tests document intent instead of drowning in struct
literals. Mature, low-cadence, exactly what you want from a factory
library. I’ll note the honest alternative — many teams now prefer plain
factory *functions* in a test support module over a library — and
that’s a legitimate choice. I still reach for ExMachina out of habit and
because `build`/`insert`/`params_for`
cover the cases I hit; if you’re starting fresh and want zero deps,
plain functions are fine.

**[StreamData](https://hex.pm/packages/stream_data)** —
property-based testing. I don’t property-test everything, but for
anything with a parser, a serializer, a state machine, or non-trivial
data transformation, generating a thousand random valid inputs finds the
edge case your three hand-written examples missed. Reach for it on the
gnarly 20% of the codebase; example-based tests for the rest.

**[Credo](https://hex.pm/packages/credo)** —
static analysis for consistency. On a solo project it’s a second
opinion; on a team it’s how you stop bikeshedding style in code review.
I wire it into CI on day one so the conversation is “the linter says”
rather than “I personally prefer.”

**[Dialyxir](https://hex.pm/packages/dialyxir)** — the
friendly Mix wrapper over Dialyzer for success typing. The honest take:
Dialyzer’s error messages can be genuinely cryptic, and the first PLT
build is slow. But the class of bug it catches — passing the wrong shape
into a function three call-hops away — is exactly the class that’s
expensive to catch any other way. I add it, I tune it to not be noisy,
and I accept that it’s a long-term investment, not an instant payoff.
Skip it on a throwaway prototype; add it the moment the project is going
to outlive the quarter.

**[Sobelow](https://hex.pm/packages/sobelow)** —
security-focused static analysis built specifically for Phoenix. It
catches the Phoenix-shaped footguns: missing CSRF protection,
`raw/1` on untrusted content (XSS), SQL injection via string
interpolation, config leaks. Runs in seconds, lives in CI, and the
false-positive rate is low enough that the signal is worth the noise.
Day-one install on anything with a web surface.

## The AI-era stack — where Elixir is quietly excellent

This is the section that didn’t exist on my list three years ago, and
it’s the reason this post anchors the site’s Elixir-and-AI cluster. The
short version: the BEAM is a genuinely good place to run AI workloads —
concurrent, fault-tolerant, with first-class streaming — and I’ve argued
separately that [Elixir is one of
the languages AI writes best](https://sublimecoding.com/blog/elixir-ai-codes-best). Here’s what I install.

**[Nx](https://hex.pm/packages/nx)** —
multi-dimensional tensors and numerical computing, the foundation of the
entire Elixir ML stack. You rarely use it directly in a product, but
it’s the layer everything else stands on, and when you need to do math
on embeddings — cosine similarity, normalization — this is where it
lives.

**[Bumblebee](https://hex.pm/packages/bumblebee)** —
pre-trained transformer models (embeddings, Whisper, BERT-family, and
more) running *in your BEAM process*, no Python sidecar. The
single highest-leverage use I reach for: generating embeddings locally
for search and RAG instead of paying a per-token API for every chunk.
It’s a young library (downloads are modest), so I treat it as “excellent
for the embedding/inference cases it covers” rather than “drop-in for
any model on HuggingFace.” Verify your specific model is supported
before betting on it.

**[Instructor](https://hex.pm/packages/instructor)** —
structured, schema-validated output from LLMs. You define an Ecto
schema, the library coaxes the model into returning data that conforms
to it, and validation failures get fed back to the model to retry. The
honest status: it’s [early-stage](https://hex.pm/packages/instructor) with low
adoption and an infrequent release cadence, so I wrap it behind my own
boundary module (so I can swap the implementation) rather than letting
it leak through the codebase. But the *pattern* — “make the LLM
return validated structs, not strings I have to parse” — is
non-negotiable for any serious LLM feature, and Instructor is the
cleanest expression of it in Elixir today. If you’d rather not take the
dependency, the pattern is reproducible in ~50 lines with Req + Ecto
changesets + a retry loop.

**[pgvector](https://hex.pm/packages/pgvector)** — vector
columns and nearest-neighbor search wired into Ecto. This is my honest
recommendation for RAG in Elixir, and it’s deliberately *not* a
RAG framework. There are dedicated RAG libraries on Hex —
`arcana` and `rag_ex` both exist and have had
recent releases — but their adoption is so thin (hundreds to
low-thousands of downloads) that I won’t put them under a production
feature yet. What I actually build with is the boring, durable
combination: **pgvector** for storage and similarity
search, **Bumblebee** or an embedding API for the vectors,
and **Req** for the LLM calls — no framework in the middle.
Postgres you already operate, queries you can read, an escape hatch at
every layer. I’m writing a full walkthrough of this exact stack
(*coming soon*); the headline is that you do not need a RAG
framework to ship RAG in Elixir, and in 2026 you’re better off without
one.

## Operations — the two I add before I think I need them

**[Hammer](https://hex.pm/packages/hammer)**
— rate limiting with pluggable backends. The day-one case: you have one
external API endpoint, one expensive LLM call, one auth route that you
do *not* want hammered, and Hammer gives you a clean token-bucket
without standing up Redis if you don’t want to. Cheap to add, painful to
retrofit under load.

**[PromEx](https://hex.pm/packages/prom_ex)**
— Prometheus metrics plus pre-built Grafana dashboards for the libraries
you’re already running (Phoenix, Ecto, Oban, the BEAM itself). I add
observability before the incident, not during it. The “before I need it”
framing is the whole point: the metrics you wish you had are always the
ones you didn’t instrument.

**[tzdata](https://hex.pm/packages/tzdata)**
— the unglamorous one. If your app does anything with time zones, you
need a tz database, and `tzdata` is it. I mention it only
because forgetting it produces confusing runtime errors the first time
you try to shift a datetime into a named zone. Install it, configure it
to auto-update, move on.

## The rubric — how to judge your own stack

I didn’t pick these by popularity. Each one clears four bars, and you
can apply the same bars to anything you’re tempted to add:

Criterion
The question I ask

**Maintained**
Is there a recent release, or is recency simply unnecessary because
it’s *done*? (Mox being quiet is fine; a half-finished thing
being quiet is not.)

**Does one thing**
Can I describe its job in a single sentence? Libraries that do five
things are five things that can break.

**Escape hatch**
When it doesn’t do what I need, can I drop to the layer below — or
am I trapped inside an abstraction? (Req → Finch is the model
here.)

**Debuggable at 2am**
When it fails in production, does the stack trace make sense, and
could I read the source if I had to?

A dependency that fails any of these is one I’d rather replace with
code I own. That’s the trade behind the whole list: I’ll write a little
more myself to depend on a little less that I don’t understand.

## The verdict

Layer
Reach for
Skip / dropped

HTTP
**Req** (Finch underneath)
HTTPoison; Tesla unless you need its middleware

JSON
**Jason** (watch stdlib `JSON`)
—

Data
**Ecto** + ecto_sql
—

Background jobs
**Oban**
Quantum (Oban Cron covers it)

Stream ingestion
**Broadway** *(when you have a pipeline)*
—

Web UI
**Phoenix LiveView** + **Bandit**
Heavy SPA unless truly needed

Testing
**Mox**, **ExMachina**,
**StreamData**
hand-rolled mocks; `:meck`

Static analysis
**Credo**, **Dialyxir**,
**Sobelow**
—

AI / ML
**Nx**, **Bumblebee**,
**Instructor**, **pgvector**
dedicated RAG frameworks (too early)

Ops
**Hammer**, **PromEx**,
**tzdata**
—

Fifteen-ish names, but really one principle: a small stack of
well-understood, single-purpose libraries with escape hatches beats a
sprawling one you can’t fully reason about. Steal the ones that fit your
project, run them through the rubric, and drop anything that fails it —
including, if it fails for you, anything on this list.

If you’re building on the BEAM for AI specifically, start with [why the runtime fits agentic
work](https://sublimecoding.com/blog/elixir-beam-ai-agents), then [the concurrency
model that makes it cheap](https://sublimecoding.com/blog/elixir-concurrency-model). The Oban-for-agents and pgvector-RAG deep
dives are next in the cluster — and they’re built on exactly this
stack.

---

## Build an AI Agent Loop in 50 Lines of Elixir

URL: https://sublimecoding.com/blog/build-ai-agent-loop-50-lines-elixir
Published: 2026-06-21
Tags: AI, agents, Elixir

>
**TL;DR:** Claude Code, Cursor, Devin, Codex — they all
run the same pattern under the hood. The LLM proposes an action. Your
code executes it. You feed the result back. The LLM proposes the next
action. Repeat until done. That’s it. In Elixir, this is a GenServer
with a recursive message loop. Fifty lines. No framework, no SDK beyond
an HTTP client. Once you see the pattern, you can’t unsee it — and
you’ll never need to treat agent frameworks as black boxes again.

## What every agent framework is hiding from you

Open the source of any AI agent framework — [LangChain](https://github.com/langchain-ai/langchain), [CrewAI](https://github.com/crewAIInc/crewAI), Anthropic’s own
[Claude Code](https://github.com/anthropics/claude-code) —
and strip away the plugin registries, the YAML configs, the abstract
base classes, the middleware stacks. What’s left is always the same
thing:

`loop do
 response = call_llm(messages)

 if response.wants_to_use_a_tool?
 result = run_tool(response.tool_name, response.tool_args)
 messages = messages ++ [response, result]
 else
 break response.final_answer
 end
end
```

That’s the agent loop. The entire intellectual content of “agentic
AI” is a while loop with an LLM call inside it.

The frameworks add real value on top — tool registries, memory
management, multi-agent routing, streaming, token tracking — but none of
those are the *core*. The core is this loop. And if you don’t
understand the loop, you can’t debug the framework. You can’t reason
about costs. You can’t explain why your agent got stuck in an infinite
cycle or burned $40 on a task that should’ve cost $0.50.

So let’s build it.

## The pattern: observe → decide → act → loop

Before we write code, here’s the pattern with proper names. It’s
called a [ReAct loop](https://arxiv.org/abs/2210.03629) in
the literature, but you don’t need the paper. It’s four steps:

- **Observe** — gather the current state. On the first
turn, this is the user’s goal. On subsequent turns, it’s the result of
the last tool call.

- **Decide** — send the accumulated context to the LLM.
It either proposes a tool call or returns a final answer.

- **Act** — if it proposed a tool call, execute it. If it
returned a final answer, you’re done.

- **Loop** — append the tool result to the conversation
and go back to step 2.

That’s it. The LLM is the decision engine. Your code is the executor.
The conversation history is the state.

In Elixir, this maps perfectly to a GenServer. The conversation
history is the process state. Each iteration is a
handle_info` callback that sends itself the next
`:step` message. Supervision, crash isolation, and process
monitoring come free from OTP — the things agent frameworks in other
languages spend thousands of lines reimplementing.

## The 50-line agent

Here it is. The full module — GenServer, API call, tool dispatch,
loop. Count the lines yourself.

`[](#cb2-1)defmodule Agent.Loop do
[](#cb2-2) use GenServer
[](#cb2-3)
[](#cb2-4) def run(goal, tools \\ []), do: GenServer.start_link(__MODULE__, {goal, tools})
[](#cb2-5)
[](#cb2-6) @impl true
[](#cb2-7) def init({goal, tools}) do
[](#cb2-8) send(self(), :step)
[](#cb2-9) {:ok, %{messages: [%{role: "user", content: goal}], tools: tools}}
[](#cb2-10) end
[](#cb2-11)
[](#cb2-12) @impl true
[](#cb2-13) def handle_info(:step, state) do
[](#cb2-14) case call_llm(state.messages, state.tools) do
[](#cb2-15) {:tool_use, name, input, assistant_msg, tool_use_id} ->
[](#cb2-16) result = dispatch_tool(name, input)
[](#cb2-17)
[](#cb2-18) tool_msg = %{
[](#cb2-19) role: "user",
[](#cb2-20) content: [%{type: "tool_result", tool_use_id: tool_use_id, content: result}]
[](#cb2-21) }
[](#cb2-22)
[](#cb2-23) send(self(), :step)
[](#cb2-24) {:noreply, %{state | messages: state.messages ++ [assistant_msg, tool_msg]}}
[](#cb2-25)
[](#cb2-26) {:done, answer} ->
[](#cb2-27) IO.puts("\n✅ Agent finished: #{answer}")
[](#cb2-28) {:stop, :normal, state}
[](#cb2-29) end
[](#cb2-30) end
[](#cb2-31)
[](#cb2-32) defp call_llm(messages, tools) do
[](#cb2-33) body = %{model: "claude-sonnet-5", max_tokens: 4096, messages: messages}
[](#cb2-34) body = if tools == [], do: body, else: Map.put(body, :tools, tools)
[](#cb2-35)
[](#cb2-36) headers = [{"x-api-key", System.get_env("ANTHROPIC_API_KEY")}, {"anthropic-version", "2023-06-01"}]
[](#cb2-37) {:ok, %{status: 200, body: resp}} = Req.post("https://api.anthropic.com/v1/messages", json: body, headers: headers)
[](#cb2-38)
[](#cb2-39) case resp["stop_reason"] do
[](#cb2-40) "tool_use" ->
[](#cb2-41) tool = Enum.find(resp["content"], &(&1["type"] == "tool_use"))
[](#cb2-42) assistant_msg = %{role: "assistant", content: resp["content"]}
[](#cb2-43) {:tool_use, tool["name"], tool["input"], assistant_msg, tool["id"]}
[](#cb2-44)
[](#cb2-45) _ ->
[](#cb2-46) text = resp["content"] |> Enum.find(&(&1["type"] == "text")) |> Map.get("text", "")
[](#cb2-47) {:done, text}
[](#cb2-48) end
[](#cb2-49) end
[](#cb2-50)
[](#cb2-51) defp dispatch_tool(name, input) do
[](#cb2-52) IO.puts("🔧 #{name}(#{inspect(input)})")
[](#cb2-53)
[](#cb2-54) case name do
[](#cb2-55) "read_file" -> input["path"] |> File.read() |> then(fn {:ok, c} -> c; {:error, r} -> "Error: #{r}" end)
[](#cb2-56) "list_files" -> (input["path"] || ".") |> File.ls!() |> Enum.join("\n")
[](#cb2-57) "write_file" -> File.write!(input["path"], input["content"]); "ok"
[](#cb2-58) _ -> "Unknown tool: #{name}"
[](#cb2-59) end
[](#cb2-60) end
[](#cb2-61)end
```

That’s it. No framework. No agent SDK. The init` callback
sets up the conversation with the user’s goal and sends the first
`:step` message. Each `:step` calls the LLM,
checks whether it wants to use a tool, and either dispatches the tool
and loops or prints the final answer and stops.

Two things to notice about the message format. The assistant message
(`%{role: "assistant", content: resp["content"]}`) carries
the raw content blocks from the API response — including both text and
tool_use blocks. The `tool_use_id` is passed separately and
only used to build the `tool_result` message back. This keeps
the messages clean: everything sent back to the API on the next turn
matches the [Messages
API schema](https://docs.anthropic.com/en/docs/build-with-claude/tool-use/overview) exactly.

## Running it

To see this work, define your tools using Anthropic’s tool use schema
and call `run/2`:

`[](#cb3-1)tools = [
[](#cb3-2) %{
[](#cb3-3) name: "read_file",
[](#cb3-4) description: "Read the contents of a file at the given path.",
[](#cb3-5) input_schema: %{
[](#cb3-6) type: "object",
[](#cb3-7) properties: %{
[](#cb3-8) path: %{type: "string", description: "Absolute or relative file path"}
[](#cb3-9) },
[](#cb3-10) required: ["path"]
[](#cb3-11) }
[](#cb3-12) },
[](#cb3-13) %{
[](#cb3-14) name: "list_files",
[](#cb3-15) description: "List all files in a directory.",
[](#cb3-16) input_schema: %{
[](#cb3-17) type: "object",
[](#cb3-18) properties: %{
[](#cb3-19) path: %{type: "string", description: "Directory path. Defaults to current directory."}
[](#cb3-20) },
[](#cb3-21) required: []
[](#cb3-22) }
[](#cb3-23) }
[](#cb3-24)]
[](#cb3-25)
[](#cb3-26)Agent.Loop.run("Read my mix.exs and tell me what Elixir version this project uses.", tools)
```

You’ll see output like:

🔧 read_file(%{"path" => "mix.exs"})
✅ Agent finished: Based on your mix.exs, this project uses Elixir ~> 1.18.
```

The agent read the file, interpreted it, and answered — two turns,
one tool call, done. For a more complex goal, it’ll chain multiple tool
calls automatically. Ask it to “find all test files and summarize what
they test” and you’ll watch it call list_files`, then
`read_file` on each result, then synthesize. The loop just
keeps going until the model is satisfied.

## What you get for free from OTP

If you’ve built agent loops in Python or TypeScript, you’ve probably
added code for:

- **Crash isolation** — if a tool call throws, the agent
shouldn’t crash your whole application.

- **Supervision** — if the agent process dies, something
should notice and optionally restart it.

- **Concurrency** — running multiple agents
simultaneously without blocking.

- **Process monitoring** — knowing when an agent finishes
or dies.

In Elixir, you have all of these before you write a single line. The
`GenServer` is a process. If it crashes, only that process
dies. Wrap it in a `Supervisor` and it restarts
automatically. Start ten agents and they run concurrently on the BEAM
scheduler — no threads, no async/await, no event loop. Call
`Process.monitor/1` on the agent’s PID to get a message when
it finishes.

This is what I mean when I say [the BEAM is the runtime AI agents
want](https://sublimecoding.com/blog/elixir-beam-ai-agents). The primitives that agent frameworks bolt on top of Python —
Celery for durability, threading for concurrency, signal handlers for
cleanup — are the default in OTP. You start with them.

## The three guardrails that separate a toy from a tool

The 50-line version works, but it’ll happily loop forever if the LLM
gets confused, burn through your API budget on a runaway task, or hang
indefinitely on a slow model call. Here are the three guardrails that
make it safe for real use.

### 1. Max iterations

The simplest protection: a counter.

`[](#cb5-1)def init({goal, tools}) do
[](#cb5-2) send(self(), :step)
[](#cb5-3)
[](#cb5-4) {:ok, %{
[](#cb5-5) messages: [%{role: "user", content: goal}],
[](#cb5-6) tools: tools,
[](#cb5-7) step: 0,
[](#cb5-8) max_steps: 30
[](#cb5-9) }}
[](#cb5-10)end
[](#cb5-11)
[](#cb5-12)def handle_info(:step, %{step: step, max_steps: max} = state) when step >= max do
[](#cb5-13) IO.puts("⛔ Agent hit step limit (#{max}). Stopping.")
[](#cb5-14) {:stop, :normal, state}
[](#cb5-15)end
[](#cb5-16)
[](#cb5-17)def handle_info(:step, state) do
[](#cb5-18) # ... same loop logic, plus:
[](#cb5-19) {:noreply, %{state | step: state.step + 1, messages: updated_messages}}
[](#cb5-20)end
```

Thirty steps is generous for most tasks. A runaway agent hitting this
limit is a signal that the goal was too vague or the tools are
insufficient — both things you want to know about rather than papering
over with more iterations.

### 2. Token budget

Model calls cost money. Track cumulative usage and kill the run when
it crosses a threshold.

[](#cb6-1)defp call_llm(messages, tools) do
[](#cb6-2) # ... same Req.post call ...
[](#cb6-3)
[](#cb6-4) usage = %{
[](#cb6-5) input: resp["usage"]["input_tokens"],
[](#cb6-6) output: resp["usage"]["output_tokens"]
[](#cb6-7) }
[](#cb6-8)
[](#cb6-9) case resp["stop_reason"] do
[](#cb6-10) "tool_use" ->
[](#cb6-11) tool = Enum.find(resp["content"], &(&1["type"] == "tool_use"))
[](#cb6-12) assistant_msg = %{role: "assistant", content: resp["content"]}
[](#cb6-13) {:tool_use, tool["name"], tool["input"], assistant_msg, tool["id"], usage}
[](#cb6-14)
[](#cb6-15) _ ->
[](#cb6-16) text = resp["content"] |> Enum.find(&(&1["type"] == "text")) |> Map.get("text", "")
[](#cb6-17) {:done, text, usage}
[](#cb6-18) end
[](#cb6-19)end
```

Then in handle_info`:

`[](#cb7-1)def handle_info(:step, state) do
[](#cb7-2) case call_llm(state.messages, state.tools) do
[](#cb7-3) {:tool_use, name, input, assistant_msg, tool_use_id, usage} ->
[](#cb7-4) total = state.tokens_used + usage.input + usage.output
[](#cb7-5)
[](#cb7-6) if total > state.token_budget do
[](#cb7-7) IO.puts("💸 Token budget exhausted (#{total} / #{state.token_budget})")
[](#cb7-8) {:stop, :normal, state}
[](#cb7-9) else
[](#cb7-10) result = dispatch_tool(name, input)
[](#cb7-11) tool_msg = %{role: "user", content: [%{type: "tool_result", tool_use_id: tool_use_id, content: result}]}
[](#cb7-12) send(self(), :step)
[](#cb7-13) {:noreply, %{state | tokens_used: total, messages: state.messages ++ [assistant_msg, tool_msg]}}
[](#cb7-14) end
[](#cb7-15)
[](#cb7-16) {:done, answer, _usage} ->
[](#cb7-17) IO.puts("\n✅ Agent finished: #{answer}")
[](#cb7-18) {:stop, :normal, state}
[](#cb7-19) end
[](#cb7-20)end
```

A sensible default for Sonnet is 100,000 tokens per run. That’s
roughly $0.80 — enough for a substantial task, cheap enough that a
runaway won’t surprise you on the bill.

### 3. Timeout

LLM API calls hang sometimes. The tempting fix is
Process.send_after(self(), :timeout, 60_000)` — but that
won’t fire while the process is blocked inside a synchronous HTTP call.
The message just sits in the mailbox until `Req.post`
returns, which defeats the point.

The real fix is simpler: tell Req itself to enforce the deadline.

`[](#cb8-1)defp call_llm(messages, tools) do
[](#cb8-2) body = %{model: "claude-sonnet-5", max_tokens: 4096, messages: messages}
[](#cb8-3) body = if tools == [], do: body, else: Map.put(body, :tools, tools)
[](#cb8-4)
[](#cb8-5) case Req.post("https://api.anthropic.com/v1/messages",
[](#cb8-6) json: body,
[](#cb8-7) headers: [
[](#cb8-8) {"x-api-key", System.get_env("ANTHROPIC_API_KEY")},
[](#cb8-9) {"anthropic-version", "2023-06-01"}
[](#cb8-10) ],
[](#cb8-11) receive_timeout: 60_000
[](#cb8-12) ) do
[](#cb8-13) {:ok, %{status: 200, body: resp}} ->
[](#cb8-14) # ... parse response as before ...
[](#cb8-15)
[](#cb8-16) {:error, %Req.TransportError{reason: :timeout}} ->
[](#cb8-17) {:error, :timeout}
[](#cb8-18)
[](#cb8-19) {:error, reason} ->
[](#cb8-20) {:error, reason}
[](#cb8-21) end
[](#cb8-22)end
```

Then handle it in the loop:

[](#cb9-1){:error, :timeout} ->
[](#cb9-2) IO.puts("⏱️ LLM call timed out after 60s. Stopping.")
[](#cb9-3) {:stop, :normal, state}
```

Sixty seconds is conservative. Most Sonnet calls return in 3–15
seconds. If you’re waiting sixty, the API is having a bad day and
burning more retries won’t help.

## Where to go from here

You now own the primitive. Every agent framework is this loop with
more stuff on top. Here’s what that “more stuff” looks like when you’re
ready:

**Multi-tool registries.** The
dispatch_tool` function in the example is a hard-coded
`case` statement. For a real system, you’d define a behaviour
— `@callback execute(map()) :: String.t()` — and register
modules dynamically. Each tool becomes its own module with its own
tests.

**Durable runs that survive deploys.** The GenServer
version loses everything if the node restarts. For long-running agents
(tens of minutes, dozens of steps), you want the conversation state in
Postgres, not in process memory. I wrote a full walkthrough of this
pattern: [Oban as a Durable
AI Agent Runtime in Elixir](https://sublimecoding.com/blog/oban-durable-ai-agent-runtime).

**Streaming.** The example waits for the full response
before acting. For interactive use — showing the user what the agent is
thinking in real-time — you’d stream tokens back through a LiveView
socket. The pattern is covered in [Streaming LLM Tokens in
LiveView, the 2026 Way](https://sublimecoding.com/blog/streaming-llm-tokens-liveview-2026).

**Memory and context management.** After enough tool
calls, the conversation history exceeds the model’s context window.
Production agents prune old tool results, summarize prior steps, or use
a sliding window. This is where the frameworks genuinely earn their
weight.

But you don’t need any of that to start. You need the loop, three
guardrails, and a goal. Fifty lines. One GenServer. The rest is
iteration.

---

## I Read Oban's Source Code So You Don't Have To

URL: https://sublimecoding.com/blog/oban-source-code-read
Published: 2026-09-03
Tags: Elixir, engineering, agents

>
**TL;DR:** Oban’s README tells you it’s “reliable” and
“backed by Postgres.” It doesn’t tell you that job fetching is one
`UPDATE ... FOR UPDATE SKIP LOCKED` wrapped in a CTE to stop
the query planner from optimizing it into something slower, that leader
election is a plain `INSERT ... ON CONFLICT` upsert against a
single-row table, or that the same mechanism that makes it reliable —
high-frequency `UPDATE`s on one table — is exactly what fills
that table with dead tuples faster than autovacuum can clean them. I
read `lib/oban/engines/basic.ex`,
`lib/oban/notifiers/postgres.ex`,
`lib/oban/peers/database.ex`, and
`lib/oban/stager.ex` in the `oban-bg/oban` repo at
tag `v2.23.1` (the current stable release, tagged August 2,
2026) to write this. Every mechanism below is a function I opened, not a
claim I remembered from a blog post.

## Scope note

I’ve [written before
about running an AI agent loop on Oban](https://sublimecoding.com/blog/oban-durable-ai-agent-runtime) — the case for using it as an
agent runtime instead of hand-rolling a `GenServer`. This
post is the other half: *why the mechanics hold up* under that
workload, and where they don’t. It assumes you’re comfortable with [the concurrency model underneath
it](https://sublimecoding.com/blog/elixir-concurrency-model) — processes, supervision, and a single Postgres primary doing
more work than people expect. If you’re deciding whether to use Oban,
read that one first. If you’re already running it and want to know
what’s actually happening under `Oban.insert/1`, read this
one.

## How does Oban fetch jobs without workers fighting over them?

Oban fetches a batch of jobs with a single `UPDATE`
statement that uses Postgres’s `FOR UPDATE SKIP LOCKED` row
lock — any row another transaction already has locked is silently
skipped rather than blocked on, so N workers polling the same queue
never queue behind each other waiting for a lock.

The actual query lives in
`Oban.Engines.Basic.fetch_jobs/3`. It’s built with Ecto, and
the shape matters more than the syntax:

`[](#cb1-1)WITH subset AS (
[](#cb1-2) SELECT id FROM oban_jobs
[](#cb1-3) WHERE state = 'available'
[](#cb1-4) AND queue = $1
[](#cb1-5) ORDER BY priority, scheduled_at, id
[](#cb1-6) LIMIT $2
[](#cb1-7) FOR UPDATE SKIP LOCKED
[](#cb1-8))
[](#cb1-9)UPDATE oban_jobs
[](#cb1-10)SET state = 'executing',
[](#cb1-11) attempted_at = now(),
[](#cb1-12) attempt = attempt + 1
[](#cb1-13)FROM subset
[](#cb1-14)WHERE oban_jobs.id = subset.id
[](#cb1-15) AND oban_jobs.attempt < oban_jobs.max_attempts
[](#cb1-16)RETURNING oban_jobs.*;
```

Two things in the real source are easy to miss reading the docs
instead of the code. First, the subset` CTE isn’t decorative
— the comment directly above it in `basic.ex` explains why it
exists: “the Postgres planner may choose to generate a plan that
executes a nested loop over the LIMITing subquery, causing more UPDATEs
than LIMIT,” so the CTE acts as an “optimization fence” forcing Postgres
not to flatten the query. Without it, a clever planner could re-run the
inner scan per outer row and update more jobs than the
`LIMIT` requested — silently over-claiming work. Second, the
whole fetch runs inside `Repo.transaction/2`, and the claim
(the `UPDATE`) and the read (`RETURNING`) happen
atomically — a worker either gets a job it now owns, with
`attempt` already incremented, or it gets nothing.

The practical consequence: two nodes running 20 workers each against
the same queue never contend for the same row. Each fetch takes what’s
free, skips what isn’t, and the lock is released the instant the
transaction commits. That’s the whole trick — it’s a boring
`UPDATE`, not a distributed consensus protocol, and that’s
exactly why it scales as well as it does under a single Postgres
primary.

## When does Oban still poll?

Oban still polls once a second by default, through a
`GenServer` called the Stager, to move `scheduled`
and `retryable` jobs into `available` state —
`LISTEN`/`NOTIFY` tells workers a new job exists,
but Postgres has no equivalent push mechanism for “this row’s scheduled
time has now arrived.”

`Oban.Stager` (`lib/oban/stager.ex`) runs
`Engine.stage_jobs/3` on a timer
(`interval: :timer.seconds(1)` in the struct default) inside
`handle_info(:stage, state)`. `stage_jobs/3` in
`basic.ex` is another `UPDATE`, this one moving
rows whose `scheduled_at <= now()` from
`scheduled`/`retryable` into
`available`. Only then does it call
`notify_queues/1`, which fires
`Notifier.notify(conf, :insert, payload)` — the actual
`LISTEN`/`NOTIFY` push that wakes idle worker
processes.

So the two mechanisms are layered, not redundant: staging is
time-based polling because Postgres has no `NOTIFY` for “a
timestamp condition became true,” and notification is push-based because
once a row is staged, telling every listening producer is cheap and
instant via `pg_notify`. `Oban.Notifiers.Postgres`
(`lib/oban/notifiers/postgres.ex`) implements this over a
dedicated `Postgrex.SimpleConnection` that issues raw
`LISTEN "channel";` statements and relays incoming
notifications to registered listener PIDs — one persistent connection
per node, not one per queue. Worth noting from the moduledoc directly:
this notifier doesn’t work behind a transaction-mode PgBouncer, because
`LISTEN` needs a session-scoped connection; Oban ships
`Oban.Notifiers.PG` (distributed Erlang `:pg`, no
database round-trip at all) as the alternative for that topology.

The one-second stage interval is also why a scheduled job in Oban is
never *exactly* on time — it fires within roughly a second of its
`scheduled_at`, which is fine for retry backoff and
cron-adjacent work, and the wrong tool if you need sub-second scheduling
precision.

## How does a cluster of nodes agree on one leader?

Oban elects a leader with a plain `INSERT ... ON CONFLICT`
upsert against a single-row-per-cluster `oban_peers` table —
whichever node’s insert lands first holds a lease with an expiry, and
every other node’s conflicting insert just fails to claim it, so there’s
no separate consensus algorithm to reason about.

This matters because several plugins — the cron scheduler, the
pruner, the stager’s own “am I the one that runs the global stage”
decision — must run exactly once across a cluster, not once per node.
`Oban.Peers.Database`
(`lib/oban/peers/database.ex`) runs its own election on a
timer (default 30 seconds, halved when the node currently holds
leadership, so a leader re-asserts twice as often as challengers probe).
The core of it is `regular_upsert/2`:

`[](#cb2-1)defp regular_upsert(
[](#cb2-2) %{node: node, expires_at: expires_at}
[](#cb2-3) = peer_data,
[](#cb2-4) state
[](#cb2-5) ) do
[](#cb2-6) on_conflict =
[](#cb2-7) "oban_peers"
[](#cb2-8) |> where([p], p.node == ^node)
[](#cb2-9) |> update([p],
[](#cb2-10) set: [expires_at: ^expires_at]
[](#cb2-11) )
[](#cb2-12)
[](#cb2-13) repo_opts = [
[](#cb2-14) conflict_target: :name,
[](#cb2-15) on_conflict: on_conflict
[](#cb2-16) ]
[](#cb2-17)
[](#cb2-18) case Repo.insert_all(
[](#cb2-19) state.conf, "oban_peers",
[](#cb2-20) [peer_data], repo_opts
[](#cb2-21) ) do
[](#cb2-22) {0, nil} -> false
[](#cb2-23) {_, nil} -> true
[](#cb2-24) end
[](#cb2-25)end
```

conflict_target: :name` means the table has a unique
constraint on the cluster name — one row per Oban instance,
cluster-wide. If the insert actually adds a row, `{1, nil}`
comes back and that node becomes leader. If a row already exists and
belongs to a *different* node, the `on_conflict`
update doesn’t fire either — look closely at its
`where([p], p.node == ^node)`, where `node` is
destructured from the *inserting* node’s own
`peer_data`. A challenger’s update matches no row, Postgres
reports zero rows affected, the `case` falls to
`{0, nil} -> false`, and that node isn’t leader.
Crucially, the other node’s row is never touched; if it were, a
challenger would be extending the sitting leader’s lease for it.

That same `WHERE` clause is what makes renewal work. When
the current leader re-upserts, the clause matches its own row,
`expires_at` slides forward, `{1, nil}` comes
back, and it keeps leadership. No heartbeat protocol, no Raft — one
unique index and one `WHERE` clause doing the work of
both.

Leadership has a lease, not a permanent hold:
`delete_expired_peers/1` runs before every election and
removes any peer row past its `expires_at`, so a leader that
crashes without a clean shutdown loses leadership within one
`interval` window, not indefinitely. On a graceful shutdown,
`terminate/2` deletes the leader’s own row and calls
`Notifier.notify(conf, :leader, %{down: ...})` so the next
election happens immediately instead of waiting out the timer.

## What each mechanism costs you

None of this is free. Every mechanism above leans on a specific
Postgres feature, and every one of those features has a bill that shows
up somewhere else.

Mechanism
Postgres feature it leans on
What it costs you

Job fetch (`fetch_jobs/3`)
`FOR UPDATE SKIP LOCKED` + CTE optimization fence
Every fetch is still a full `UPDATE`, so it generates a
dead tuple even for a job that runs in 4ms

Scheduling wake-up (`stage_jobs/3`)
1-second polling `UPDATE` (Stager)
A fixed floor on scheduling precision, plus one more periodic
`UPDATE` on the same hot table

New-job notification (`Notifier.notify/3`)
`LISTEN`/`NOTIFY` over a dedicated session
connection
Breaks under PgBouncer transaction/statement pooling; needs
`Oban.Notifiers.PG` as a fallback there

Leader election (`Peers.Database`)
`INSERT ... ON CONFLICT` unique-constraint upsert
A single row is a hot write path every 15–30s per node; fine at
normal scale, a real bottleneck only at extreme node counts

State transitions (complete/error/discard)
`UPDATE` in place, never `DELETE` until
pruned
Every one of these row updates is a new tuple version MVCC has to
keep around until `VACUUM` reclaims it

## The cost nobody puts in the getting-started guide

That last row is the one that actually bites in production, and it’s
the one an agent workload makes worse, not better. Every job in Oban’s
`oban_jobs` table goes through several `UPDATE`s
in its lifetime — claimed (fetch), completed or errored, and eventually
deleted by the pruner — and Postgres’s MVCC model doesn’t overwrite a
row in place. Each `UPDATE` writes a new tuple version and
marks the old one dead; `VACUUM` is the only thing that
reclaims that space. Oban’s own [scaling guide](https://oban.hexdocs.pm/scaling.html) says it
plainly: “the MVCC system only flags rows for deletion later. Then,
those rows are deleted when the auto-vacuum runs” — and warns that “the
default auto vacuum settings are conservative and may fall behind on
active tables.”

Reliability and vacuum pressure come from the same line of code. You
don’t get one without the other.

This is exactly the profile an AI agent workload produces: short jobs
(one LLM call, one tool call), enqueued in bursts, each one going
through claim → complete in seconds, at a volume that can run into the
tens of thousands per hour on a busy agent fleet. That’s high UPDATE
churn on one table, which is precisely what [PlanetScale’s
writeup on keeping a Postgres queue healthy](https://planetscale.com/blog/keeping-a-postgres-queue-healthy) and [Richard
Yen’s piece on the consequences of using Postgres as a job queue](https://richyen.com/postgres/2026/05/04/postgres_job_queue.html)
both flag as the failure mode nobody budgets for: dead tuples
accumulating faster than autovacuum can clean them, tables ballooning to
tens of gigabytes when the live data is a few megabytes, and — the
sharper point — `SKIP LOCKED` doesn’t mean rows are invisible
to a scan. Postgres still has to find each dead or locked row and check
its status before moving on, so a bloated table makes the *fetch
itself* slower, not just `VACUUM`.

If you’re running an agent fleet that enqueues a job per LLM turn,
tune this before it becomes an incident — not after the table is into
the tens of gigabytes the Postgres writeups keep describing. It’s also
past what your application monitoring is watching for, which is the same
blind spot [instrumenting LLM calls
with telemetry](https://sublimecoding.com/blog/phoenix-llm-telemetry) exists to close.

Oban’s own guide gives the concrete counter-move: tune autovacuum on
`oban_jobs` specifically, more aggressively than Postgres’s
cluster-wide defaults —

`[](#cb3-1)ALTER TABLE oban_jobs SET (
[](#cb3-2) autovacuum_vacuum_scale_factor = 0,
[](#cb3-3) autovacuum_vacuum_threshold = 100
[](#cb3-4));
```

— which tells autovacuum to trigger after roughly 100 dead tuples
regardless of table size, instead of the default scale-factor-based
threshold that waits for a percentage of the table to go dead first
(irrelevant on a small table, dangerously slow on a large one). The
Oban.Plugins.Pruner` plugin, which deletes
`completed`/`cancelled`/`discarded`
jobs after a configurable `max_age` (default 60 seconds), is
the other half — it keeps the table itself small so each
`VACUUM` pass has less to do. And at genuinely extreme
volume, Oban’s docs point at table partitioning as the real fix, because
“dropping tables entirely is instantaneous and leaves zero bloat” — a
mechanism only available in Oban Pro, not the open-source core I read
for this post.

## The takeaway from reading it instead of trusting the README

Oban’s reliability isn’t magic — it’s a small set of ordinary
Postgres primitives (row locks, `LISTEN`/`NOTIFY`,
a unique-constraint upsert, MVCC) composed carefully, with the comments
in `basic.ex` showing real awareness of how the query planner
could betray you. That composition is also why the failure modes are
ordinary Postgres failure modes: vacuum pressure, connection pooler
incompatibility, a hot single-row table under extreme leader-election
load. None of that shows up if you only read the README. It shows up the
first time your agent workload runs hot enough to out-churn autovacuum,
which is a Tuesday, not an edge case.

Reading it end to end is also a reminder of what source code is
*for*. (The next one in this series is [a Linux distribution’s
AGENTS.md](https://sublimecoding.com/blog/omarchy-agents-md-source-read) — a codebase where “run the tests” means booting a VM.) The reliability model came out of following intent across
files and building a picture the README never states — a human habit,
and one worth keeping as [the notation around
our code drifts toward readers who aren’t people](https://sublimecoding.com/blog/languages-written-for-agents-not-humans).

If you’re deciding whether Oban’s operational model fits your team’s
AI infrastructure — or you’ve already got one of these tables quietly
growing past what your monitoring is watching for — that’s a
conversation worth having before the growth curve gets ahead of you. I
help AI startups get this kind of infrastructure review done as part of
[fractional CTO engagements](https://sublimecoding.com/consulting).

---

## Oban as a Durable AI Agent Runtime in Elixir

URL: https://sublimecoding.com/blog/oban-durable-ai-agent-runtime
Published: 2026-07-02
Tags: Elixir, agents, AI

>
**TL;DR:** If you’re building an AI agent in Elixir — a
ReAct/tool-call loop that thinks, calls a tool, observes, repeats — you
do not need a bespoke agent framework or a hand-rolled
`GenServer` to drive it. You need a durable job queue, and
you already have the best one in the ecosystem: [Oban](https://hexdocs.pm/oban). It’s Postgres-backed,
idempotent by construction, retry-with-backoff out of the box, and its
jobs survive a node restart because the state lives in a table, not in
process memory. Model each agent step as one Oban job that enqueues the
next, and your agent run keeps its place through an LLM 429, a flaky
tool call, or a deploy that rolls the node mid-loop. The honest
tradeoff: Oban is the wrong tool for sub-second interactive streaming
(that’s LiveView’s job) and for genuinely ephemeral single-shot calls
(just call the API). It shines precisely when a run is long, multi-step,
costs real money per step, and absolutely must not silently die.

## The agent that lost forty steps to a deploy

Here’s the scenario that made me write this. You’ve got an agent
doing something genuinely useful — say, working through a backlog of
support tickets, or refactoring a module across a dozen files, or
researching a topic across thirty web fetches. It’s a loop: the model
proposes a tool call, your code runs the tool, you feed the result back,
the model proposes the next call. Twenty, forty, sixty iterations deep.
Each model turn costs money. Each tool call has side effects.

You drive it with the obvious thing: a `GenServer` holding
the conversation history and the loop counter in its state, recursing on
`handle_info`.

Then you ship a deploy.

The release rolls the node. The supervisor shuts down your
`GenServer`. Its state — forty steps of accumulated context,
the half-finished plan, the tool results you already paid for —
evaporates with the process. There’s no `terminate/2` heroics
that save you here, because the work isn’t “flush a buffer,” it’s “an
entire stateful computation lives only in this process’s heap.” When the
node comes back, the agent is just gone. No error. No resume. You find
out because a customer asks why their ticket never got answered.

I’ve watched a team burn an afternoon adding “checkpointing” to
exactly this design — serializing the `GenServer` state to a
table every N steps, reloading on boot, reconciling partial writes. By
the time they finished, they had rebuilt a worse version of a durable
job queue. They’d named it `AgentRunner`. They could have
named it Oban.

## An agent loop is just a job queue you haven’t named yet

Strip an agent down to its control flow and look at it honestly:

- **Think** — call the LLM, get back a tool call (or a
final answer).

- **Act** — execute the tool. This can fail, time out, or
partially succeed.

- **Observe** — append the result to the running
context.

- **Repeat** — until the model says “done” or you hit a
step budget.

Now look at the failure modes that actually bite in production:

- The LLM returns a **429** and you need to back off and
try again — without losing the run.

- A tool call **fails transiently** (network blip,
downstream 503) and should be retried, but a *different* failure
(malformed args, a 400) should not be retried forever.

- A step **partially succeeds** — the tool wrote a row,
then the process died before recording that it did — and a naive retry
double-writes or double-charges.

- The node **restarts** mid-run and the whole thing must
resume from where it left off.

Every one of those is a solved problem in a durable job queue.
Retry-with-backoff: solved. Distinguishing “retry this” from “give up on
this”: solved. Idempotency so a retried unit doesn’t double-act: solved.
Surviving a restart because state lives in Postgres, not in a process:
that’s the entire point of the thing.

This is the reframe: **a step of a ReAct loop maps one-to-one
onto an Oban job.** Thinking and acting happen inside
`perform/1`. The “repeat” is the job enqueuing its successor.
The conversation state — the part that must outlive any single process —
lives in a row in your own `agent_runs` table, keyed by a run
ID that every job in the chain carries in its args. The queue is durable
because [Oban persists every job to
Postgres](https://hexdocs.pm/oban) and only marks it `completed` after
`perform/1` returns successfully. Crash before that, and the
job is still `executing` or `available` — it will
run again.

You don’t get this for free with a `GenServer`. You get it
for free with Oban because durability is the substrate, not a feature
you bolt on.

## The code: one agent step as a durable, idempotent Oban job

Here’s a real worker. It performs **one** step of an
agent loop, persists the result, and enqueues the next step. It’s
open-source Oban (v2.22 at time of writing) — no Pro features — and the
comments call out the durability property at each point. APIs here are
checked against the [current
`Oban.Worker` docs](https://oban.hexdocs.pm/Oban.Worker.html).

First, the run state. This is the part that must survive a restart,
so it lives in a table you own:

`[](#cb1-1)defmodule MyApp.Agent.Run do
[](#cb1-2) use Ecto.Schema
[](#cb1-3)
[](#cb1-4) schema "agent_runs" do
[](#cb1-5) field :goal, :string
[](#cb1-6) field :status, Ecto.Enum, values: [:running, :completed, :failed], default: :running
[](#cb1-7) field :step, :integer, default: 0
[](#cb1-8) field :max_steps, :integer, default: 40
[](#cb1-9) # The full ReAct transcript: every think/act/observe turn, appended in order.
[](#cb1-10) # Because it's a column, a node restart can't lose it.
[](#cb1-11) field :messages, {:array, :map}, default: []
[](#cb1-12) field :result, :string
[](#cb1-13)
[](#cb1-14) timestamps(type: :utc_datetime_usec)
[](#cb1-15) end
[](#cb1-16)end
```

Now the worker — one step of the loop:

[](#cb2-1)defmodule MyApp.Agent.StepWorker do
[](#cb2-2) use Oban.Worker,
[](#cb2-3) queue: :agents,
[](#cb2-4) # An agent run is long. We want generous headroom for transient LLM/tool
[](#cb2-5) # failures, but NOT infinite — a poison step must eventually give up.
[](#cb2-6) max_attempts: 8,
[](#cb2-7) # Idempotency at the QUEUE level: never let two jobs for the same
[](#cb2-8) # (run_id, step) exist concurrently. If an enqueue is retried, or two
[](#cb2-9) # producers race, Oban dedupes instead of double-running the step.
[](#cb2-10) unique: [
[](#cb2-11) period: :infinity,
[](#cb2-12) fields: [:worker, :args],
[](#cb2-13) keys: [:run_id, :step],
[](#cb2-14) # Only dedupe against jobs that haven't finished. A genuinely new
[](#cb2-15) # attempt at the same step after a cancel is still allowed.
[](#cb2-16) states: [:available, :scheduled, :executing, :retryable]
[](#cb2-17) ]
[](#cb2-18)
[](#cb2-19) alias MyApp.Agent.{Run, Loop}
[](#cb2-20) alias MyApp.Repo
[](#cb2-21)
[](#cb2-22) @impl Oban.Worker
[](#cb2-23) def perform(%Oban.Job{args: %{"run_id" => run_id, "step" => step}}) do
[](#cb2-24) run = Repo.get!(Run, run_id)
[](#cb2-25)
[](#cb2-26) cond do
[](#cb2-27) run.status != :running ->
[](#cb2-28) # Idempotency at the DATA level: if a retry lands after the run already
[](#cb2-29) # finished (e.g. the job died AFTER committing but BEFORE Oban marked it
[](#cb2-30) # completed), do nothing. Re-running must be a no-op, not a double-spend.
[](#cb2-31) {:cancel, :run_already_finished}
[](#cb2-32)
[](#cb2-33) run.step >= run.max_steps ->
[](#cb2-34) finish(run, :failed, "step budget exhausted")
[](#cb2-35)
[](#cb2-36) true ->
[](#cb2-37) run_step(run, step)
[](#cb2-38) end
[](#cb2-39) end
[](#cb2-40)
[](#cb2-41) defp run_step(run, step) do
[](#cb2-42) # THINK: one LLM call. May raise on a 429 — see classify_error/1 below,
[](#cb2-43) # which decides retry-with-backoff vs. give-up. If this raises, perform/1
[](#cb2-44) # never returns :ok, so Oban leaves the job retryable and the run resumes.
[](#cb2-45) case Loop.next_action(run) do
[](#cb2-46) {:final, answer} ->
[](#cb2-47) finish(run, :completed, answer)
[](#cb2-48)
[](#cb2-49) {:tool_call, call} ->
[](#cb2-50) # ACT: run the tool. Wrap side effects so a retried step can detect
[](#cb2-51) # "I already did this" — see the idempotency note in execute_tool/2.
[](#cb2-52) observation = Loop.execute_tool(run, call)
[](#cb2-53)
[](#cb2-54) # OBSERVE + DURABLY ADVANCE: persist the new transcript and bump the
[](#cb2-55) # step counter in ONE transaction with enqueuing the next job. If the
[](#cb2-56) # node dies an instant later, either the whole advance committed (next
[](#cb2-57) # job is queued) or none of it did (this job is still retryable). There
[](#cb2-58) # is no torn state where we advanced but forgot to schedule the next step.
[](#cb2-59) advance(run, step, call, observation)
[](#cb2-60) end
[](#cb2-61) end
[](#cb2-62)
[](#cb2-63) defp advance(run, step, call, observation) do
[](#cb2-64) next_step = step + 1
[](#cb2-65)
[](#cb2-66) new_messages =
[](#cb2-67) run.messages ++
[](#cb2-68) [%{"role" => "assistant", "tool_call" => call},
[](#cb2-69) %{"role" => "tool", "result" => observation}]
[](#cb2-70)
[](#cb2-71) changeset = Ecto.Changeset.change(run, messages: new_messages, step: next_step)
[](#cb2-72)
[](#cb2-73) # Multi ties the state write and the next-step enqueue into one DB
[](#cb2-74) # transaction. Oban.insert/4 inside a Multi is first-class — the inserted
[](#cb2-75) # job only becomes visible if the whole transaction commits.
[](#cb2-76) Ecto.Multi.new()
[](#cb2-77) |> Ecto.Multi.update(:run, changeset)
[](#cb2-78) |> Oban.insert(:next, __MODULE__.new(%{"run_id" => run.id, "step" => next_step}))
[](#cb2-79) |> Repo.transaction()
[](#cb2-80) |> case do
[](#cb2-81) {:ok, _} -> :ok
[](#cb2-82) {:error, _op, reason, _changes} -> {:error, reason}
[](#cb2-83) end
[](#cb2-84) end
[](#cb2-85)
[](#cb2-86) defp finish(run, status, text) do
[](#cb2-87) # Terminal write: flip status and record the result/error text. Idempotent
[](#cb2-88) # because the status guard in perform/1 makes a re-run of a finished run a no-op.
[](#cb2-89) run |> Ecto.Changeset.change(%{status: status, result: text}) |> Repo.update()
[](#cb2-90) :ok
[](#cb2-91) end
[](#cb2-92)end
```

Kick off a run by inserting the first step:

[](#cb3-1){:ok, run} =
[](#cb3-2) %MyApp.Agent.Run{goal: "Triage and reply to ticket #4821"}
[](#cb3-3) |> Ecto.Changeset.change()
[](#cb3-4) |> MyApp.Repo.insert()
[](#cb3-5)
[](#cb3-6)# Step 0. The `unique` opts mean enqueuing this twice is harmless — the second
[](#cb3-7)# insert is deduped, not run.
[](#cb3-8)%{"run_id" => run.id, "step" => 0}
[](#cb3-9)|> MyApp.Agent.StepWorker.new()
[](#cb3-10)|> Oban.insert()
```

The shape to notice: **each step enqueues exactly the next
step, transactionally, alongside the state write.** That’s the
hand-rolled durable loop. There’s no driver process holding the run
together. The Postgres row *is* the run; the chain of jobs is
just how it walks forward. Roll a deploy at step 23 and the in-flight
job goes back to available`/`retryable`; when the
node returns, Oban picks it up, re-reads the row, and continues. Nothing
was lost because nothing important ever lived in a process.

## Handling the failures that are specific to LLMs

A generic job queue handles “the tool 503’d.” Agents add three
failure modes that need deliberate handling.

**1. The 429, and backoff that respects
`Retry-After`.** When the LLM rate-limits you, the
right move is to back off and try again *without consuming a real
failure*. Oban gives you two levers. For a clean retry that doesn’t
burn an attempt the same way an error does, return
`{:snooze, seconds}` — it reschedules the job for later.
(Note the documented quirk: [snoozing increments
`max_attempts`](https://oban.hexdocs.pm/Oban.Worker.html) so total retries are preserved, which
slightly skews the default backoff curve.) For genuine errors that
*should* count, return `{:error, reason}` and let
`backoff/1` space out the retries:

`[](#cb4-1)# In MyApp.Agent.StepWorker
[](#cb4-2)
[](#cb4-3)@impl Oban.Worker
[](#cb4-4)# backoff/1 must return non_neg_integer() seconds. The default is exponential
[](#cb4-5)# (min 15s + jitter); we override to add a floor and cap. When a tool or model
[](#cb4-6)# hands us an explicit Retry-After, honor it instead of guessing.
[](#cb4-7)def backoff(%Oban.Job{attempt: attempt, meta: %{"retry_after" => secs}})
[](#cb4-8) when is_integer(secs),
[](#cb4-9) do: secs
[](#cb4-10)
[](#cb4-11)def backoff(%Oban.Job{attempt: attempt}) do
[](#cb4-12) base = trunc(:math.pow(2, attempt)) * 5
[](#cb4-13) min(base, 300) + :rand.uniform(10)
[](#cb4-14)end
```

And inside perform/1`, classify the failure so a
retryable 429 backs off while an un-retryable 400 dies cleanly instead
of churning through all eight attempts:

`[](#cb5-1)defp classify_and_raise({:rate_limited, retry_after}) do
[](#cb5-2) # Tell Oban to wait this long, then re-raise so the job goes retryable.
[](#cb5-3) # (Stash retry_after via a meta update, or snooze — pick one; don't do both.)
[](#cb5-4) {:snooze, retry_after}
[](#cb5-5)end
[](#cb5-6)
[](#cb5-7)defp classify_and_raise({:http_error, status}) when status in [400, 422],
[](#cb5-8) # A malformed request will fail identically every time. Don't retry it 8x and
[](#cb5-9) # don't poison the run — cancel this step and let the run record the failure.
[](#cb5-10) do: {:cancel, {:permanent_tool_error, status}}
[](#cb5-11)
[](#cb5-12)defp classify_and_raise({:http_error, status}) when status in 500..599,
[](#cb5-13) # Transient. Return an error so backoff/1 spaces out the retry.
[](#cb5-14) do: {:error, {:transient_tool_error, status}}
```

{:cancel, reason}` is the one to internalize: it stops
the job *and* stops retrying. That’s exactly what you want for a
poison step — a step that will fail identically on every attempt.
Burning all eight `max_attempts` on a guaranteed-400 is just
slow money-burning. (`:discard` does the same thing but is [deprecated in favor of
`{:cancel, reason}`](https://oban.hexdocs.pm/Oban.Worker.html).)

**2. Partial tool output and double-execution.** This is
the subtle one. Say the tool charges a payment API or POSTs to a
downstream system. The job runs the tool successfully, the side effect
happens — and then the node dies *before* `perform/1`
returns and before the transaction commits. Oban, correctly, will retry
the job, because from its perspective the step never completed. Now
you’re about to run the tool a second time.

The queue can’t solve this for you, because the side effect is
outside Postgres. You make the *tool* idempotent:

`[](#cb6-1)defp execute_tool(run, %{"name" => "charge", "args" => args} = call) do
[](#cb6-2) # Derive a stable idempotency key from the run + step. A retry produces the
[](#cb6-3) # SAME key, so the payment API (or your own ledger) dedupes the second call.
[](#cb6-4) # This is what makes "retried step doesn't double-charge" actually true.
[](#cb6-5) idem_key = "run:#{run.id}:step:#{run.step}:charge"
[](#cb6-6) PaymentAPI.charge(args, idempotency_key: idem_key)
[](#cb6-7)end
```

Most serious APIs accept an idempotency key. For your own writes,
enforce it with a unique constraint keyed on {run_id, step}`
and treat the constraint violation as “already done.” The queue’s
`unique` option stops *duplicate jobs*; the
idempotency key stops *duplicate side effects from one job’s
retries*. You need both, and they operate at different layers.

**3. The poison run, not just the poison step.**
`max_attempts` bounds a single step. Bound the whole run too
— that’s the `run.step >= run.max_steps` guard in
`perform/1`. An agent stuck in a tool-call loop will happily
enqueue itself forever; the step budget in your own table is the circuit
breaker. Cheap, and it lives in the same durable place as everything
else.

If your runs are complex enough to need fan-out, branches, or a real
dependency graph (step C waits on A *and* B), that’s where [**Oban
Pro’s `Workflow`**](https://oban.pro/docs/pro/Oban.Pro.Workflow.html) earns its license — it models
jobs as a DAG and supports appending jobs to a running workflow when you
don’t know all the steps up front. That’s a **paid tier**
feature; the self-enqueuing chain above is plain open-source Oban and
covers the linear ReAct loop, which is the common case.

## When NOT to reach for Oban here

This pattern is not a hammer. Three cases where it’s the wrong
call:

**Sub-second interactive streaming.** If a human is
watching tokens stream into a chat UI and expects a reply in two
seconds, Oban is the wrong layer — full stop. Queue latency, even when
small, is latency you’re adding to a tight interactive loop, and you
lose token streaming entirely. That’s [**LiveView’s**](https://sublimecoding.com/blog/elixir-concurrency-model)
job: stream the model output straight to the client over the socket,
keep the in-flight conversation in the LiveView process, and persist
when the turn completes. (A dedicated post on the LiveView-streaming
side of this is coming soon.) Oban is for the *durable
background* agent, not the *live foreground* chat.

**Genuinely ephemeral single-shot calls.** If the whole
“agent” is one prompt, one response, no tools, no loop — just call the
API in a `Task` and move on. Wrapping a single stateless call
in a durable job is ceremony with no payoff. The durability machinery
earns its keep only when there’s multi-step state worth protecting.

**An in-memory, long-lived conversation that must stay
hot.** If you genuinely need a conversation held live in memory
across many turns with sub-step-latency access to working state — a
`GenServer` (ideally under a `DynamicSupervisor`,
keyed in a `Registry`) is the right model, and that’s a
legitimate use of a process: it models real concurrent, stateful,
isolated work. The honest line is the one I drew at the top: reach for
the `GenServer` when state must stay *hot in memory*;
reach for Oban when state must stay *durable on disk and survive the
process dying*. Most batch/background agents want the latter. For
more on when a process is and isn’t the answer, see [the BEAM-for-agents piece](https://sublimecoding.com/blog/elixir-beam-ai-agents) and [the concurrency-model deep
dive](https://sublimecoding.com/blog/elixir-concurrency-model).

## Verdict

Oban (self-enqueuing chain)
Plain GenServer loop
Dedicated agent framework

**Survives a deploy mid-run**
Yes — state in Postgres, job resumes
No — state dies with the process
Depends; usually you provide the store

**Retry + backoff on 429/tool failure**
Built in (`backoff/1`, `{:snooze, _}`,
`{:error, _}`)
You build it
Usually built in

**Idempotency / no double-charge**
`unique` jobs + your idem keys
You build all of it
Varies, often weak

**Sub-second token streaming**
No — use LiveView
Yes (in-process)
Varies

**Distributed across nodes**
Yes — any node pulls the job
No — pinned to one process
Varies

**Operational visibility**
Oban Web / DB queries
You build it
Varies

**Best when**
Long, multi-step, costly, must-not-die runs
Hot in-memory live conversation
You want the framework’s batteries and will pay the lock-in

The rule I’d give a team: **if losing the run is a real cost —
money already spent, side effects already committed, a customer waiting
— make each step an Oban job.** You’ll get durability, retries,
idempotency, backoff, and cross-node distribution from a library your
Elixir app probably already runs, instead of reinventing a worse queue
and calling it an agent framework. Save the `GenServer` for
the genuinely live, in-memory case, and reach for [Oban Pro](https://oban.pro) only when a linear chain isn’t
enough and you need real DAGs.

A companion post on RAG retrieval as a set of cached, idempotent Oban
jobs is coming soon. The durability argument there is the same — it just
happens one layer down, at the tool.

---

*Building an Elixir agent system and trying to decide where the
durability boundary goes? That’s exactly the kind of call I help teams
make as a fractional CTO. [Get in touch.](https://sublimecoding.com/consulting)*

*Primary sources: [Oban docs](https://hexdocs.pm/oban)
· [`Oban.Worker`](https://oban.hexdocs.pm/Oban.Worker.html)
· [Oban on GitHub](https://github.com/oban-bg/oban) · [Oban Pro
`Workflow`](https://oban.pro/docs/pro/Oban.Pro.Workflow.html).*

---

## 12 Phoenix LiveView Patterns for Production

URL: https://sublimecoding.com/blog/phoenix-liveview-patterns-production
Published: 2026-07-21
Tags: Elixir, Phoenix, engineering

>
**TL;DR:** Every pattern below is a scar. I learned each
one because the absence of it took down a page, leaked a row a user
shouldn’t have seen, or pinned a node’s memory until it OOM’d. This
isn’t a tips list — tips are cheap and forgettable. It’s twelve named
failure modes and the LiveView construct that prevents each. If you’re
shipping LiveView past the tutorial stage, you’ve already hit some of
these or you’re about to. The meta-pattern at the end is the one that
actually matters: the boundary between the dead render and the connected
socket is where most LiveView bugs live.

LiveView tutorials age well — the framework is stable, the docs are
excellent, and a fresh `mix phx.gen.live` gets you a working
CRUD page in minutes. What the tutorials don’t give you is the
second-order knowledge: the things that work fine with ten rows and fall
over with ten thousand, the patterns that are obviously correct in
hindsight and obviously absent in every junior PR — and in [most of what AI coding agents generate](https://sublimecoding.com/blog/things-ai-coding-agents-get-wrong).

So this is the listicle I wish existed when I was three production
LiveView apps in. Each item is a pattern, the specific failure it
prevents, a short real snippet, and — because honesty matters more than
completeness — a note on when the pattern is overkill. Verified against
Phoenix LiveView 1.1.

I’ve grouped them loosely: rendering and data, async, real-time,
navigation and UI, and safety. Read top to bottom or jump to the one
that’s currently on fire.

---

## Rendering and data

### 1. Use streams for unbounded lists — don’t hold 10k rows in assigns

**The failure mode:** You
`assign(:messages, all_messages)` and it works great in dev.
In production the list grows to 8,000 rows, and now every LiveView
process holds all 8,000 in memory, re-diffs the whole collection on
every update, and ships fat payloads down the socket. A few hundred
concurrent users and the node’s memory chart looks like a hockey
stick.

Streams keep the collection in the DOM, not in the socket’s assigns.
The server holds only the IDs it needs to address.

`[](#cb1-1)def mount(_params, _session, socket) do
[](#cb1-2) {:ok, stream(socket, :messages, Chat.recent_messages())}
[](#cb1-3)end
[](#cb1-4)
[](#cb1-5)def handle_info({:new_message, msg}, socket) do
[](#cb1-6) # prepend, and prune the DOM to 50 items so it never grows unbounded
[](#cb1-7) {:noreply, stream_insert(socket, :messages, msg, at: 0, limit: 50)}
[](#cb1-8)end
```

[](#cb2-1)<div id="messages" phx-update="stream">
[](#cb2-2) <div :for={{dom_id, msg} <- @streams.messages} id={dom_id}>
[](#cb2-3) {msg.body}
[](#cb2-4) </div>
[](#cb2-5)</div>
```

The :limit` option prunes the rendered DOM;
`at: 0` prepends. To replace the whole list (filter changes,
sort changes), use
`stream(socket, :messages, new_list, reset: true)`.

**When it doesn’t apply:** Small, bounded lists — a nav
menu, a status dropdown, the seven items in a wizard. Streams add a
layer of indirection (you lose direct access to the collection in the
socket; you can’t `Enum.count` it server-side). For genuinely
small write-once data, plain assigns are simpler and that simplicity is
worth it.

---

### 2. Don’t query the database in `mount` unconditionally

**The failure mode:** `mount/3` runs
**twice** — once for the dead render (the initial HTTP
request, no socket) and once when the client establishes the WebSocket.
If you run an expensive query at the top of `mount` with no
guard, every page load hits the database twice, and the dead-render
query blocks the first byte of HTML.

Gate the work on `connected?/1`, or hand it to
`assign_async` (pattern 3). The `connected?/1`
branch gives you a cheap dead render and does the real load only on the
live connection:

`[](#cb3-1)def mount(_params, _session, socket) do
[](#cb3-2) socket =
[](#cb3-3) if connected?(socket) do
[](#cb3-4) assign(socket, :dashboard, Analytics.expensive_rollup())
[](#cb3-5) else
[](#cb3-6) assign(socket, :dashboard, nil)
[](#cb3-7) end
[](#cb3-8)
[](#cb3-9) {:ok, socket}
[](#cb3-10)end
```

**When it doesn’t apply:** SEO-critical routes that must
render real content in the dead pass for crawlers. There you *do*
want the data on the first render — but back it with a cache so the dead
render is cheap, and don’t lean on connected?` to skip it.
Know which kind of page you’re on before you reach for this.

---

### 3. Use `assign_async` for slow mounts so first paint isn’t blocked

**The failure mode:** A dashboard mount fans out to
three slow services. With everything inline, the user stares at a blank
page until the slowest one returns. Worse, if one call raises, the whole
mount crashes and the user gets a reconnect loop instead of a partial
page.

`assign_async/3` runs the work in a linked task (only when
connected), paints the page immediately, and patches the result in when
it arrives. The assign becomes a
`Phoenix.LiveView.AsyncResult` struct, and the
`<.async_result>` component handles the loading and
failed states declaratively. For LLM output specifically, where the result arrives token by token rather than all at once, [streaming has its own set of patterns](https://sublimecoding.com/blog/streaming-llm-tokens-liveview-2026).

`[](#cb4-1)def mount(_params, _session, socket) do
[](#cb4-2) {:ok,
[](#cb4-3) assign_async(socket, :stats, fn ->
[](#cb4-4) {:ok, %{stats: Analytics.expensive_rollup()}}
[](#cb4-5) end)}
[](#cb4-6)end
```

[](#cb5-1)<.async_result :let={stats} assign={@stats}>
[](#cb5-2) <:loading><.spinner /></:loading>
[](#cb5-3) <:failed :let={_reason}>Couldn't load stats. Retrying…</:failed>
[](#cb5-4) <.stat_grid data={stats} />
[](#cb5-5)</.async_result>
```

Use start_async/3` (paired with the
`handle_async/3` callback) when you need full control over
what happens with the result — side effects, multiple assigns, custom
error handling — rather than dropping it straight into an assign.

**When it doesn’t apply:** Fast queries. Async has real
overhead — a task, a message round-trip, an `AsyncResult`
wrapper, and loading-state markup in your template. If the query returns
in single-digit milliseconds, wrapping it in `assign_async`
is ceremony that buys you nothing and makes the template noisier.

---

### 4. Use `temporary_assigns` to shrink the diff for write-once data

**The failure mode:** You render a large block of data —
a long report, a big rendered article body — and keep it in assigns.
LiveView holds it for the life of the process and keeps it in the
diff-tracking set, even though it never changes after the first render.
Memory per connection climbs for no benefit.

`temporary_assigns` tells LiveView to render the value
once and then reset it, so it isn’t retained between renders:

`[](#cb6-1)def mount(_params, _session, socket) do
[](#cb6-2) {:ok,
[](#cb6-3) assign(socket, :report, Reports.generate()),
[](#cb6-4) temporary_assigns: [report: nil]}
[](#cb6-5)end
```

After the first render, @report` is `nil` in
the process state. The rendered HTML stays in the DOM; you just stop
paying to keep the source data resident.

**When it doesn’t apply:** Anything you read back or
re-render. The moment you need the value again on a later event,
`temporary_assigns` becomes a footgun — `@report`
is `nil` and you’ll be confused for ten minutes. Streams have
largely replaced this pattern for *collections*; reach for
`temporary_assigns` for large, singular, write-once blobs,
not lists.

---

## Real-time

### 5. Subscribe to PubSub only when `connected?` — and handle the fan-in

**The failure mode:** You call
`Phoenix.PubSub.subscribe/2` at the top of
`mount`. Because `mount` runs twice, you subscribe
during the dead render too — a process that’s about to be thrown away
gets a subscription, and depending on your setup you can end up with
duplicate handlers or leaked subscriptions. Always gate the subscribe on
the live connection:

`[](#cb7-1)def mount(%{"room_id" => room_id}, _session, socket) do
[](#cb7-2) if connected?(socket) do
[](#cb7-3) Phoenix.PubSub.subscribe(MyApp.PubSub, "room:#{room_id}")
[](#cb7-4) end
[](#cb7-5)
[](#cb7-6) {:ok, stream(socket, :messages, Chat.list_messages(room_id))}
[](#cb7-7)end
[](#cb7-8)
[](#cb7-9)def handle_info({:new_message, msg}, socket) do
[](#cb7-10) {:noreply, stream_insert(socket, :messages, msg, at: -1, limit: -50)}
[](#cb7-11)end
```

Note how this composes with pattern 1: PubSub fans messages
*in*, and stream_insert` with a `:limit`
keeps the rendered list bounded no matter how long the session runs.

**When it doesn’t apply:** It always applies if you’re
subscribing in `mount` — the `connected?` guard is
non-negotiable there. The judgment call is *what* you broadcast:
don’t broadcast full structs to a topic with thousands of subscribers if
a lightweight ID-plus-event would let each LiveView fetch what it needs.
Fan-out cost is real.

---

### 6. Get `phx-update="stream"` and DOM ids right

**The failure mode:** You use streams but hand-roll the
container, forget `phx-update="stream"`, or put the wrong
`id` on the row elements. The symptoms are maddening: items
duplicate on insert, deletes don’t remove the right row, reordering
scrambles. Streams are entirely driven by stable DOM ids, and if the ids
don’t line up with what the server computes, the patches land on the
wrong nodes.

The contract: the container needs `phx-update="stream"`,
and each child’s `id` must be the `dom_id` the
stream yields.

`[](#cb8-1)<table>
[](#cb8-2) <tbody id="users" phx-update="stream">
[](#cb8-3) <tr :for={{dom_id, user} <- @streams.users} id={dom_id}>
[](#cb8-4) <td>{user.name}</td>
[](#cb8-5) </tr>
[](#cb8-6) </tbody>
[](#cb8-7)</table>
```

If your items don’t have a usable id` field, set a custom
id generator when you create the stream:
`stream(socket, :users, users, dom_id: &"user-#{&1.email}")`.
Don’t compute ids two different ways in two places — that mismatch is
exactly the bug.

**When it doesn’t apply:** Non-stream collections. If
you’re rendering a plain `:for` over a small assign (pattern
1’s escape hatch), you don’t add `phx-update="stream"` — that
attribute is a promise to the client that this container is
stream-managed, and using it without an actual stream behind it will
misbehave.

---

## Forms

### 7. Build forms with `to_form/2` and validate at the boundary

**The failure mode:** You pass a raw changeset or a bare
map into the template and reach into `@changeset.changes` and
`@changeset.errors` by hand in the markup. It’s verbose, it’s
easy to get wrong, and you lose the framework’s form helpers. Worse, if
you don’t validate on `phx-change`, the first time the user
learns their input is invalid is when they hit submit.

`to_form/2` wraps a changeset (or map) into a
`Phoenix.HTML.Form` the components understand, and
`phx-change` runs your changeset on every keystroke with
`action: :validate` so errors surface inline:

`[](#cb9-1)def mount(_params, _session, socket) do
[](#cb9-2) {:ok, assign(socket, :form, to_form(Accounts.change_user(%User{})))}
[](#cb9-3)end
[](#cb9-4)
[](#cb9-5)def handle_event("validate", %{"user" => params}, socket) do
[](#cb9-6) form =
[](#cb9-7) %User{}
[](#cb9-8) |> Accounts.change_user(params)
[](#cb9-9) |> Map.put(:action, :validate)
[](#cb9-10) |> to_form()
[](#cb9-11)
[](#cb9-12) {:noreply, assign(socket, :form, form)}
[](#cb9-13)end
```

[](#cb10-1)<.form for={@form} phx-change="validate" phx-submit="save">
[](#cb10-2) <.input field={@form[:email]} label="Email" />
[](#cb10-3)</.form>
```

**When it doesn’t apply:** It nearly always applies for
changeset-backed forms. The only time I skip to_form` is a
trivial single-input control (a search box) where there’s no validation
and no changeset — there, a plain input with a `phx-keyup` or
`phx-change` event handler is less machinery.

---

## Navigation and UI

### 8. Choose `push_patch` vs `push_navigate` deliberately

**The failure mode:** You use
`push_navigate/2` to move between tabs of the same LiveView.
Each click fully unmounts and remounts the LiveView — every async load
restarts, every stream resets, every subscription tears down and
re-establishes. The page flickers and feels heavy, and you’ve thrown
away all your in-process state for what should have been a URL
change.

`push_patch/2` stays in the *same* LiveView and
invokes `handle_params/3` — state survives, you just react to
the new params. `push_navigate/2` is for crossing into a
*different* LiveView, and it pays the full remount cost on
purpose.

`[](#cb11-1)# Same LiveView, different filter — cheap, state preserved:
[](#cb11-2){:noreply, push_patch(socket, to: ~p"/orders?status=open")}
[](#cb11-3)
[](#cb11-4)# Different LiveView entirely — full remount, intended:
[](#cb11-5){:noreply, push_navigate(socket, to: ~p"/orders/#{order}")}
```

The heuristic: same live` module →
`push_patch`. Different module → `push_navigate`.
Getting this wrong doesn’t break correctness, but it’s the difference
between a snappy app and a janky one.

**When it doesn’t apply:** When you genuinely
*want* a fresh start — clearing all transient state, resetting a
multi-step flow. There, the remount that `push_navigate`
forces is a feature, not a cost.

---

### 9. Use `Phoenix.LiveView.JS` for UI that should never round-trip to the server

**The failure mode:** You toggle a dropdown, open a
modal, or flip an accordion by sending an event to the server, flipping
a boolean assign, and re-rendering. Now every menu open is a network
round-trip — latency-bound, fragile on flaky connections, and pointless
because the server doesn’t care whether a dropdown is open.

`Phoenix.LiveView.JS` runs these as client-side commands,
with no server involvement. They’re DOM-patch aware, so the state sticks
across server-driven patches:

`[](#cb12-1)<button phx-click={JS.toggle(to: "#menu")}>Menu</button>
[](#cb12-2)<nav id="menu" class="hidden">…</nav>
[](#cb12-3)
[](#cb12-4)<button phx-click={JS.add_class("ring-2", to: "#card")}>Highlight</button>
```

Commands like JS.toggle/1`, `JS.show/1`,
`JS.hide/1`, `JS.add_class/2`,
`JS.remove_class/2`, and `JS.dispatch/2` cover
most pure-UI interactions. When you *do* need the server,
`JS.push/2` lets you push an event while also running client
commands optimistically.

**When it doesn’t apply:** Anything the server is the
source of truth for. If opening a panel needs to load data, mark
something read, or persist a preference, that’s a real event — use
`phx-click` with a handler. `JS` commands are for
*presentational* state the server has no stake in.

---

### 10. Reach for `live_component` only when you need stateful isolation — usually you don’t

**The failure mode:** This one’s overkill in the other
direction. Teams reach for `live_component` (stateful)
reflexively for every reusable chunk, when a plain function component
would do. `live_component` carries real weight: it has its
own `update/2` lifecycle, its own assigns state, its own
`handle_event/3`, and it requires a unique `:id`.
Used where you only needed to render markup from inputs, it’s complexity
with no payoff and a slower mental model for the next reader.

The rule: if the component is a pure function of its inputs, use a
**function component**. Only use
`live_component` when the component needs to own state the
parent shouldn’t manage, or handle its own events in isolation.

`[](#cb13-1)<%!-- Function component: pure render, the default choice --%>
[](#cb13-2)<.user_badge user={@user} />
[](#cb13-3)
[](#cb13-4)<%!-- Live component: owns state + handles its own events --%>
[](#cb13-5)<.live_component module={MyAppWeb.CartWidget} id="cart" user={@user} />
```

**When it does apply:** A self-contained widget with its
own form and validation lifecycle, a piece of UI that needs to react to
its own events without bubbling everything up to the parent, or a
repeated stateful element where you genuinely want isolation. Those are
real — they’re just rarer than the reflex suggests.

---

## Safety

### 11. Know what survives a reconnect — and supervise accordingly

**The failure mode:** A LiveView process crashes, or the
client’s connection drops and reconnects (laptop sleeps, network blips,
deploy rolls the node). The client automatically re-establishes and
**mount` runs again from scratch**. Any state
you held only in the process’s assigns is *gone* — the
half-filled wizard, the accumulated scroll position, the “you have
unsaved changes” flag. If your UX assumed that state was durable, it
just silently evaporated.

The pattern is to be deliberate about durability. Anything that must
survive a reconnect lives somewhere outside the process: the database, a
supervised GenServer/ETS table keyed by user or session, or the URL
itself (which is why pattern 8’s `push_patch` matters —
params survive because they’re in the URL). Process assigns are
*ephemeral by design*.

`[](#cb14-1)def mount(%{"id" => id}, _session, socket) do
[](#cb14-2) # Reconstruct from a durable source on every mount — including reconnects.
[](#cb14-3) # Don't assume prior in-process state exists.
[](#cb14-4) {:ok, assign(socket, :draft, Drafts.get_or_init(id))}
[](#cb14-5)end
```

And the corollary from the iron laws: any long-lived process you
stand up to hold that state — a per-room presence tracker, a session
cache — must be **supervised**. An unsupervised GenServer
holding durable state is a single crash away from losing exactly the
thing you moved out of the LiveView to protect.

**When it doesn’t apply:** Genuinely transient UI state
— which tab is active, whether a tooltip is showing. Losing that on
reconnect is invisible to the user. Don’t build durable storage for
state nobody would miss; reserve the machinery for state whose loss is a
bug.

---

### 12. Authorize on *every* event handler, not just mount`

**The failure mode:** This is the one that turns into a
security incident. You check authorization in `mount` — the
user can see this page, good. Then you write
`handle_event("delete", %{"id" => id}, socket)` and call
`Orders.delete(id)` directly, trusting that because they
reached the page, the action is fine.

It is not fine. The client is fully untrusted. A malicious user can
open the WebSocket and push **any event with any payload**
— events your UI never renders a button for, ids belonging to other
tenants. Mount authorization gates *viewing the page*. It does
nothing for the events. Every `handle_event` that mutates or
reads sensitive data must re-check authorization against the *current
user* and the *specific resource*:

`[](#cb15-1)def handle_event("delete", %{"id" => id}, socket) do
[](#cb15-2) order = Orders.get_order!(id)
[](#cb15-3)
[](#cb15-4) case Bodyguard.permit(Orders, :delete_order, socket.assigns.current_user, order) do
[](#cb15-5) :ok ->
[](#cb15-6) {:ok, _} = Orders.delete_order(order)
[](#cb15-7) {:noreply, stream_delete(socket, :orders, order)}
[](#cb15-8)
[](#cb15-9) {:error, _} ->
[](#cb15-10) {:noreply, put_flash(socket, :error, "Not allowed.")}
[](#cb15-11) end
[](#cb15-12)end
```

Scope the lookup to the user where you can
(Orders.get_order!(current_user, id)`) so you can’t even
load another tenant’s row. Use whatever authorization you already have —
a policy module, a scoped query, a plain function. The non-negotiable
part is that the check happens *here*, on every event, not only
at the door.

**When it doesn’t apply:** Never, for sensitive actions.
The only events you can skip it on are ones with no security surface at
all — a pure-UI event you’d happily let any visitor fire (and those, per
pattern 9, often shouldn’t be hitting the server in the first place).
When in doubt, authorize.

---

## The meta-pattern

If you squint, eleven of these twelve are the same insight wearing
different clothes: **the dead render and the connected socket are
two different worlds, and the client side of that boundary is
hostile.**

`mount` runs twice because of it (patterns 2, 5). You gate
expensive work and subscriptions on `connected?` because of
it (2, 3, 5). State evaporates across the reconnect boundary because the
process on the other side is new (11). And the client can push anything
across the WebSocket because the boundary is a trust boundary, not just
a transport (12). Streams, async, and `JS` commands are all,
in part, about deciding *which side of that line* a piece of work
belongs on — memory and rendering on the client where it scales, truth
and authorization on the server where it’s safe.

Internalize the boundary and most LiveView “weirdness” stops being
weird. Forget it, and you’ll keep rediscovering these twelve scars the
hard way.

For the data-modeling and concurrency machinery underneath all this,
see [the Elixir concurrency
model](https://sublimecoding.com/blog/elixir-concurrency-model) and [why the BEAM is a
natural fit for AI agents](https://sublimecoding.com/blog/elixir-beam-ai-agents). A deeper dive on streaming patterns
specifically — backpressure, infinite scroll, and live tables at scale —
is coming soon as a dedicated post, along with a companion piece on
building RAG on Phoenix.

## Quick reference

#
Pattern
Failure it prevents

1
Streams for unbounded lists
Per-connection memory blowup; fat diffs

2
No unconditional DB query in `mount`
Double query; blocked dead render

3
`assign_async` for slow mounts
Blank page on slow load; mount crash on error

4
`temporary_assigns` for write-once data
Memory retained for data that never changes

5
PubSub subscribe only when `connected?`
Leaked/duplicate subscriptions

6
`phx-update="stream"` + correct DOM ids
Duplicated, mis-deleted, scrambled rows

7
`to_form/2` + `phx-change` validation
Hand-rolled forms; late error feedback

8
`push_patch` vs `push_navigate`
deliberately
Needless full remounts; janky navigation

9
`Phoenix.LiveView.JS` for pure UI
Pointless server round-trips for UI state

10
`live_component` only when stateful
Complexity with no payoff

11
Know what survives reconnect; supervise
Silent state loss; unsupervised crashes

12
Authorize every `handle_event`
Privilege escalation via crafted events

Twelve patterns, twelve scars. If even one of these saves you a 2
a.m. memory alert or a security postmortem, the read paid for
itself.

---

## Build an MCP Server in Phoenix With Hermes

URL: https://sublimecoding.com/blog/build-mcp-server-phoenix-hermes
Published: 2026-07-09
Tags: Elixir, Phoenix, mcp

>
**TL;DR:** An MCP server turns your Phoenix app into a
set of tools that Claude — or any MCP-aware agent — can call directly,
instead of you copy-pasting database dumps into a chat window. The
Elixir library for this is [Hermes](https://hermes-mcp.hexdocs.pm/), and it has first-class
Phoenix support: you define tools as components, mount one plug, and
you’re live. The honest catch is that MCP is young and the spec has
churned — the transport moved from HTTP+SSE to Streamable HTTP, and the
auth story is still maturing. The whole discipline of building a
*good* MCP server is restraint: you expose a small, typed,
authorized allow-list of tools, never raw database access. If you only
have one consumer and a stable internal API already, you may not need
one yet. Read to the end for the decision guidance.

## You want Claude to answer questions about your real app, not a stale CSV

Here’s the scenario that makes this concrete. You’re on a call and
someone asks: “Which customers are over their plan limit right now?” You
*could* answer that. You’d open a console, write a query, paste
the rows somewhere, and reason about them. Or you’d ask Claude — except
Claude can’t see your database. So you dump a CSV into the chat, it’s
already stale by the time you hit enter, and you’ve just pasted customer
data into a context window with no audit trail and no access
control.

An MCP server fixes the shape of that problem. Instead of moving
*data* to the agent, you give the agent a *tool* it can
call: `customers_over_limit`. The agent decides when to call
it, your app runs the actual query under your actual authorization
rules, and only the result comes back. Claude becomes a client of your
application’s capabilities — the same way a mobile app or a cron job is
a client — except the “API” is described in a format the model already
knows how to consume.

That’s the entire pitch. Your app, as a set of tools an agent can
call. The interesting engineering is not in wiring it up — Hermes makes
that almost boring — it’s in deciding *which* tools to expose and
how to guard them. We’ll build a real one, then spend the back half of
this post on the boundary, because the boundary is the whole point.

## What MCP actually is, in three nouns

The Model Context Protocol is a JSON-RPC-based protocol that
standardizes how an AI application (the *host*, like Claude
Desktop or an agent runtime) talks to external systems. Strip away the
spec language and there are three things a server can offer:

- **Tools** — functions the model can *call*, with
typed inputs and structured outputs. This is the active verb:
`lookup_customer`, `create_ticket`,
`customers_over_limit`. Tools are the part you’ll use 90% of
the time.

- **Resources** — data the model can *read*,
addressed by URI. Think of these as GET endpoints: a file, a record, a
document the host can pull into context.

- **Prompts** — reusable prompt templates your server
hands to the host, so common workflows aren’t reinvented in every
client.

Underneath those sits a **transport**. Originally MCP
used stdio (for local servers spawned as subprocesses) and HTTP with
Server-Sent Events for remote ones. As of the 2025-03-26 revision of the
spec, the remote transport is **Streamable HTTP**: the
client POSTs JSON-RPC messages to one endpoint, and the server can
either reply with a plain JSON response or upgrade to an SSE stream for
server-to-client messages. The older HTTP+SSE transport still exists in
the wild, but Streamable HTTP is what you should target for anything
new. (MCP moves fast — as of mid-2026, verify the current transport
revision against the spec before you ship, because this is exactly the
kind of detail that shifts between releases.)

For a Phoenix app, Streamable HTTP is the natural fit: it’s just
another route. You don’t need a separate process, a sidecar, or a
subprocess dance. The same BEAM that’s serving your LiveViews serves
your MCP endpoint, which is part of why Elixir is a genuinely good home
for this. If you want the longer argument for that, I wrote it up in [Why the BEAM is a great fit for AI
agents](https://sublimecoding.com/blog/elixir-beam-ai-agents) and, more fundamentally, in [the Elixir concurrency
model](https://sublimecoding.com/blog/elixir-concurrency-model).

## The build: one server, one real tool, Streamable HTTP into Phoenix

Let’s expose exactly one tool — a read-only customer lookup — and
wire it into an existing Phoenix app. I’m using the Hermes server API as
documented for the current `hermes_mcp` release line (0.14 at
the time of writing). Add the dependency:

`[](#cb1-1)# mix.exs
[](#cb1-2)defp deps do
[](#cb1-3) [
[](#cb1-4) {:hermes_mcp, "~> 0.14"}
[](#cb1-5) # ... your existing deps
[](#cb1-6) ]
[](#cb1-7)end
```

### The tool component

In Hermes, each tool is its own module — a *component*. You
declare its type, define an input schema, and implement
execute/2`. The schema is a small DSL that doubles as
runtime validation *and* the JSON Schema the model sees, which is
exactly the property you want: the contract the agent reads is the
contract your code enforces.

`[](#cb2-1)defmodule MyApp.MCP.Tools.CustomersOverLimit do
[](#cb2-2) @moduledoc "List active customers currently over their plan's usage limit."
[](#cb2-3)
[](#cb2-4) use Hermes.Server.Component, type: :tool
[](#cb2-5)
[](#cb2-6) alias Hermes.Server.Response
[](#cb2-7) alias MyApp.Billing
[](#cb2-8)
[](#cb2-9) # This schema is the tool's typed boundary. It's validated before
[](#cb2-10) # execute/2 ever runs, and it's what the model is shown as the
[](#cb2-11) # tool's input contract. Keep it tight.
[](#cb2-12) schema do
[](#cb2-13) field :limit, :integer,
[](#cb2-14) required: false,
[](#cb2-15) description: "Max rows to return (1-100). Defaults to 25."
[](#cb2-16) end
[](#cb2-17)
[](#cb2-18) @impl true
[](#cb2-19) def execute(params, frame) do
[](#cb2-20) # The authenticated account was placed on the frame by the server's
[](#cb2-21) # init/2 (see below). We scope EVERY query to it — the agent never
[](#cb2-22) # gets to pick whose data it reads.
[](#cb2-23) account = frame.assigns.current_account
[](#cb2-24) row_limit = clamp(params[:limit] || 25, 1, 100)
[](#cb2-25)
[](#cb2-26) # A named context function — NOT raw SQL. The query lives in your
[](#cb2-27) # billing context, where it's tested and authorized like any other.
[](#cb2-28) customers = Billing.customers_over_limit(account, limit: row_limit)
[](#cb2-29)
[](#cb2-30) response =
[](#cb2-31) Response.tool()
[](#cb2-32) |> Response.json(%{
[](#cb2-33) count: length(customers),
[](#cb2-34) customers:
[](#cb2-35) Enum.map(customers, fn c ->
[](#cb2-36) %{
[](#cb2-37) id: c.id,
[](#cb2-38) name: c.name,
[](#cb2-39) plan: c.plan_name,
[](#cb2-40) usage_pct: c.usage_pct
[](#cb2-41) }
[](#cb2-42) end)
[](#cb2-43) })
[](#cb2-44)
[](#cb2-45) {:reply, response, frame}
[](#cb2-46) end
[](#cb2-47)
[](#cb2-48) defp clamp(n, lo, hi), do: n |> max(lo) |> min(hi)
[](#cb2-49)end
```

A few things worth noticing. The execute/2` callback
returns `{:reply, response, frame}` — the frame is threaded
through so any state you accumulated (the authenticated account, for
instance) stays available. `Hermes.Server.Response` gives you
`tool/0` to start a response, then `text/2`,
`json/2`, and `error/2` to build it up. I’m
returning structured JSON rather than prose, because a downstream agent
reasons better over data than over a paragraph.

And critically: this tool calls
`Billing.customers_over_limit/2`, a function in my own
context. It does **not** take a query string from the
model. More on why that matters shortly.

### The server module

The server declares its identity, its capabilities, and which
components it exposes. It also runs `init/2` once per
session, which is your hook for setup.

`[](#cb3-1)defmodule MyApp.MCP.Server do
[](#cb3-2) use Hermes.Server,
[](#cb3-3) name: "MyApp",
[](#cb3-4) version: "1.0.0",
[](#cb3-5) capabilities: [:tools]
[](#cb3-6)
[](#cb3-7) alias MyApp.Accounts
[](#cb3-8)
[](#cb3-9) # Register each tool component. Add only what you've deliberately
[](#cb3-10) # decided to expose — this list IS your allow-list.
[](#cb3-11) component MyApp.MCP.Tools.CustomersOverLimit
[](#cb3-12)
[](#cb3-13) @impl true
[](#cb3-14) def init(_client_info, frame) do
[](#cb3-15) # Authenticate once per session, right here. The frame's transport
[](#cb3-16) # carries the request's HTTP headers; we read the bearer token,
[](#cb3-17) # resolve it to an account, and stash that on the frame so every
[](#cb3-18) # tool can scope to it. A failed auth stops the session before any
[](#cb3-19) # tool runs. (Why init/2 and not a plug? See the next section.)
[](#cb3-20) with "Bearer " <> token <- get_in(frame.transport, [:headers, "authorization"]),
[](#cb3-21) {:ok, account} <- Accounts.account_from_api_token(token) do
[](#cb3-22) {:ok, assign(frame, current_account: account)}
[](#cb3-23) else
[](#cb3-24) _ -> {:stop, :unauthorized}
[](#cb3-25) end
[](#cb3-26) end
[](#cb3-27)end
```

### Authentication and transport, into the endpoint

Notice that the authentication lives in the server’s
init/2` above — not in a Phoenix plug. That distinction
trips people up, so it’s worth stating plainly: **a
`Plug.Conn` assign does not reach a Hermes tool.**
Hermes builds its own `Frame` from the transport’s metadata,
so an `assign(conn, :current_account, ...)` you set in a plug
before the `forward` never appears on
`frame.assigns`. The seam Hermes actually gives you is
`init/2`, where `frame.transport` exposes the
request headers — so that is where authentication belongs.

The good news is you almost certainly already have the mechanism.
MCP’s own auth story (OAuth-based) is still settling, but for a server
that lives inside *your* Phoenix app talking to *your*
internal agents, you have an API token or a signed header today. Read it
off the transport in `init/2`, resolve it to an account, put
the account on the frame, and let every tool read it from
`frame.assigns` — which is exactly what the server above
does.

That leaves the router with just the transport plug —
`Hermes.Server.Transport.StreamableHTTP.Plug`, pointed at
your server. No auth plug in the pipeline; `init/2` owns that
now:

`[](#cb4-1)# lib/my_app_web/router.ex
[](#cb4-2)pipeline :mcp do
[](#cb4-3) plug :accepts, ["json"]
[](#cb4-4)end
[](#cb4-5)
[](#cb4-6)scope "/mcp" do
[](#cb4-7) pipe_through :mcp
[](#cb4-8)
[](#cb4-9) forward "/", Hermes.Server.Transport.StreamableHTTP.Plug,
[](#cb4-10) server: MyApp.MCP.Server
[](#cb4-11)end
```

Finally, supervise the server. Hermes needs its registry, and the
server runs as a supervised process with the Streamable HTTP transport
selected:

[](#cb5-1)# lib/my_app/application.ex
[](#cb5-2)children = [
[](#cb5-3) # ... Repo, Endpoint, PubSub, etc.
[](#cb5-4) Hermes.Server.Registry,
[](#cb5-5) {MyApp.MCP.Server, transport: :streamable_http}
[](#cb5-6)]
```

That’s the whole server. One supervised process, one plug forward,
one authenticated tool. Point an MCP client at
https://yourapp.com/mcp` with a bearer token and Claude can
now ask your app, in your words, which customers are over their limit —
and the answer is computed live, scoped to the caller’s account, through
code you already test.

>
**A note on API confidence.** I verified these module
and function names against the current Hermes documentation:
`use Hermes.Server` with
`name`/`version`/`capabilities`, the
`component/1` macro, `init/2`,
`use Hermes.Server.Component, type: :tool`, the
`schema do ... end` DSL, `execute/2` returning
`{:reply, response, frame}`,
`Hermes.Server.Response` (`tool/0`,
`text/2`, `json/2`, `error/2`),
authentication in `init/2` reading
`frame.transport` headers, the
`Hermes.Server.Frame` `assigns` map and
`assign/3`, `Hermes.Server.Registry`, and
`Hermes.Server.Transport.StreamableHTTP.Plug` with the
`server:` option. Hermes is moving quickly, so before you
copy this into production, run it against the version you’ve pinned and
check the [server quick
start](https://hermes-mcp.hexdocs.pm/) — small naming and option details are the kind of thing that
shifts between minor releases.

## The safety boundary is the actual product

Everything above is plumbing. This section is the opinion, and it’s
the reason I’d trust this pattern in production: **you never hand
an LLM raw access to anything.**

The lazy version of an MCP server exposes a single
`run_sql` tool, hands the model your database connection, and
calls it “flexible.” It is flexible the way leaving your front door open
is flexible. An LLM is a probabilistic text generator that an attacker
can influence through the data it reads — a customer’s name, a support
ticket, a webhook payload. If the model can emit arbitrary SQL, then
anyone who can get text in front of the model can, in effect, run
arbitrary SQL. That’s prompt injection escalated to data exfiltration,
and no amount of “please only read” in the system prompt closes it.

So the boundary is a hard rule with four properties, and each tool
must satisfy all four:

- **Tools, not raw access.** Every tool maps to a
named function in one of your contexts —
`Billing.customers_over_limit/2`, not a query the model
assembles. The set of things the agent can do is exactly the set of
tools you wrote, no more. This is an allow-list, and the allow-list is
short by design.

- **Authorization on every call.** The account is
established at the edge and scoped into every query. The model does not
get to specify *whose* data it reads; that’s pinned from the
authenticated session. This is the same discipline you already apply in
a LiveView’s `handle_event` — authorize the actor, scope the
query — just enforced one layer further out.

- **Validation at the boundary.** The
`schema` block validates inputs before `execute/2`
runs, and I still clamp `limit` to a sane range inside the
tool. Treat tool arguments as exactly as untrusted as a form submission,
because that’s what they are — a form filled out by a model that another
party may be steering. Never build atoms from tool input, never
interpolate it into a query, never trust a range.

- **Auditability.** Because every action is a
discrete, named tool call, you can log it: who called, which tool, what
arguments, what result size. That log is your incident timeline when
something goes wrong, and it’s the thing a raw-SQL tool can never give
you cleanly.

If a tool can’t satisfy all four, it doesn’t ship. The instinct to
expose “just a little more” — a generic search, a flexible filter, a
write tool with broad scope — is the instinct to resist. A small,
boring, well-guarded toolset is the entire value proposition. The
model’s cleverness is supposed to live in *deciding which tool to
call and how to use the result*, not in inventing new capabilities
you didn’t authorize.

A related discipline for write-heavy or long-running tools: don’t do
the work inline in `execute/2`. Enqueue an idempotent
background job and return a handle the agent can poll. That keeps the
tool call fast, gives you a retry boundary, and means a misbehaving
agent can’t pile up half-finished mutations. (I’ve got a dedicated post
on driving agent workloads through Oban coming soon — that’s where this
thread continues.)

## When you should NOT build one yet

I’d be doing you a disservice if I made this sound like free money.
Here’s when to wait:

- **You have exactly one consumer, and it’s your own
code.** MCP’s payoff is *standardization* — many hosts,
one protocol. If the only thing calling your app is a script you also
wrote, a plain function call or an internal API is simpler, faster, and
has none of the protocol’s moving parts. Build the MCP server when you
have multiple agent clients, or a client you don’t control (Claude
Desktop, a teammate’s agent), or a genuine “any MCP host should be able
to use this” requirement.

- **The spec is still moving and you can’t absorb
churn.** The transport already shifted from HTTP+SSE to
Streamable HTTP. Auth is mid-evolution. If you’re shipping something
that has to be stable for two years with no maintenance, you’re going to
be chasing spec revisions. Fine for an internal tool you can update;
risky for a fire-and-forget integration.

- **Your auth model isn’t ready for it.** If you don’t
already have a clean way to authenticate a non-browser, non-human caller
and scope it to an account, build *that* first. The MCP server is
only as safe as the boundary in front of it, and bolting auth on
afterward is how you end up with the open-front-door version.

- **You can’t yet name the small set of tools.** If
your honest answer to “which tools?” is “I’m not sure, let’s expose a
lot and see,” stop. The constrained allow-list isn’t a phase-two
cleanup; it’s the design. A vague toolset is a security incident waiting
for an excuse.

- **You don’t have an agent that benefits.** MCP is
infrastructure for agents. If nobody on your side is actually running [an
agent that would call these tools](https://sublimecoding.com/blog/build-ai-agent-loop-50-lines-elixir), you’re building a road to nowhere.
Have the consumer first.

## Verdict

Build an MCP server when you have a real agent consumer — especially
one you don’t fully control — that would benefit from calling your app’s
capabilities live, and when you already have an authentication boundary
you trust. In Elixir, Hermes makes the mechanics genuinely pleasant:
tools as components, one plug forward, supervised on the same BEAM that
runs the rest of your app. The Streamable HTTP transport drops into a
Phoenix router like any other route.

But treat the library as the easy 20%. The 80% is judgment: a short
allow-list of typed, authorized, auditable tools that wrap named context
functions, with validation at the boundary and not one inch of raw
database access handed to the model. Get that part right and an MCP
server is one of the highest-leverage things you can give an agent — a
safe, live window into your real system. Get it wrong and you’ve built a
remote code execution endpoint with a friendly name.

Start with one tool. Make it read-only. Scope it to the caller. Ship
that, watch the logs, and only then add the second.

---

## Streaming LLM Tokens in LiveView, the 2026 Way

URL: https://sublimecoding.com/blog/streaming-llm-tokens-liveview-2026
Published: 2026-07-14
Tags: Elixir, Phoenix, AI

>
**TL;DR:** You do not need React, a WebSocket
microservice, or a client-side state library to stream an LLM’s response
token-by-token into your UI. Phoenix LiveView already holds an open
connection to the browser, and a spawned process feeding tokens back
over it is all the plumbing you need. The naive version — accumulate
each token into an assign and re-render — works beautifully in a demo
and falls over in production for three specific reasons: it keeps paying
the model when the user navigates away mid-stream, it re-renders your
entire template on every single token, and it leaves a half-finished
message on screen when the upstream call dies. This post shows the
modern build using LiveView 1.1’s async primitives and
`Req`’s streaming `:into` callback, then fixes all
three failure modes the tutorials skip.

## The connection is already open — stop reaching for a second one

Every few weeks someone building their first AI feature in Phoenix
asks the same question: “How do I stream the model’s tokens into the UI
as they arrive?” And almost every answer they find points them somewhere
strange — spin up a Channel, bolt on a React island, proxy the OpenAI
stream through a Node sidecar.

None of that is necessary. A LiveView is a stateful server process
with a persistent connection to exactly one browser tab. That is the
entire problem already solved. The model emits tokens; you push them
down the wire that is already open. The only real work is wiring an HTTP
streaming response on one side to that socket on the other, and doing it
without lying to yourself about what happens when things go wrong.

If you want the deeper argument for why the BEAM is unusually
well-suited to this — cheap processes, per-user isolation, supervision —
I made it in [the BEAM is quietly
the best runtime for AI agents](https://sublimecoding.com/blog/elixir-beam-ai-agents) and, more fundamentally, in [how Elixir’s concurrency model
actually works](https://sublimecoding.com/blog/elixir-concurrency-model). This post is the concrete, runnable version of that
thesis for the single most common AI UI task.

A note on versions: the code below targets **Phoenix LiveView
1.1+** and **Req 0.5+**. The async callbacks
(`start_async/4`, `handle_async/3`,
`Phoenix.LiveView.AsyncResult`) and the `send/2` +
`handle_info/2` pattern are stable from 1.1 onward; I
verified them against the current LiveView docs (1.2.1 at the time of
writing) and they are unchanged. Req’s streaming `:into`
callback shape is likewise stable in the current 0.6.x line. The LLM
calls target OpenAI’s **Responses API**
(`POST /v1/responses`) — the current surface as of 2026, not
the older Chat Completions endpoint.

## The naive version that demos beautifully

Here is the version you will write first, and honestly it is a
reasonable starting point. The model’s tokens land in an assign, the
assign drives the template, LiveView ships the diff. Token appears.
Magic.

`[](#cb1-1)defmodule MyAppWeb.ChatLive do
[](#cb1-2) use MyAppWeb, :live_view
[](#cb1-3)
[](#cb1-4) def mount(_params, _session, socket) do
[](#cb1-5) {:ok, assign(socket, prompt: "", answer: "")}
[](#cb1-6) end
[](#cb1-7)
[](#cb1-8) def handle_event("ask", %{"prompt" => prompt}, socket) do
[](#cb1-9) parent = self()
[](#cb1-10)
[](#cb1-11) # Spawn a task that streams tokens back to us.
[](#cb1-12) Task.start(fn -> stream_completion(prompt, parent) end)
[](#cb1-13)
[](#cb1-14) {:noreply, assign(socket, prompt: prompt, answer: "")}
[](#cb1-15) end
[](#cb1-16)
[](#cb1-17) # Each token arrives as a message and gets appended to the assign.
[](#cb1-18) def handle_info({:token, text}, socket) do
[](#cb1-19) {:noreply, assign(socket, answer: socket.assigns.answer <> text)}
[](#cb1-20) end
[](#cb1-21)
[](#cb1-22) defp stream_completion(prompt, parent) do
[](#cb1-23) Req.post!("https://api.openai.com/v1/responses",
[](#cb1-24) headers: [{"authorization", "Bearer #{System.fetch_env!("OPENAI_API_KEY")}"}],
[](#cb1-25) json: %{
[](#cb1-26) model: "gpt-4o-mini",
[](#cb1-27) stream: true,
[](#cb1-28) input: prompt
[](#cb1-29) },
[](#cb1-30) into: fn {:data, data}, {req, resp} ->
[](#cb1-31) for token <- parse_sse_tokens(data), do: send(parent, {:token, token})
[](#cb1-32) {:cont, {req, resp}}
[](#cb1-33) end
[](#cb1-34) )
[](#cb1-35) end
[](#cb1-36)end
```

[](#cb2-1)<div class="chat">
[](#cb2-2) <form phx-submit="ask">
[](#cb2-3) <input type="text" name="prompt" value={@prompt} />
[](#cb2-4) <button type="submit">Ask</button>
[](#cb2-5) </form>
[](#cb2-6) <div class="answer">{@answer}</div>
[](#cb2-7)</div>
```

Run that locally, type a prompt, and you will watch text materialize
word by word. It feels like you shipped something real. You have not —
you have shipped a demo. Here is what it does the moment it meets actual
users.

## The three ways it breaks the instant it meets real users

**1. It keeps paying for tokens nobody will ever see.**
The user fires a long prompt, watches three sentences stream in, decides
it is wrong, and closes the tab. Their LiveView process terminates. But
the Task.start` process is *not* linked to it in any
way that matters here, and even if it were, `Task.start`
gives you no handle to cancel it. The `Req` request upstream
keeps the socket to OpenAI open and keeps pulling tokens — which you
keep being billed for — until the model emits its final token into a
mailbox that no longer has a reader. On a chat feature with any volume,
this is a line item.

**2. It re-renders the entire template on every token.**
`socket.assigns.answer <> text` replaces the whole
`answer` assign on each message. LiveView is smart about
diffing, but you have still made the *entire* `answer`
string a single changed assign, so the full text re-serializes and
re-diffs on every token. A 600-token response is 600 diffs of a string
that grows to 600 tokens long — quadratic work for what should be
append-only. Add a few concurrent users and you will see the LiveView’s
reduction count climb for no good reason.

**3. A mid-stream error leaves a corpse on screen.**
OpenAI returns a 500 four tokens in. `Req.post!` raises
inside the task. The task dies. Your LiveView never finds out — it just
stops receiving `{:token, _}` messages and sits there forever
showing half a sentence, no spinner, no error, no retry. The user
reloads and tries again, because you gave them nothing else to do.

All three share a root cause: the naive version models the happy path
and nothing else. The streaming part was never the hard part.
*Lifecycle* is the hard part. Let’s build the version that takes
it seriously.

## The right build: own the request, track its lifecycle, append cleanly

The fix starts with one decision: **wrap the LLM call behind a
project-owned module and give the LiveView a real handle to the work in
flight.** We use `start_async/4` so LiveView
supervises the task and tells us — via `handle_async/3` —
when it finishes or crashes. Inside that task, the streaming
`:into` callback forwards tokens back to the LiveView pid as
they arrive. The async result is the *terminal* signal (done, or
failed); the `send/2` messages are the *incremental*
stream. That split is the whole design.

First, the boundary module. Never call `Req` against a
third-party API directly from a LiveView — wrap it so the streaming
protocol, auth, and SSE parsing live in one testable place:

`[](#cb3-1)defmodule MyApp.LLM do
[](#cb3-2) @moduledoc "Project-owned boundary around the streaming completions API."
[](#cb3-3)
[](#cb3-4) @endpoint "https://api.openai.com/v1/responses"
[](#cb3-5)
[](#cb3-6) @doc """
[](#cb3-7) Streams a completion, sending `{:llm_token, ref, text}` messages to `dest`
[](#cb3-8) as tokens arrive. Returns `:ok` on a clean finish or raises on transport
[](#cb3-9) error (so the caller's async task surfaces it as `{:exit, reason}`).
[](#cb3-10) """
[](#cb3-11) def stream(prompt, dest, ref) do
[](#cb3-12) Req.post!(@endpoint,
[](#cb3-13) headers: [{"authorization", "Bearer #{api_key()}"}],
[](#cb3-14) json: %{
[](#cb3-15) model: "gpt-4o-mini",
[](#cb3-16) stream: true,
[](#cb3-17) input: prompt
[](#cb3-18) },
[](#cb3-19) # Bound the wait between chunks so a stalled upstream can't hang forever.
[](#cb3-20) receive_timeout: 30_000,
[](#cb3-21) into: fn {:data, data}, {req, resp} ->
[](#cb3-22) for text <- parse_sse(data) do
[](#cb3-23) send(dest, {:llm_token, ref, text})
[](#cb3-24) end
[](#cb3-25) {:cont, {req, resp}}
[](#cb3-26) end
[](#cb3-27) )
[](#cb3-28)
[](#cb3-29) :ok
[](#cb3-30) end
[](#cb3-31)
[](#cb3-32) # The Responses API streams typed Server-Sent Events: each is an
[](#cb3-33) # `event: <type>` line plus a `data: {json}` line. Text arrives in
[](#cb3-34) # `response.output_text.delta` events (the JSON carries a `delta`); the
[](#cb3-35) # stream ends with a `response.completed` event, not a `[DONE]` sentinel.
[](#cb3-36) # Matching on the payload's `type` means interleaved events (created,
[](#cb3-37) # output_text.done, completed) are simply ignored. Real code should
[](#cb3-38) # buffer partial lines across chunks; kept inline here for readability.
[](#cb3-39) defp parse_sse(chunk) do
[](#cb3-40) chunk
[](#cb3-41) |> String.split("\n")
[](#cb3-42) |> Enum.flat_map(fn
[](#cb3-43) "data: " <> json ->
[](#cb3-44) case Jason.decode(json) do
[](#cb3-45) {:ok, %{"type" => "response.output_text.delta", "delta" => text}} -> [text]
[](#cb3-46) _ -> []
[](#cb3-47) end
[](#cb3-48) _ -> []
[](#cb3-49) end)
[](#cb3-50) end
[](#cb3-51)
[](#cb3-52) defp api_key, do: System.fetch_env!("OPENAI_API_KEY")
[](#cb3-53)end
```

Now the LiveView. Note Phoenix.LiveView.AsyncResult`
driving the UI state machine, and that we accumulate tokens into an IO
list, not by repeatedly concatenating a string:

`[](#cb4-1)defmodule MyAppWeb.ChatLive do
[](#cb4-2) use MyAppWeb, :live_view
[](#cb4-3) alias Phoenix.LiveView.AsyncResult
[](#cb4-4)
[](#cb4-5) def mount(_params, _session, socket) do
[](#cb4-6) {:ok,
[](#cb4-7) socket
[](#cb4-8) |> assign(:prompt, "")
[](#cb4-9) |> assign(:stream_ref, nil)
[](#cb4-10) |> assign(:tokens, []) # IO list of received tokens
[](#cb4-11) |> assign(:completion, AsyncResult.loading() |> AsyncResult.ok(:idle))}
[](#cb4-12) end
[](#cb4-13)
[](#cb4-14) def handle_event("ask", %{"prompt" => prompt}, socket) do
[](#cb4-15) # A ref lets us ignore stray tokens from a request we've since abandoned.
[](#cb4-16) ref = make_ref()
[](#cb4-17) pid = self()
[](#cb4-18)
[](#cb4-19) socket =
[](#cb4-20) socket
[](#cb4-21) |> assign(:prompt, prompt)
[](#cb4-22) |> assign(:stream_ref, ref)
[](#cb4-23) |> assign(:tokens, [])
[](#cb4-24) |> assign(:completion, AsyncResult.loading())
[](#cb4-25) # start_async supervises the task and routes its result to handle_async/3.
[](#cb4-26) |> start_async(:completion, fn ->
[](#cb4-27) MyApp.LLM.stream(prompt, pid, ref)
[](#cb4-28) end)
[](#cb4-29)
[](#cb4-30) {:noreply, socket}
[](#cb4-31) end
[](#cb4-32)
[](#cb4-33) # Incremental: one token arrived. Only append if it belongs to the
[](#cb4-34) # current request — a late token from an abandoned stream is dropped.
[](#cb4-35) def handle_info({:llm_token, ref, text}, %{assigns: %{stream_ref: ref}} = socket) do
[](#cb4-36) {:noreply, update(socket, :tokens, &[&1 | text])}
[](#cb4-37) end
[](#cb4-38)
[](#cb4-39) def handle_info({:llm_token, _stale_ref, _text}, socket) do
[](#cb4-40) {:noreply, socket} # token from a cancelled/superseded request
[](#cb4-41) end
[](#cb4-42)
[](#cb4-43) # Terminal: the task returned cleanly. The stream is done.
[](#cb4-44) def handle_async(:completion, {:ok, :ok}, socket) do
[](#cb4-45) {:noreply, assign(socket, :completion, AsyncResult.ok(socket.assigns.completion, :done))}
[](#cb4-46) end
[](#cb4-47)
[](#cb4-48) # Terminal: the task crashed mid-stream (HTTP 500, timeout, etc).
[](#cb4-49) def handle_async(:completion, {:exit, reason}, socket) do
[](#cb4-50) {:noreply, assign(socket, :completion, AsyncResult.failed(socket.assigns.completion, reason))}
[](#cb4-51) end
[](#cb4-52)end
```

[](#cb5-1)<div class="chat">
[](#cb5-2) <form phx-submit="ask">
[](#cb5-3) <input type="text" name="prompt" value={@prompt} />
[](#cb5-4) <button type="submit" disabled={@completion.loading}>Ask</button>
[](#cb5-5) </form>
[](#cb5-6)
[](#cb5-7) <div class="answer" id="answer">{@tokens}</div>
[](#cb5-8)
[](#cb5-9) <.async_result :let={_status} assign={@completion}>
[](#cb5-10) <:loading><span class="cursor">▌</span></:loading>
[](#cb5-11) <:failed :let={_reason}>
[](#cb5-12) <p class="error">
[](#cb5-13) The model dropped the connection.
[](#cb5-14) <button phx-click="ask" phx-value-prompt={@prompt}>Retry</button>
[](#cb5-15) </p>
[](#cb5-16) </:failed>
[](#cb5-17) </.async_result>
[](#cb5-18)</div>
```

This is already a different animal. The start_async` task
is supervised by the LiveView; if the LiveView dies, the task dies with
it. `handle_async` gives us a real terminal event, so a
crashed stream flips the UI into a failed state with a retry button
instead of freezing. The `ref` guard means tokens from an
abandoned request are silently dropped instead of corrupting the next
answer.

But “the task dies with the LiveView” is necessary, not sufficient.
Killing the task does not, by itself, hang up on OpenAI. That is failure
mode one, and it deserves its own section.

## Cancellation done right: hang up on the model, not just the socket

Here is the subtle, expensive part. When `start_async`’s
task is killed — because the LiveView terminated, because the user hit
“stop,” because they navigated away — the task process disappears. But
the `Req` request running *inside* that task opened a
TCP connection to OpenAI and is mid-stream. Does killing the task close
that connection?

It depends entirely on whether the HTTP work happens in the task’s
own process. With `Req`’s default function-form
`:into` (the shape above), the request is driven
synchronously inside the calling process — the `start_async`
task itself. When the BEAM kills that process, the socket it owns is
closed as part of process teardown, which tears down the upstream
connection. That is the behavior you want, and it is the reason to keep
the `Req` call *inside* the `start_async`
function rather than handing it off to some longer-lived pool
worker.

The trap is `into: :self`. That mode spawns a
*separate* process to pump the socket and mails chunks to you.
Kill your task and that helper can outlive it, happily draining (and
billing) the full response into a mailbox nobody reads. For LiveView
token streaming, prefer the function-form `:into` so the
request’s lifetime is bound to the task’s lifetime. Cancellation becomes
free: it is just process death.

For the explicit “stop generating” button, cancel the named async and
let teardown do the rest:

`[](#cb6-1)def handle_event("stop", _params, socket) do
[](#cb6-2) # cancel_async kills the supervised task; its Req socket closes on teardown,
[](#cb6-3) # which closes the upstream connection. We also clear the ref so any
[](#cb6-4) # tokens already in our mailbox are ignored on arrival.
[](#cb6-5) {:noreply,
[](#cb6-6) socket
[](#cb6-7) |> cancel_async(:completion)
[](#cb6-8) |> assign(:stream_ref, nil)
[](#cb6-9) |> assign(:completion, AsyncResult.ok(socket.assigns.completion, :cancelled))}
[](#cb6-10)end
```

For the navigate-away case you usually do not need to write anything:
when the client disconnects, the LiveView process terminates, which
kills its start_async` task, which closes the
`Req` socket. The chain holds *as long as the HTTP request
lives inside the supervised task.* If you have moved the LLM call
into a separate long-lived GenServer or a pooled worker — a reasonable
thing to do for other reasons — then you have severed that chain and you
are back to leaking requests. In that case you must trap the LiveView’s
`terminate/2` (or monitor the LiveView pid from the worker)
and explicitly cancel the in-flight request. The general rule:
**the process that owns the HTTP socket must die when the user
goes away.** Architect so that happens for free, or wire it up
explicitly. There is no third option that does not leak money.

If you want belt-and-suspenders confirmation in dev, watch your
provider dashboard’s active-request count while you spam open-and-close
on a long prompt. If it climbs and never settles, your cancellation
chain is broken somewhere.

## Backpressure and the re-render storm — and the actual fix

Now failure mode two: re-rendering the world on every token. The
naive version stored the answer as one big string and rebuilt it per
token. Two things fix this, and they compose.

**Append, don’t concatenate.** The right build above
stores tokens as an IO list: `[&1 | text]` conses the
accumulated iodata with each new fragment in O(1) and renders in arrival
order — no `Enum.reverse` needed. The point is you are no
longer rebuilding an ever-growing binary on the server every message the
way `answer <> text` did; that concatenation was the
quadratic cost. Shrinking what actually goes over the wire on each diff
is the next paragraph’s job.

**Render the streamed region with
`phx-update="ignore"` and let the client own appends, or
batch on the server.** Two legitimate strategies:

The server-light approach adds `phx-update="ignore"` to
that answer div: LiveView stops diffing the node after first render, and
a small client hook appends incoming token text directly to the DOM. The
server still sends tokens, but it is not re-diffing a growing block on
every one. This is the lowest-overhead option for high token rates.

The server-only approach, if you would rather not write a hook, is to
**batch tokens before assigning.** Do not push one diff per
token — buffer tokens for, say, 50ms or 20 tokens, whichever comes
first, then flush one update. The user cannot perceive sub-50ms
batching, but your reduction count drops by an order of magnitude:

`[](#cb7-1)# In the LiveView: accumulate, and flush on a short timer instead of per-token.
[](#cb7-2)def handle_info({:llm_token, ref, text}, %{assigns: %{stream_ref: ref}} = socket) do
[](#cb7-3) socket = update(socket, :buffer, &[&1 | text])
[](#cb7-4) # Schedule a single flush if one isn't already pending.
[](#cb7-5) socket =
[](#cb7-6) if socket.assigns.flush_pending do
[](#cb7-7) socket
[](#cb7-8) else
[](#cb7-9) Process.send_after(self(), {:flush, ref}, 50)
[](#cb7-10) assign(socket, :flush_pending, true)
[](#cb7-11) end
[](#cb7-12)
[](#cb7-13) {:noreply, socket}
[](#cb7-14)end
[](#cb7-15)
[](#cb7-16)def handle_info({:flush, ref}, %{assigns: %{stream_ref: ref}} = socket) do
[](#cb7-17) {:noreply,
[](#cb7-18) socket
[](#cb7-19) |> update(:tokens, &[&1 | socket.assigns.buffer])
[](#cb7-20) |> assign(:buffer, [])
[](#cb7-21) |> assign(:flush_pending, false)}
[](#cb7-22)end
```

Either way, the principle is the same: **the rate at which the
model emits tokens should not equal the rate at which you
re-render.** Decouple them. A model that emits 80 tokens/second
does not need 80 LiveView diffs/second; it needs a smooth-looking
stream, which 20 batched updates/second delivers indistinguishably while
doing a quarter of the work. Multiply that saving across every
concurrent chat session and the difference is the number of users one
node can hold.

There is no Req`-level backpressure knob to reach for
here — the model sets the pace and you cannot ask it to slow down.
Backpressure in this design means *not amplifying* the model’s
token rate into an equal-or-greater render rate on your side. Batching
is how you absorb it.

## When NOT to build it this way

I am bullish on LiveView for this. I am not unconditional about
it.

**Skip LiveView if you genuinely need offline or sub-frame
client rendering.** A native mobile app, an offline-capable PWA,
or anything where the UI must keep updating with the network cut —
LiveView’s whole model is a live server connection, so there is no
server to stream from when you are offline. Run the model client-side or
against a local edge and render in whatever owns the device. LiveView is
the wrong layer for that, full stop.

**Reconsider at extreme fan-out.** If a *single*
generated stream must be delivered to thousands of simultaneous viewers
— a broadcast, not a per-user chat — then one LiveView process per
viewer, each holding its own assigns, is a lot of duplicated state. That
is the case where a dedicated `Phoenix.Channel` (or a PubSub
topic that all viewers subscribe to, with the model streamed once into
the topic) earns its complexity. For the overwhelmingly common case —
one user, one private generation — that machinery is pure overhead, and
the per-user LiveView process is exactly right.

**Skip the streaming UI entirely for short, non-interactive
completions.** If the model returns two sentences in 400ms, token
streaming is theater. Show a spinner, await the result with a plain
`assign_async/4`, render it whole. Streaming earns its
lifecycle complexity only when the response is long enough that watching
it arrive is genuinely better than waiting for it.

## Verdict

Streaming LLM tokens into a Phoenix UI is not a React problem, a
Channel problem, or a sidecar problem. It is a process-lifecycle problem
wearing a streaming-UI costume. The connection you need is already open;
the supervision you need is already in the box. The same is true in the other direction, for data going *to* the model: [wiring LiveView uploads to Claude Vision](https://sublimecoding.com/blog/liveview-uploads-claude-vision) is an upload-lifecycle problem wearing a multimodal one.

Build it with `start_async/4` so the work is supervised
and its terminal state is a real event you can render. Stream tokens
back over `send/2` and append them — as an IO list or via a
client hook — so you are not rebuilding the answer on every message.
Keep the `Req` call inside the supervised task so
cancellation is free: when the user leaves, the process dies, the socket
closes, the model stops billing you. Batch your re-renders so the
model’s token rate is not your diff rate. Do those four things and the
version that demos well is also the version that survives contact with
real traffic.

The naive version is not wrong, exactly. It is just unfinished — it
solved streaming and skipped lifecycle, which is the actual job. Finish
it.

*Two follow-ups are coming soon: running these generations as
durable, retryable Oban jobs when they outlive a page view, and the RAG
retrieval layer that feeds the prompt before any of this streaming
begins. Both build directly on the boundary module and supervision
patterns above.*

---

## RAG in Phoenix: Hand-Rolled pgvector or Arcana?

URL: https://sublimecoding.com/blog/phoenix-rag-pgvector-arcana
Published: 2026-06-18
Tags: Elixir, AI

>
**TL;DR:** RAG needs exactly three things: a place to
put vectors, a way to fill it, and one query that returns nearest
neighbors. The Postgres you are already running does all three through
pgvector, and the hand-rolled Phoenix version is one migration, one
schema change, and one Ecto query — all of it below. The new Elixir RAG
libraries (Arcana, rag_ex) are real and worth your attention, but they
solve a different problem than the one people reach for them to solve:
they own the *pipeline* — chunking, query rewriting, reranking,
evals — not the storage. Figure out which problem you actually have and
the decision makes itself. Here’s the honest version of both paths,
including where Postgres itself is the wrong call.

## You don’t have a vector database problem

The first decision most teams get wrong about RAG happens before any
code gets written: they go shopping for a vector database. It’s an
understandable reflex — nearly every RAG tutorial is written in Python,
and the Python ecosystem’s default posture is “stand up Pinecone,
Qdrant, or Chroma first, ask questions later.” So the Phoenix developer
dutifully provisions a second stateful service before writing line
one.

Stop and count what that second service actually costs you. It’s
another thing to deploy, monitor, back up, and upgrade. It’s a second
source of truth, which means a synchronization problem: when a document
is updated or deleted in Postgres, something has to guarantee its
embeddings change in the other store, and that something is now code you
own, with failure modes you own. And it’s a network hop in the middle of
your hottest read path.

Now count what it buys you at your scale. If you’re doing what most
product teams are actually doing — Q&A over your docs, semantic
search across support tickets, “find related items” over a catalog —
your corpus is thousands to low millions of chunks. Postgres with the [pgvector
extension](https://github.com/pgvector/pgvector-elixir) handles that range comfortably, with real indexes (HNSW),
real distance operators, and one property no bolt-on vector store can
offer: your embeddings live in the same database as your data, deleted
in the same `DELETE`, written in the same transaction. The
sync problem doesn’t get solved. It stops existing.

This is the same argument I made about [cutting a
service footprint from ten to six](https://sublimecoding.com/blog/ruby-to-elixir-migration-ten-to-six-services), pointed at a different layer:
every piece of infrastructure you don’t run is operational surface you
don’t pay for. The BEAM lets you collapse app servers; pgvector lets you
collapse the data layer back to one database. Take the win.

## The hand-rolled path: a migration, a schema, a query

Here is the entire storage layer, end to end. I want you to see how
little there is, because the size of this section is itself the
argument.

Add the dependency:

`[](#cb1-1)# mix.exs
[](#cb1-2){:pgvector, "~> 0.4"}
```

Teach Postgrex about the vector type, and point your Repo at it:

[](#cb2-1)# lib/my_app/postgrex_types.ex
[](#cb2-2)Postgrex.Types.define(
[](#cb2-3) MyApp.PostgrexTypes,
[](#cb2-4) Pgvector.extensions() ++ Ecto.Adapters.Postgres.extensions(),
[](#cb2-5) []
[](#cb2-6))
[](#cb2-7)
[](#cb2-8)# config/config.exs
[](#cb2-9)config :my_app, MyApp.Repo, types: MyApp.PostgrexTypes
```

One migration enables the extension, creates a chunks table, and adds
the index. Use HNSW unless you have a measured reason not to — it builds
slower than IVFFlat but doesn’t need retraining as data grows, and query
speed is what you’ll feel:

[](#cb3-1)def up do
[](#cb3-2) execute "CREATE EXTENSION IF NOT EXISTS vector"
[](#cb3-3)
[](#cb3-4) create table(:chunks) do
[](#cb3-5) add :document_id, references(:documents, on_delete: :delete_all), null: false
[](#cb3-6) add :body, :text, null: false
[](#cb3-7) add :embedding, :vector, size: 1536
[](#cb3-8) timestamps()
[](#cb3-9) end
[](#cb3-10)
[](#cb3-11) execute """
[](#cb3-12) CREATE INDEX chunks_embedding_idx ON chunks
[](#cb3-13) USING hnsw (embedding vector_cosine_ops)
[](#cb3-14) """
[](#cb3-15)end
```

Note the on_delete: :delete_all`. That single option is
the entire “keep the vector store in sync” subsystem you’d otherwise be
writing.

The schema:

`[](#cb4-1)defmodule MyApp.RAG.Chunk do
[](#cb4-2) use Ecto.Schema
[](#cb4-3)
[](#cb4-4) schema "chunks" do
[](#cb4-5) field :body, :string
[](#cb4-6) field :embedding, Pgvector.Ecto.Vector
[](#cb4-7) belongs_to :document, MyApp.RAG.Document
[](#cb4-8) timestamps()
[](#cb4-9) end
[](#cb4-10)end
```

Ingestion is the part with actual decisions in it, but the mechanics
are small: split the document, embed each piece, insert.
text-embedding-3-small` is the boring, correct default —
cheap enough that re-embedding your whole corpus when you change
chunking strategy (you will) costs pocket change:

`[](#cb5-1)defp embed!(texts) when is_list(texts) do
[](#cb5-2) Req.post!("https://api.openai.com/v1/embeddings",
[](#cb5-3) auth: {:bearer, System.fetch_env!("OPENAI_API_KEY")},
[](#cb5-4) json: %{model: "text-embedding-3-small", input: texts}
[](#cb5-5) ).body["data"]
[](#cb5-6) |> Enum.map(& &1["embedding"])
[](#cb5-7)end
[](#cb5-8)
[](#cb5-9)def ingest!(document) do
[](#cb5-10) chunks = split(document.body, max_tokens: 500, overlap: 50)
[](#cb5-11)
[](#cb5-12) chunks
[](#cb5-13) |> embed!()
[](#cb5-14) |> Enum.zip(chunks)
[](#cb5-15) |> Enum.map(fn {embedding, body} ->
[](#cb5-16) %{document_id: document.id, body: body, embedding: Pgvector.new(embedding)}
[](#cb5-17) end)
[](#cb5-18) |> then(&MyApp.Repo.insert_all(MyApp.RAG.Chunk, &1))
[](#cb5-19)end
```

Run that in an Oban job, not in the request — embedding calls are
exactly the kind of flaky, retryable external work job queues exist
for.

And retrieval is one query:

[](#cb6-1)import Pgvector.Ecto.Query
[](#cb6-2)
[](#cb6-3)def retrieve(question, k \\ 5) do
[](#cb6-4) [query_embedding] = embed!([question])
[](#cb6-5)
[](#cb6-6) from(c in MyApp.RAG.Chunk,
[](#cb6-7) order_by: cosine_distance(c.embedding, ^Pgvector.new(query_embedding)),
[](#cb6-8) limit: ^k
[](#cb6-9) )
[](#cb6-10) |> MyApp.Repo.all()
[](#cb6-11)end
```

That’s it. Interpolate the top-k chunk bodies into your prompt, send
it to the model, you have RAG. The whole thing is maybe 150 lines
including the chunker, it’s all code a mid-level Elixir developer can
read in one sitting, and — this is the underrated part — it’s
*Ecto*. Want only chunks from documents the current user can see?
Add a join` and a `where`. That composability is
something the dedicated vector stores make you reimplement through their
metadata-filter DSLs.

## Where the hand-rolled version stops

Now the honest part. What you built above is *retrieval*. The
gap between retrieval and a RAG system your users describe as “good”
lives almost entirely outside that Ecto query, and it’s worth naming the
pieces, because each one looks like an afternoon and the sum is a
quarter:

- **Chunking strategy.** Naive fixed-size splitting is
why most RAG demos disappoint. Respecting document structure — headings,
paragraphs, code blocks — matters more to answer quality than any index
tuning you will ever do.

- **Query rewriting.** Users ask “why is it slow?”; the
chunk that answers says “latency regression in the connection pool.”
Embedding the raw question and hoping is the weakest link in the naive
pipeline. Good systems rewrite, expand with synonyms, and split
multi-part questions into focused sub-queries.

- **Reranking and filtering.** Top-5 by cosine distance
includes near-duplicates and confidently-irrelevant chunks. A scoring
pass that filters them is the difference between a model that answers
and a model that hedges.

- **Evals.** Without a question set scored against
expected sources, every one of the above is vibes-driven development.
This is the same discipline argument as [instrumenting
your AI product before reaching for a better model](https://sublimecoding.com/blog/your-ai-product-needs-telemetry-before-better-model) — you cannot tune
what you don’t measure.

You can absolutely build all of this yourself. I have. But recognize
that the moment you start, you’ve left “an afternoon with pgvector” and
entered pipeline engineering — and that’s the actual decision point, not
SQL versus vector DB.

## What Arcana and rag_ex actually buy you

This is where the new Elixir libraries come in, and the first thing
worth noticing is what they *didn’t* build: storage. [Arcana](https://github.com/georgeguimaraes/arcana) is
explicitly embeddable — it plugs into the Ecto Repo and Postgres you
already have, pgvector underneath, no separate service, no indexing
daemon. [rag_ex](https://github.com/nshkrdotcom/rag_ex) (a
fork of bitcrowd’s `rag`) takes the same posture with
pluggable vector stores. The ecosystem looked at the Python default of
“stand up another database” and declined, which tells you the argument
in the first section isn’t just mine.

What they did build is the pipeline. Arcana ships the agentic
retrieval loop — query rewriting, sub-query splitting, per-chunk
relevance scoring and filtering, answer generation — plus ingestion for
real-world document formats and a dashboard to watch it work. rag_ex
adds multi-LLM routing and GraphRAG-style knowledge-graph retrieval if
your corpus has structure worth exploiting. Every bullet in the previous
section, somebody already wrote, in Elixir, against the database you
already run.

The trade is the usual one with frameworks: you adopt their opinions.
Their chunking, their pipeline shape, their schema in your database.
When your retrieval problem fits those opinions, that’s months of
pipeline engineering for free. When it doesn’t, you’ll be reading
library source to find the extension point — both projects are young,
and young libraries’ extension points are where the sharp edges
live.

## The decision

**Hand-roll when** the corpus is one kind of thing, the
questions are direct, and answer quality at “good top-k retrieval” is
good enough — docs Q&A, related-content, semantic search where the
user sees a result list rather than a synthesized answer. You’ll own 150
transparent lines, you’ll understand every moving part, and you’ll have
no dependency on a pre-1.0 library’s roadmap. This is also unambiguously
the right first move if your team hasn’t built RAG before: you can’t
evaluate what a pipeline library is doing for you until you’ve felt the
failure modes it exists to fix.

**Reach for Arcana when** the pipeline gap is the
product gap — when users ask multi-part questions, when ingestion means
PDFs and wikis and tickets rather than clean markdown, when you’re about
to spend a sprint on reranking and evals you could adopt instead of
write. The embeddable design means the migration path is gentle: it’s
the same Postgres, so trying it isn’t a re-platforming.

The quiet good news is that this isn’t a one-way door. Both paths put
your vectors in the same database, behind the same Repo. Moving from
hand-rolled to Arcana — or back — is a refactor, not a data migration.
That is *not* true of the Pinecone path, which is precisely why I
keep insisting the storage decision is the one to get right first.

## Where Postgres is the wrong call

Fairness requires the boundary. pgvector stops being the answer when
vector search *is* the product at scale: tens of millions of
embeddings, high-QPS approximate-nearest-neighbor with heavy metadata
filtering, recall targets you tune weekly, sharding across nodes. The
dedicated stores earn their operational cost there — that workload is
what they’re for. If you’re building semantic search as the core of the
business rather than a feature of it, [hire for that infrastructure](https://sublimecoding.com/blog/nadella-token-capital-small-teams)
honestly.

But be suspicious of arriving at that conclusion early. “We might
need to scale” is how teams end up operating three databases for a
corpus that fits in Postgres’s shared buffers. Measure first; the
migration is real but it’s a known road, and you’ll travel it with evals
and production query logs you didn’t have on day one.

## The BEAM was already good at this

One closing observation. A RAG request is an IO-bound fan-out: an
embedding call, a database query, a model call, sometimes several of
each in parallel under one user interaction that lives for seconds. That
shape — many concurrent, slow, failure-prone external calls per user,
isolated from every other user — is [the workload the BEAM’s
concurrency model was built for](https://sublimecoding.com/blog/elixir-concurrency-model), the same reason [it’s the runtime that fits
agents](https://sublimecoding.com/blog/elixir-beam-ai-agents). Python teams buy infrastructure to get isolation and
concurrency around their pipelines. In Phoenix, the pipeline runs
*inside* the same runtime properties, against the same database,
in one deployable.

The stack you already run was the right one. Add a migration and find
out.

---

## Why We'd Pick Elixir for an AI Startup Backend

URL: https://sublimecoding.com/blog/why-elixir-for-ai-startup-backend
Published: 2026-07-23
Tags: Elixir, AI startups, founders

>
**TL;DR:** For the shape most AI product backends
actually have — many concurrent, independently-failing, stateful,
long-running operations: agents holding sessions, tokens streaming to a
UI, six flaky tool calls per turn, multi-step jobs that must survive a
crash — the BEAM’s failure-isolation and concurrency model is a
genuinely better fit than the Python or Node default you’d reach for by
reflex. That’s the whole argument, and the deeper posts in this cluster
each prove one piece of it. But this is a decision, not a love letter:
the model and ML layer is still Python’s and will be for years, your
hiring pool is smaller, and if your backend is mostly stateless CRUD
with an orchestrator already handling supervision, Elixir buys you
little. Here’s the founder-altitude version — what the BEAM wins, what
it costs, and the table you can actually decide from.

## Your AI backend is not a CRUD app

When a founder asks me “should *we* use Elixir for our AI
thing?”, the useful first move is to ignore the language entirely and
describe the *shape* of the system they’re building. Because the
language argument is downstream of the shape, and the shape of an AI
product backend is genuinely different from the shape of the SaaS apps
most stacks were chosen for.

A typical AI backend, stripped of the marketing, is a crowd of
long-lived, stateful, independently-failing operations running at
once:

- **Agents** that hold a session — the conversation, the
scratchpad, the half-finished plan — for seconds or for hours, making a
sequence of unreliable calls the whole time.

- **Token streams** pushing model output to a live UI as
it arrives, one open connection per active user, all of them expecting
sub-second responsiveness.

- **Tool calls** — six per turn, each one a flaky HTTP
request to a third-party API that rate-limits, times out, or returns
malformed JSON because a stochastic model asked for it badly.

- **Durable multi-step jobs** — the ingestion pipeline,
the batch summarization, the agent run that has to survive a deploy and
a crash and still finish exactly once.

- **RAG** retrieval feeding all of the above, embeddings
and vector search sitting in the hot path of every answer.

Look at that list and notice what it is *not*: it is not a
request that comes in, hits a database, renders a template, and returns
in 40 milliseconds. It’s a swarm of concurrent things that each live a
long time, hold state, and fail constantly and independently of one
another. And that shape stresses exactly the property that Python and
Node concurrency are weakest at: keeping thousands of independent
failures *isolated*, so one agent’s bad afternoon doesn’t take
down the four hundred sessions running next to it.

That’s not a knock on Python the language. It’s a statement about the
runtime. In a shared-address-space model — goroutines, async event
loops, threads — an unhandled failure in one in-flight operation has a
blast radius bigger than that operation. You paper over it with
defensive `try/catch` at every boundary, hand-rolling,
imperfectly, the isolation you wish you had for free. I went deep on
exactly that failure mode in [Elixir’s concurrency model](https://sublimecoding.com/blog/elixir-concurrency-model),
and on why the agent workload in particular is the BEAM’s home turf in
[the BEAM is the runtime AI agents
want](https://sublimecoding.com/blog/elixir-beam-ai-agents). The short version for a founder: your backend’s defining
characteristic is “individual units fail constantly and unpredictably,”
and isolated-fate concurrency is the correct default for that — not the
exception you bolt on.

## The four things the BEAM gets right for this shape

There are exactly four properties that make the BEAM fit an AI
backend, and each one answers a problem the workload above creates. None
of them is an AI feature — that’s the point. I’ll make each case briefly
and hand you to the post that proves it.

### 1. Failure isolation: “let it crash” is the right policy for flaky LLM and tool calls

The single defining fact of an AI backend is that its operations fail
all the time, in ways you didn’t anticipate, because the failure is
coming from a model deciding to emit malformed JSON or a third-party
tool having a bad afternoon. The *correct* response to that is
almost never “take down the server.” It’s “this one operation failed;
retry it, reset it to its last good state, or let this one die — and
leave everything else completely untouched.”

On the BEAM that’s not a pattern you build. It’s the runtime’s native
behavior. Every process has its own heap and its own crash domain; when
one dies, its siblings never notice. Supervision trees — the thing
Ericsson built for telephone switches that weren’t allowed to go down —
turn “let it crash” into a *declared* retry strategy instead of
defensive code smeared through your business logic. An agent step that
fails on a bad model response *should* crash and restart from its
checkpoint; that’s the designed-for case, not a workaround. That argument stopped being only mine when [Together AI raised $800M on a stack that leans the same way](https://sublimecoding.com/blog/together-ai-elixir).

The deep version of this argument lives in two posts: the mechanics
of why isolated processes beat shared-fate concurrency in [Elixir’s concurrency model](https://sublimecoding.com/blog/elixir-concurrency-model),
and why the agent workload specifically is the shape the BEAM was
purpose-built for in [the BEAM is
the runtime AI agents want](https://sublimecoding.com/blog/elixir-beam-ai-agents). If you read one thing past this hub,
read those.

### 2. Durable, idempotent, retry-aware multi-step agent runs

An agent run is not one model call. It’s a sequence — plan, retrieve,
call a tool, call another, summarize, write a result — that has to
survive a deploy, a crash, and a flaky network, and finish *exactly
once* even if a step gets retried three times. That’s a
durable-execution problem, and it’s where a lot of teams reach for
Temporal or a hand-rolled job table with a state machine bolted on.

On Elixir you mostly already have the answer in [Oban](https://sublimecoding.com/blog/oban-durable-ai-agent-runtime): a Postgres-backed
job runtime with retries, backoff, uniqueness, and persistence that
survives restarts — running in the same node as your agents, no separate
orchestration cluster to operate. The discipline it forces is the
discipline you want anyway: jobs must be idempotent, their arguments are
plain serializable data, and a step that fails reschedules itself
instead of losing the run. That post is the one to send an engineer who
asks “but how do you make agent runs durable without standing up
Temporal?” — it treats Oban as the durable agent runtime, which for a
startup-sized team is a real reduction in moving parts.

### 3. First-class streaming UIs without a separate frontend stack

Streaming model tokens to a user as they generate is table stakes for
an AI product now, and in most stacks it’s a project: a websocket layer,
a separate frontend framework, a message protocol between them, state
reconciliation when a connection drops. On Phoenix it’s close to free,
because LiveView already holds a stateful connection to every user and
async assigns push tokens to the browser as they arrive — the agent and
the UI live in the same supervised tree, so there’s no wire to design
between them.

The concrete patterns — backpressure, partial-message rendering, what
happens when the user navigates away mid-stream — are in [streaming LLM tokens
with LiveView in 2026](https://sublimecoding.com/blog/streaming-llm-tokens-liveview-2026). The broader production discipline of building
LiveView UIs that don’t fall over under real traffic is in [Phoenix LiveView
patterns for production](https://sublimecoding.com/blog/phoenix-liveview-patterns-production). For a founder the takeaway is a headcount
one: the streaming-UI problem that costs a typical team a frontend
specialist and a websocket service is, on this stack, a property of the
framework you already have.

### 4. Exposing your app to agents safely, a lean durable dependency stack, and RAG without a framework

Three smaller wins that compound. First, the moment your product is
good, someone wants an agent to *use* it — which means exposing
your app over the Model Context Protocol, safely, with authorization on
every tool call. Doing that natively in the same app that owns the data,
rather than as a bolted-on gateway, is the subject of [building an MCP server in
Phoenix with Hermes](https://sublimecoding.com/blog/build-mcp-server-phoenix-hermes).

Second, the dependency stack. AI startups accrete dependencies fast,
and every one is supply-chain surface and operational weight. The Elixir
libraries I actually reach for — and the ones I deliberately don’t — are
in [the Elixir
libraries I reach for in 2026](https://sublimecoding.com/blog/elixir-libraries-i-reach-for-2026). The theme is leanness: a small,
durable set of well-supported libraries beats a sprawling framework you
have to keep feeding.

Third, RAG. You do not need a vector framework to do
retrieval-augmented generation. Postgres with pgvector, sitting in the
database you already run, handles it for most products — the comparison
of doing it yourself versus reaching for a managed layer is in [Phoenix RAG with pgvector vs
Arcana](https://sublimecoding.com/blog/phoenix-rag-pgvector-arcana). For an early-stage team, “one fewer system to operate” is
frequently worth more than the marginal feature a dedicated vector store
buys you.

## The honest costs

I’d be doing you a disservice if this read like the BEAM has no
downside. It has three that matter, and pretending otherwise is how
founders talk themselves into a stack they’ll resent.

**The model and ML layer is not the BEAM’s, and won’t be
soon.** If the center of mass of your system is the *model
itself* — you’re training, fine-tuning, running local inference,
living in tensor math and eval harnesses — that’s Python’s world, full
stop. Nx and Bumblebee are genuinely impressive and you can run a
Llama-class model from Elixir today, but the frontier models, the
day-one SDKs, the research code, the sheer gravity of the ecosystem are
Python and will stay there for years. The flip side, and the reason this
matters less than it sounds for most products: in the overwhelmingly
common case your “model call” is an HTTPS request to a hosted API, which
is identical in every language. I argued the language-and-ecosystem
angle of this — including why even *writing* Elixir with AI
assistance is more workable than people assume — in [why AI codes Elixir best](https://sublimecoding.com/blog/elixir-ai-codes-best), and the
three-way split of who-owns-which-layer in [why every AI agent framework is
written in Go](https://sublimecoding.com/blog/go-ai-agent-frameworks).

**The hiring pool is smaller.** You will hire Elixir
engineers more slowly than Python or Node engineers. For some teams that
single fact outweighs every architectural argument above, and it should
— be honest about whether you’re optimizing for the system’s properties
or your own enjoyment of them. The mitigating reality is that the
engineers who *do* write Elixir tend to be senior and the
language is unusually learnable for a strong generalist, so “we can’t
hire for it” is more often “we haven’t tried” than a hard wall. But it’s
a real cost on day one and you should price it in.

**There are Erlang-isms, and it’s not a
number-cruncher.** The error messages can be cryptic, some
libraries are thin, and you’ll occasionally be the first person to hit a
rough edge. And the BEAM is not built for raw CPU-bound throughput — if
your hard problem is crunching numbers in a tight loop rather than
orchestrating concurrent I/O, you’ve picked the wrong runtime.

Notice the costs all cluster in the same place: the model and compute
layer. The BEAM’s weaknesses are precisely Python’s strengths, which is
why the honest architecture for a serious system is frequently
*both* — let the BEAM supervise the agents and serve the UI, let
Python run the model, and put a wire between them.

## When NOT to pick Elixir for your AI backend

Skip Elixir — genuinely, no hedging — if any of these describe
you:

- **Your product *is* the model.** You’re a
research or ML-heavy shop where the backend exists to serve training,
fine-tuning, and local inference. That’s Python’s job. Don’t fight the
gravity.

- **You’re a tiny team that only knows Node and ships next
week.** The most pragmatic stack is frequently the boring one you
can already staff and move fast in. A stack you have to learn under
deadline pressure is a tax you can’t afford pre-product-market-fit. The
[boring stack ships](https://sublimecoding.com/blog/ruby-boring-ships) argument
applies here in full.

- **You’re hard-constrained on hiring.** If you know
you’ll need to scale headcount fast in a market where you can only find
Python or Node people, the smaller talent pool can dominate every
technical advantage. Architecture you can’t staff is architecture you
don’t have.

- **Your backend is mostly stateless CRUD and someone else
handles supervision.** If your “AI” is a thin wrapper over a
hosted API, your operations are short and stateless, and an orchestrator
like Temporal or your serverless platform already gives you durability
and retries, the BEAM’s core advantage — isolating and supervising
massive numbers of long-lived stateful things — simply doesn’t apply to
you. Use the stack your team is fastest in.

The pattern in all four: the BEAM’s advantage is specifically
*stateful, concurrent, failure-prone, long-lived* operations. The
less your backend looks like that, the less reason there is to leave the
stack you already know.

## A founder’s decision framework

Here’s the table I’d actually use. Find the row that matches your
backend’s *dominant* shape — not the shape you wish it had, the
one it actually has — and read across.

Your dominant backend shape
Does the BEAM’s advantage apply?
Verdict

Many long-lived stateful **agents** holding sessions,
calling flaky tools, failing independently
**Strongly** — this is the native unit
**Pick Elixir.** It’s not close.

Heavy **token streaming** to live UIs, real-time
interaction in the hot path
**Strongly** — LiveView gives it to you nearly
free
**Pick Elixir** (Phoenix earns its keep here
alone)

**Durable multi-step pipelines** that must survive
crashes and run exactly once
**Yes** — Oban replaces a Temporal-shaped problem
**Lean Elixir**, unless you already run that
orchestration well

Mostly **stateless CRUD** over a hosted model API,
short requests
**Weakly** — little to isolate or supervise
**Use what you know.** Elixir is fine but not a
differentiator.

**Model-centric / ML-heavy** — training, fine-tuning,
local inference
**No** — wrong runtime for the hard problem
**Python.** Don’t fight it.

**Tiny team, ships next week, knows Node**
N/A — the constraint is people, not architecture
**Use what you can staff today.**

**Multi-shape** (agents *and* a model
*and* a tool to distribute)
Partially — to the stateful/concurrent parts
**Split it.** BEAM supervises, Python thinks, Go ships
binaries.

The decision rule underneath the table: **pick your runtime by
where your system’s hard problem actually lives.** If the hard
problem is keeping an enormous number of independent, stateful,
failure-prone operations alive and isolated for hours, that’s the BEAM
and it’s not a close call. If the hard problem is the model, that’s
Python. If the hard problem is shipping a tool into a thousand machines,
that’s Go. Most real AI products are more than one of these, and the
mature answer is to not force one language across a seam it doesn’t
belong on.

## The one-paragraph version to send your co-founder

If you’re skimming for the decision: an AI product backend is a swarm
of long-lived, stateful, independently-failing operations — agents,
streams, tool calls, durable jobs — and that shape stresses exactly what
Python and Node concurrency are weakest at. The BEAM was purpose-built
for that shape decades ago, for telephone switches, and Phoenix throws
in near-free streaming UIs on top. That makes Elixir a genuinely strong
default for the *orchestration* layer of an AI product — provided
you keep the model on the other side of a wire, you can staff the team,
and your backend really does have that concurrent-stateful shape rather
than being CRUD in a trench coat. If it does, the cluster posts linked
above each prove one piece of why. If it doesn’t, the honest answer is
to use the stack you’ll move fastest in, and I’ll tell you that to your
face rather than sell you a runtime you don’t need.

---

## Testing AI Agent Outputs in Elixir with ExUnit

URL: https://sublimecoding.com/blog/testing-ai-agent-outputs-exunit
Published: 2026-07-20
Tags: Elixir, AI, agents

>
**TL;DR:** Search “testing LLM outputs Elixir” and every
result is a library README or an announce thread — Tribunal, Ash’s evals
tooling, a forum post asking how anyone does this. Nobody’s written the
practitioner version: how you actually wire an eval framework into an
ExUnit suite for a real agent loop, running in CI, without an API bill
on every push. Here it is. I stub the LLM call boundary
deterministically with `Req.Test`, add one
`:telemetry` event so I can assert the *order* tool
calls happen in — not just what the final string says — and split
deterministic checks from LLM-judge checks so `mix test`
stays free and fast while `mix test --only eval` catches
drift. The one thing that doesn’t change: an agent that survives a
deploy still needs tests that survive a model upgrade, and those are two
different test suites.

## The SERP for this is all library docs

I went looking for “how do I test an AI agent’s behavior in CI” and
every first-page result is one of three things: a hex.pm package README,
a launch thread on X, or a forum post from someone who clearly hasn’t
solved it yet. [Tribunal](https://github.com/georgeguimaraes/tribunal) — George
Guimarães’s LLM evaluation framework for Elixir, same author behind [Arcana](https://github.com/georgeguimaraes/arcana) — has a
genuinely good API. [Ash’s
evals tooling](https://github.com/ash-project/evals) does too. But both sit at the “here’s the assertion
macro” layer. Nobody’s shown the thing in between: a real agent loop,
wired into a real ExUnit suite, testing the parts that actually break —
tool-call sequencing, not just the words in the final answer.

That gap matters because the failure modes of an agent are
structurally different from the failure modes of a function. A pure
function that returns the wrong string is a bug you can
`assert_equal` your way to. An agent that calls
`read_file` twice, never calls `write_file` at
all, and then produces a final answer that *sounds* plausible is
a different class of bug entirely — and a naive test suite that only
checks the last message will pass it every time.

I’ve written before about [the 50-line agent
loop](https://sublimecoding.com/blog/build-ai-agent-loop-50-lines-elixir) itself and about [running that loop on Oban so
it survives a deploy mid-task](https://sublimecoding.com/blog/oban-durable-ai-agent-runtime). This post is the piece that sits
between those two: how you actually test the thing before you trust it
in either environment.

## Two kinds of correctness, and they cost differently

Before any code, get the taxonomy straight, because it’s the
difference between a test suite that runs on every commit for free and
one that burns your API budget on every push.

Anthropic’s own engineering team lays out the same split in their
guide to [evaluating
AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents): code-based graders (fast, cheap, deterministic, but
brittle to valid rewordings), model-based graders (an LLM judging
another LLM’s output — flexible, but non-deterministic and priced per
call), and human graders (the ceiling, and too slow for CI). Their
framing is exactly the line I draw in an ExUnit suite:

- **Deterministic assertions** — did the agent call
`read_file` before `write_file`? Does the response
contain the string “30 days”? Did it stay under the token budget? These
are pure functions over a transcript. No API call, no flakiness, run
them on every commit.

- **LLM-judge assertions** — is this answer
*faithful* to the source document? Did it hallucinate a policy
that doesn’t exist? These require a second model call to grade the first
model’s output, which means real latency and a real bill.

The mistake I see most often is treating both categories the same way
— either skipping the judge-based checks entirely because “they’re
flaky,” or running everything, judge calls included, on every single
push, which is how a team ends up with a five-minute, five-dollar test
suite and starts skipping it. Neither is right. The deterministic layer
should be exhaustive and free. The judge layer should be small, curated,
and gated to when it matters — which is a CI config decision I’ll get to
below, not a testing-philosophy compromise.

## Two seams the 50-line loop needs before it’s testable

The [agent loop
from the earlier post](https://sublimecoding.com/blog/build-ai-agent-loop-50-lines-elixir) is a GenServer: `init` sends itself
a `:step` message, `handle_info(:step, ...)` calls
the LLM, dispatches a tool if one’s requested, and loops. It’s genuinely
the whole pattern in 50 lines — but as written, it’s not testable,
because it does two things a test can’t control: it hits the real
Anthropic API, and it has no seam for observing *what happened*,
only the final `IO.puts`.

Fixing that is two small changes, not a rewrite.

**Seam one: let the HTTP call be stubbed.**
`Req` ships its own test-double module,
`Req.Test`, which registers a named plug you route requests
through instead of the network. That’s the deterministic half of the
eval story — a scripted, repeatable multi-turn transcript with zero API
cost and zero flakiness. The loop’s `call_llm/3` needs an
`opts` argument threaded through to
`Req.post/1`:

`[](#cb1-1)defp call_llm(messages, tools, opts) do
[](#cb1-2) body = %{
[](#cb1-3) model: "claude-sonnet-5",
[](#cb1-4) max_tokens: 4096,
[](#cb1-5) messages: messages
[](#cb1-6) }
[](#cb1-7)
[](#cb1-8) body =
[](#cb1-9) if tools == [],
[](#cb1-10) do: body,
[](#cb1-11) else: Map.put(body, :tools, tools)
[](#cb1-12)
[](#cb1-13) plug_opt =
[](#cb1-14) if opts[:plug],
[](#cb1-15) do: [plug: opts[:plug]],
[](#cb1-16) else: []
[](#cb1-17)
[](#cb1-18) req_opts =
[](#cb1-19) [
[](#cb1-20) url: "https://api.anthropic.com/v1/messages",
[](#cb1-21) json: body,
[](#cb1-22) headers: [
[](#cb1-23) {"x-api-key", api_key()},
[](#cb1-24) {"anthropic-version", "2023-06-01"}
[](#cb1-25) ]
[](#cb1-26) ] ++ plug_opt
[](#cb1-27)
[](#cb1-28) {:ok, %{status: 200, body: resp}} =
[](#cb1-29) Req.post(req_opts)
[](#cb1-30)
[](#cb1-31) parse_response(resp)
[](#cb1-32)end
```

opts` flows down from `run/3` through
`init/1` into `state`, and
`handle_info(:step, state)` passes `state.opts` at
the call site. In production, `opts[:plug]` is
`nil` and `plug_opt` is `[]` — the
request goes to the real API exactly as before. In a test, you pass
`plug: {Req.Test, Agent.LLM}` and every call is intercepted,
in order, by whatever you’ve scripted with
`Req.Test.expect/2` and `Req.Test.expect/3`.

**Seam two: emit an event when a tool fires.** The
existing `dispatch_tool/2` already prints a line for every
call — `IO.puts("🔧 #{name}(...)")` — which tells you it
*knows* the moment a tool executes. The fix is to stop routing
that knowledge through stdout and start routing it through
`:telemetry`, which is the standard way to observe what’s
happening inside an OTP process without coupling the process to
whoever’s watching:

`[](#cb2-1)defp dispatch_tool(name, input) do
[](#cb2-2) :telemetry.execute(
[](#cb2-3) [:agent, :tool_call],
[](#cb2-4) %{},
[](#cb2-5) %{name: name, input: input}
[](#cb2-6) )
[](#cb2-7)
[](#cb2-8) case name do
[](#cb2-9) "read_file" ->
[](#cb2-10) input["path"]
[](#cb2-11) |> File.read()
[](#cb2-12) |> then(fn
[](#cb2-13) {:ok, c} -> c
[](#cb2-14) {:error, r} -> "Error: #{r}"
[](#cb2-15) end)
[](#cb2-16)
[](#cb2-17) "list_files" ->
[](#cb2-18) (input["path"] || ".")
[](#cb2-19) |> File.ls!()
[](#cb2-20) |> Enum.join("\n")
[](#cb2-21)
[](#cb2-22) "write_file" ->
[](#cb2-23) File.write!(
[](#cb2-24) input["path"],
[](#cb2-25) input["content"]
[](#cb2-26) )
[](#cb2-27)
[](#cb2-28) "ok"
[](#cb2-29)
[](#cb2-30) _ ->
[](#cb2-31) "Unknown tool: #{name}"
[](#cb2-32) end
[](#cb2-33)end
```

Neither change touches the loop’s actual behavior. That’s the point —
a codebase you’d trust in production shouldn’t need a special “test
mode” branch in its business logic, just an observable seam at the
boundary.

## Asserting on tool-call sequences, not just the final string

A test that only checks the final string passed while the wrong tool
ran twice and the right one never ran at all.

With both seams in place, a small Agent` process can
collect the ordered list of tool names an attached telemetry handler
observes:

`[](#cb3-1)defmodule Agent.ToolTrace do
[](#cb3-2) def start do
[](#cb3-3) Agent.start_link(
[](#cb3-4) fn -> [] end,
[](#cb3-5) name: __MODULE__
[](#cb3-6) )
[](#cb3-7) end
[](#cb3-8)
[](#cb3-9) def attach do
[](#cb3-10) :telemetry.attach(
[](#cb3-11) "trace-#{inspect(self())}",
[](#cb3-12) [:agent, :tool_call],
[](#cb3-13) fn _event, _meas, meta, _cfg ->
[](#cb3-14) Agent.update(
[](#cb3-15) __MODULE__,
[](#cb3-16) &(&1 ++ [meta.name])
[](#cb3-17) )
[](#cb3-18) end,
[](#cb3-19) nil
[](#cb3-20) )
[](#cb3-21) end
[](#cb3-22)
[](#cb3-23) def calls, do: Agent.get(__MODULE__, & &1)
[](#cb3-24)end
```

Then the test itself scripts a three-turn transcript with
Req.Test.expect/2` — first turn proposes
`list_files`, second proposes `read_file`, third
returns the final answer — and asserts on the *order*, not just
the last line:

`[](#cb4-1)defmodule Agent.LoopSequenceTest do
[](#cb4-2) use ExUnit.Case, async: true
[](#cb4-3)
[](#cb4-4) setup do
[](#cb4-5) Agent.ToolTrace.start()
[](#cb4-6) Agent.ToolTrace.attach()
[](#cb4-7) :ok
[](#cb4-8) end
[](#cb4-9)
[](#cb4-10) test "reads before it answers" do
[](#cb4-11) Req.Test.expect(Agent.LLM, fn conn ->
[](#cb4-12) Req.Test.json(conn, %{
[](#cb4-13) "stop_reason" => "tool_use",
[](#cb4-14) "content" => [
[](#cb4-15) %{
[](#cb4-16) "type" => "tool_use",
[](#cb4-17) "id" => "t1",
[](#cb4-18) "name" => "list_files",
[](#cb4-19) "input" => %{}
[](#cb4-20) }
[](#cb4-21) ]
[](#cb4-22) })
[](#cb4-23) end)
[](#cb4-24)
[](#cb4-25) Req.Test.expect(Agent.LLM, fn conn ->
[](#cb4-26) Req.Test.json(conn, %{
[](#cb4-27) "stop_reason" => "tool_use",
[](#cb4-28) "content" => [
[](#cb4-29) %{
[](#cb4-30) "type" => "tool_use",
[](#cb4-31) "id" => "t2",
[](#cb4-32) "name" => "read_file",
[](#cb4-33) "input" => %{"path" => "mix.exs"}
[](#cb4-34) }
[](#cb4-35) ]
[](#cb4-36) })
[](#cb4-37) end)
[](#cb4-38)
[](#cb4-39) Req.Test.expect(Agent.LLM, fn conn ->
[](#cb4-40) Req.Test.json(conn, %{
[](#cb4-41) "stop_reason" => "end_turn",
[](#cb4-42) "content" => [
[](#cb4-43) %{
[](#cb4-44) "type" => "text",
[](#cb4-45) "text" => "Elixir ~> 1.18"
[](#cb4-46) }
[](#cb4-47) ]
[](#cb4-48) })
[](#cb4-49) end)
[](#cb4-50)
[](#cb4-51) Agent.Loop.run(
[](#cb4-52) "What Elixir version is this?",
[](#cb4-53) [],
[](#cb4-54) plug: {Req.Test, Agent.LLM}
[](#cb4-55) )
[](#cb4-56)
[](#cb4-57) Process.sleep(20)
[](#cb4-58)
[](#cb4-59) assert Agent.ToolTrace.calls() ==
[](#cb4-60) ["list_files", "read_file"]
[](#cb4-61) end
[](#cb4-62)end
```

This is the assertion a string-matching test can’t make. An agent
that calls list_files` three times before giving up and
calling `read_file` still produces a final answer that could
satisfy `assert_contains ans, "1.18"` — and the test would
pass while the agent burned three extra API calls and 90 seconds
figuring out what a one-shot `read_file` should have told it
immediately. Sequence assertions catch the thing that actually costs
money and time in production: the agent doing the right thing the slow,
expensive way.

## Hallucination and faithfulness checks with Tribunal

Sequencing is deterministic; content grounding usually isn’t, which
is where a judge-based library earns its cost. Tribunal ships an ExUnit
case template with two tiers of assertion — deterministic string checks
that run free, and LLM-as-judge checks that call out to a model:

`[](#cb5-1)def deps do
[](#cb5-2) [
[](#cb5-3) {:tribunal, "~> 1.3"},
[](#cb5-4) {:req_llm, "~> 1.2", only: :test},
[](#cb5-5) {:alike, "~> 0.1", only: :test}
[](#cb5-6) ]
[](#cb5-7)end
```

The deterministic tier costs nothing and runs on every commit:

[](#cb6-1)defmodule Agent.AnswerShapeTest do
[](#cb6-2) use ExUnit.Case
[](#cb6-3) use Tribunal.EvalCase
[](#cb6-4)
[](#cb6-5) test "answers cite the doc, don't invent" do
[](#cb6-6) {:ok, ans} =
[](#cb6-7) Agent.Loop.run(
[](#cb6-8) "Summarize our refund policy",
[](#cb6-9) docs_tools()
[](#cb6-10) )
[](#cb6-11)
[](#cb6-12) assert_contains ans, "return"
[](#cb6-13) refute_contains ans, "lifetime warranty"
[](#cb6-14) end
[](#cb6-15)end
```

The judge tier costs a model call, tags itself :eval`,
and is where faithfulness and hallucination checks actually live:

`[](#cb7-1)defmodule Agent.FaithfulnessEvalTest do
[](#cb7-2) use ExUnit.Case
[](#cb7-3) use Tribunal.EvalCase
[](#cb7-4)
[](#cb7-5) @moduletag :eval
[](#cb7-6) @docs File.read!(
[](#cb7-7) "test/fixtures/refund_policy.md"
[](#cb7-8) )
[](#cb7-9)
[](#cb7-10) test "grounded in the doc, not guessed" do
[](#cb7-11) {:ok, ans} =
[](#cb7-12) Agent.Loop.run(
[](#cb7-13) "What's our refund window?",
[](#cb7-14) docs_tools()
[](#cb7-15) )
[](#cb7-16)
[](#cb7-17) assert_faithful ans, context: @docs
[](#cb7-18)
[](#cb7-19) refute_hallucination ans, context: @docs
[](#cb7-20) end
[](#cb7-21)end
```

assert_faithful/2` and
`refute_hallucination/2` route to a judge model via the
optional `req_llm` dependency — that’s the API cost, and it’s
the reason this test is tagged, not left to run alongside the free
suite.

## Running evals in CI without burning tokens on every push

That `:eval` tag is the whole CI story.
`test/test_helper.exs` excludes it by default:

`[](#cb8-1)ExUnit.start()
[](#cb8-2)ExUnit.configure(exclude: [:eval])
```

Every mix test` on every push runs the deterministic
layer — sequencing, string checks, token-budget and timeout guardrails —
for free, in seconds, exactly like any other test suite. The judge-based
layer only runs on demand:

`[](#cb9-1)mix test --only eval
```

Wire that into a scheduled or pre-merge CI job instead of every-push,
and you’ve reproduced the shape Anthropic recommends in the same evals
guide cited above: “automated evals are especially useful pre-launch and
in CI/CD, running on each agent change and model upgrade as the first
line of defense against quality problems” — pre-launch and
per-model-upgrade, not per-commit. Their guidance to start with “20-50
simple tasks drawn from real failures” is the right target for the judge
tier too: it doesn’t need to be exhaustive, it needs to cover the ways
your agent has actually gone wrong before.

The competitive argument for having this at all is worth stating
plainly, because it’s the one that gets this line item approved: per
Anthropic, “teams without evals face weeks of testing while competitors
with evals can quickly determine the model’s strengths, tune their
prompts, and upgrade in days” when a new model version ships. That’s not
a testing-hygiene argument, it’s a time-to-ship argument.

## Evals aren’t a substitute for judgment — even Anthropic’s own postmortem shows it

One honest caveat, because a tutorial that oversells the tool it’s
teaching is worse than no tutorial. Anthropic’s own [postmortem
on a Claude Code quality regression](https://www.anthropic.com/engineering/april-23-postmortem) is candid that a system-prompt
change shipped despite “multiple weeks of internal testing and no
regressions in the set of evaluations we ran” — the eval suite that
existed simply didn’t cover the failure mode that showed up in
production. Their fix wasn’t to abandon evals, it was to broaden them:
they committed to running “a broad suite of per-model evals for every
system prompt change,” plus soak periods and gradual rollouts,
specifically because their existing suite was too narrow. The lesson
transfers directly: an eval suite is only as good as the failure modes
you’ve actually written a test for, and the failure modes you haven’t
seen yet are exactly the ones that ship anyway. Keep adding cases from
real incidents — which is also why [a
proper AI agent postmortem](https://sublimecoding.com/blog/ai-agent-postmortem) should always end with “add this
transcript to the eval fixtures,” not just “add a monitor.”

## The reliability tie-in: surviving a deploy isn’t the same as surviving a model upgrade

This is the connection that made me want to write this post instead
of just linking to Tribunal’s README. [Running an agent loop on
Oban](https://sublimecoding.com/blog/oban-durable-ai-agent-runtime) buys you durability against infrastructure failure — the
process can die mid-run and the conversation state in Postgres means it
picks back up. That’s a real reliability win, and it’s orthogonal to a
completely different kind of failure: the agent’s *behavior*
changing out from under you because the model underneath it changed.

A model upgrade isn’t a version bump you can changelog-skim past. The
new model can reason differently, follow the same prompt more or less
literally, chain tool calls in a different order, or refuse an input the
old model accepted without complaint. None of that shows up as a deploy
failure — Oban will happily persist and retry a conversation that’s now
subtly wrong. The only thing that catches it is a suite that asserts on
behavior, not uptime: the sequencing tests from the section above re-run
against the new model and either pass or tell you exactly which step
changed, and the faithfulness/hallucination tests re-run and tell you
whether the new model’s answers are still grounded in the same
documents. An agent that survives a deploy and an agent that survives a
model upgrade are protected by two different test suites, and most teams
that have built the first one haven’t built the second.

## Where this fits in an Elixir AI stack

None of this requires an AI-specific test framework at its foundation
— it’s ExUnit, :telemetry`, and `Req.Test`, three
things that are either in the standard library or already in your
`mix.exs` if you’re calling an LLM API from Elixir at all.
Tribunal adds the judge-based assertions on top, and it’s worth adopting
for that layer specifically rather than hand-rolling your own
LLM-as-judge harness. This is also, I think, a decent argument for [why the BEAM is a good
home for an AI startup’s backend](https://sublimecoding.com/blog/why-elixir-for-ai-startup-backend) in the first place:
`:telemetry` being a first-class, ecosystem-standard
observability primitive rather than a bolted-on logging library is
exactly the kind of thing that makes testing an agent’s
*behavior* — not just its uptime — a normal part of the existing
toolchain instead of a new category of infrastructure you have to buy or
build.

If you’re standing up an agent that’s going to run unattended in
production — durable runtime, real tool access, a model someone else
controls the version of — the test suite is not optional scaffolding you
add later. It’s the thing that tells you, in CI, before a customer does,
that the last model upgrade quietly changed what your agent does with a
customer’s data. If you want a second set of eyes on where that suite
should sit relative to the rest of your AI infrastructure, [that’s a conversation worth having](https://sublimecoding.com/consulting).

---

## TDD With Claude Code in Elixir: What Holds Up

URL: https://sublimecoding.com/blog/tdd-claude-code-elixir
Published: 2026-07-18
Tags: Elixir, AI tools, engineering

>
**TL;DR:** Skip the red-green-refactor tutorial — that
content is saturated and generic. The Elixir-specific question is
narrower and more useful: which parts of TDD-with-an-agent actually hold
up in an ExUnit/Phoenix codebase, and which parts quietly break. The
answer: agents follow a failing test far more reliably than they follow
a paragraph of prose instructions, which makes test-enforced discipline
— `mix precommit`, tagged `async: true`
conventions, idempotent `Ecto.Multi`/Oban contracts — the
real moat, not the prompt. Where it falls apart: LiveView test
flakiness, fixture conventions Claude can’t infer from a schema alone,
and the seductive trap of writing a test that passes for the wrong
reason. This is what actually happened writing tests alongside an agent
in a live Phoenix repo, not a hypothetical.

## Skip the preamble — here’s what’s different in Elixir

Every “TDD with AI” post on the internet is language-agnostic, which
means it’s written for Python or TypeScript with the specifics filed
off. Elixir has real specifics: OTP’s process model means a badly-scoped
test can leak state across processes in ways a synchronous language
never would; Ecto’s sandbox mode means database tests can run
concurrently *by default*, which is a gift and a trap; and
Phoenix’s LiveView tests exercise a genuinely different execution model
(a supervised process exchanging events) than a typical HTTP
request/response test. None of that shows up in a generic TDD guide, and
all of it changes what “write the test first” means in practice.

The frame I keep coming back to, working in this repo day to day: an
agent obeys a test far more reliably than it obeys a sentence in
`CLAUDE.md`. I’ve written about the limits of prose
instructions before — [what
actually earns a line in CLAUDE.md after 50 commits](https://sublimecoding.com/blog/claude-md-after-50-commits) is a short list,
because most of what you’d want to tell an agent turns out to compress
much better into an executable check than a rule. TDD isn’t a
nice-to-have discipline layered on top of agent-driven development. It’s
the enforcement mechanism that makes agent-driven development safe to
run unattended. Everything below is a specific instance of that claim,
tested against this site’s own Elixir/Phoenix codebase.

## ExUnit async vs DataCase/ConnCase: the default Claude gets wrong

The single most common ExUnit mistake I see an agent make,
unprompted, is marking a new test `async: true` without
checking whether the module actually supports it.
`ExUnit.Case, async: true` is fine for a pure module with no
shared state. `SublimeCodingWeb.ConnCase` and
`SublimeCoding.DataCase` — the two base cases that touch the
database or the Ecto sandbox — are a different story, and whether
they’re safe to run concurrently depends on the checkout mode the case
template sets up, not on the test file itself.

Grep this repo’s own test suite and the split is visible
immediately:

`test/sublime_coding_web/plugs/security_headers_test.exs:
 use ExUnit.Case, async: true

test/sublime_coding_web/seo_health_test.exs:
 use SublimeCodingWeb.ConnCase
 (no async: true)

test/sublime_coding_web/live/vciso_cost_live_test.exs:
 use SublimeCodingWeb.ConnCase, async: true
```

Three tests, three different answers to “is this safe concurrently,”
and the difference isn’t visible from the test body — it’s a property of
what the test touches (plug-level logic with no shared connection
vs. content lookups against a compiled resource vs. a LiveView test
against the sandbox). An agent asked to “add a test for this” will
pattern-match on the nearest example file and copy its
async` setting whether or not it’s the right call for the
new test’s actual dependencies. The fix isn’t a longer explanation in
the prompt — it’s letting `mix test` be the check. If a test
that shouldn’t be concurrent gets marked async, you get an intermittent,
hard-to-reproduce failure under `--seed` variation, which is
a worse failure mode than an outright wrong test, because it looks like
flakiness instead of a mistake. Catch it by running the suite more than
once locally before trusting an agent’s addition, not by writing a
longer rule about it.

## `mix precommit` is the actual instruction Claude follows

This project’s `mix.exs` defines a `precommit`
alias:

`[](#cb2-1)precommit: [
[](#cb2-2) "compile --warnings-as-errors",
[](#cb2-3) "deps.unlock --check-unused",
[](#cb2-4) "format",
[](#cb2-5) "test"
[](#cb2-6)]
```

That’s the single most load-bearing line in the project for
agent-driven work, and it has nothing to do with prompting.
--warnings-as-errors` turns an unused variable, an ambiguous
pattern match, or a deprecated function call into a build failure
instead of a scrollback warning an agent (or a tired human) skims past.
Combined with `test`, an agent can’t claim a change is “done”
without the compiler and the full suite both agreeing — and per this
repo’s own working agreement, “verify before claiming done” means
literally running `mix compile && mix test`, not
asserting it should work.

The practical shift this produces: instead of writing “make sure you
don’t break existing tests” into a prompt (which an agent will nod along
with and sometimes still violate), you make `mix precommit`
the actual gate — a CI step, a pre-commit hook, or just the command you
run before you’ll look at a diff. Prose is a suggestion; a red exit code
is not. This is the same principle this site’s own build enforces on
itself in an unrelated domain: the tag taxonomy and SERP
`<title>` length aren’t governed by an editorial style
guide anyone has to remember to reread — they’re pinned by
`test/sublime_coding_web/seo_health_test.exs`, which asserts
merged-out tag slugs are gone from the active taxonomy and that every
post’s rendered `<title>` fits in 70 characters. An
agent drafting a new post doesn’t need to have internalized the
taxonomy; it needs the test to fail loudly if it picks a dead tag. Tests
are how you hand an agent a constraint it can’t talk itself out of.

## Ecto.Multi and Oban jobs are TDD-shaped problems whether you plan it that way or not

This is the part that surprised me least, in retrospect, but that
most TDD content entirely misses: `Ecto.Multi` and Oban jobs
are *already* structured as testable contracts, because both are
built around the same idea — a named, composable step that either fully
applies or doesn’t, with an explicit success/failure shape you can
assert against without touching the database twice by accident.

An `Ecto.Multi` pipeline is naturally test-first-friendly
because each named step is independently assertable:

`[](#cb3-1)Multi.new()
[](#cb3-2)|> Multi.insert(:account, changeset)
[](#cb3-3)|> Multi.run(:ledger, fn repo, %{account: a} ->
[](#cb3-4) ledger_entry(repo, a)
[](#cb3-5)end)
[](#cb3-6)|> Repo.transaction()
```

You can write the test for the :ledger` step’s failure
branch before the implementation exists — assert that a failed ledger
insert rolls back the account insert too — and that test is exactly as
valuable written first as written after, because the behavior it’s
pinning (atomicity across two writes) is invisible in a code read and
only provable by exercising the rollback path.

Oban jobs raise the same shape one level up, and it’s the exact
idempotency argument I’ve made about [running Oban as a durable AI
agent runtime](https://sublimecoding.com/blog/oban-durable-ai-agent-runtime): a job that retries on a transient failure but isn’t
idempotent will double-charge, double-send, or double-write on its
second attempt. The test that matters isn’t “does the job succeed” —
it’s “does running `perform/1` twice with the same args
produce the same end state as running it once.” That’s a test you can
and should write before the job body, because it’s the contract the
retry mechanism depends on, and an agent generating job code from a
prompt alone has no way to know retries are even in play unless the test
says so explicitly. I’ve seen an agent write a perfectly
reasonable-looking Oban worker that increments a counter on every
`perform/1` call with zero acknowledgment that Oban
*will* call it more than once for the same job under a transient
failure. A failing idempotency test catches that in one line; a prose
reminder to “handle retries” gets forgotten by the third worker of the
session.

## Where TDD-with-agents actually breaks down

I don’t want to oversell this. Three places where the practice
genuinely struggles, honestly:

**LiveView test flakiness.**
`Phoenix.LiveViewTest` exercises real process message-passing
— `render_click`, `render_change`, and friends
dispatch through the LiveView process and wait on its reply. That’s
mostly reliable, but a test that asserts on DOM state immediately after
an action that triggers an async assign (a `Task`, a PubSub
broadcast, a debounced form) can pass locally and flake in CI depending
on scheduler timing. An agent writing a new LiveView test from scratch
has no signal that a given interaction is async under the hood unless
the LiveView module’s own code makes that obvious, and it will happily
write an assertion that’s correct 95% of the time and a source of
exactly the kind of intermittent CI failure that erodes trust in the
whole suite. The fix is procedural, not architectural:
`mix test --repeat-until-failure` locally, or just running
new LiveView tests several times before trusting them into a commit —
the same “don’t take the first green on faith” discipline you’d want
from a human, and one an agent won’t apply to its own work without being
told, every time, because it isn’t a rule that generalizes from a test
file it can imitate.

**Fixture and ExMachina conventions Claude can’t
infer.** Given a schema, an agent will write a plausible-looking
fixture — but “plausible” and “matches this project’s actual factory
conventions” (required associations, a specific `sequence/2`
pattern for uniqueness, which fields get sensible defaults vs. which
ones must be supplied by the caller) are different things, and the
difference only lives in the existing factory file, not in the schema.
If there’s an existing `ExMachina` factory module, point the
agent at it explicitly before asking for a new test — “extend the
pattern in `test/support/factory.ex`” produces something
reusable; “write a fixture for this schema” produces a one-off that
duplicates logic the next test writer (human or agent) won’t find.

**Tests that pass for the wrong reason.** This is the
general failure mode underneath the two specific ones above, and it’s
worth naming on its own: an agent under pressure to make a red test
green will sometimes narrow the assertion instead of fixing the
implementation — asserting `assert result` instead of
asserting the actual expected value, or stubbing out the exact input a
broken function happens to handle. I’ve written more broadly about this
pattern and [the other
bugs coding agents introduce reliably enough to check for by habit](https://sublimecoding.com/blog/things-ai-coding-agents-get-wrong).
The countermeasure in an Elixir codebase specifically: prefer
pattern-matching assertions
(`assert {:ok, %Account{balance: 100}} = result`) over
boolean ones, because a pattern match fails loudly and specifically if
the shape is wrong, where a bare `assert result` will happily
pass against almost anything truthy.

## The actual workflow, stated plainly

Concretely, in this repo: I ask for the failing test first,
specifying the exact base case (`DataCase` vs
`ConnCase` vs plain `ExUnit.Case`) and whether it
needs `async: true`, rather than leaving that inference to
pattern-matching against the nearest file. For anything touching
`Ecto.Multi` or an Oban worker, the idempotency test is
written before the implementation, not after, because it’s the one an
agent won’t think to add unprompted. And nothing gets called done
without `mix precommit` passing clean — not “the new test
passes,” the whole suite, with warnings as errors. None of that is
exotic. It’s the same discipline any senior Elixir engineer would apply
to a junior’s PR; the only change is that the “junior” writes code at
agent speed, which means the enforcement has to run automatically or it
silently stops happening by the fourth or fifth file of a session.

The broader point, and it applies past Elixir: this is the same
reason [technical due
diligence on an AI-native codebase](https://sublimecoding.com/blog/surviving-technical-due-diligence-ai-founder) increasingly looks for test
coverage on the paths that actually carry risk, not a percentage number
— a codebase where the enforcement lives in the test suite survives an
agent-heavy contributor list; one where it lives in a style guide
doesn’t. If you’re deciding whether Elixir is even the right backend for
an AI-heavy build in the first place, I’ve made [the fuller case for
Elixir on an AI startup backend](https://sublimecoding.com/blog/why-elixir-for-ai-startup-backend) separately — this post assumes
you’re already there and wants to get the day-to-day discipline
right.

If you’re standing up (or auditing) an Elixir codebase that’s going
to take a lot of agent-authored commits, this is exactly the kind of
workflow review I do as a fractional CTO engagement — [let’s talk about what that looks like](https://sublimecoding.com/consulting).

---

## 10 Ecto Query Patterns for N+1 and Beyond

URL: https://sublimecoding.com/blog/ecto-query-patterns-n-plus-one
Published: 2026-07-28
Tags: Elixir, engineering, Phoenix

>
**TL;DR:** Most Ecto N+1 posts are the same tutorial
wearing a different hat — here’s `preload`, here’s why loops
are bad, the end. What’s missing is the decision layer: *which*
pattern for *which* shape of data, and why. The spine is one
rule: separate queries (preload) for `has_many`, a join for
`belongs_to` — because one avoids row-explosion and the other
avoids an extra round trip you don’t need. The other nine patterns are
what you reach for once that rule stops being enough: filtered preloads,
nested trees, composable query modules, Dataloader when you don’t
control the query shape, subqueries for “latest N per group,”
denormalizing instead of joining, and raw SQL when Ecto’s DSL is
fighting you harder than the problem is worth.

## The gap every N+1 post skips

Search “Ecto N+1” and you’ll get a dozen competent walkthroughs of
the same three moves: don’t call `Repo.preload` inside a
loop, use `preload:` in your query, watch your LiveView
`handle_event` callbacks for the same mistake. [Scout
Monitoring published a solid, current version of that post in April
2026](https://www.scoutapm.com/blog/elixir-ecto-n1-queries-guide), and it’s worth reading if you need the mechanics — preload
variants, LiveView-specific N+1s, Oban worker N+1s, all covered well.
I’m not rewriting it.

What none of these posts give you is the thing you actually need six
months into a codebase: a rule for *which* pattern applies to
*which* shape of query, made before you’re staring at a slow
endpoint in production. Junior engineers on my teams don’t struggle with
“what is preload” — they struggle with “why did the senior engineer just
tell me to *undo* the preload I added and write a join instead.”
That’s a judgment call, and nobody writes the judgment call down. So
here it is, as ten decisions instead of ten syntax examples.

## The Iron Law: separate queries for has_many, join for belongs_to

Start here, because everything else is an exception to this, and the
exceptions make no sense until the rule is explicit.

Separate queries for has_many, a join for belongs_to — not a style
preference, a row-count argument.

A `belongs_to` association is one row to one row. Joining
a post to its author doesn’t multiply anything — you get exactly as many
result rows as you had posts, just with the author’s columns tacked on.
There’s no cost to paying for that in the same round trip, and doing it
in two queries is pure waste: an extra network hop for data you were
always going to need in a 1:1 shape.

`[](#cb1-1)from p in Post,
[](#cb1-2) join: u in assoc(p, :author),
[](#cb1-3) preload: [author: u]
```

A has_many` is the opposite shape. Join a post to its ten
comments and Postgres gives you back ten *rows* — the post’s
columns repeated ten times, once per comment. Do that across a page of
fifty posts with variable comment counts and you’re now shipping a
multiplied, duplicated result set over the wire just to reassemble it
back into the tree Ecto was going to build for you anyway. That’s why
plain `preload:` (no join) exists: it issues a second query —
`WHERE post_id IN (...)` — and stitches the results together
in memory. One extra round trip, zero row multiplication.

`[](#cb2-1)from p in Post, preload: [:comments]
```

[Ecto’s own docs are
explicit about the tradeoff](https://ecto.hexdocs.pm/Ecto.Query.html): a plain preload:` fetches
posts, then issues a separate query for comments and associates them in
a second pass; a join-based preload does it in one query but duplicates
every joined row. The rule is simply picking the cheaper cost for the
shape you have — an extra round trip for `has_many`, versus
wasted bytes for `belongs_to`. Most of the N+1s I’ve had to
unwind in Phoenix codebases weren’t from missing preloads at all — they
were from a `has_many` joined the way you’d correctly join a
`belongs_to`, quietly duplicating rows nobody looked at
closely enough to notice.

## Pattern 1–2: the split isn’t a style preference, it’s physics

Patterns one and two *are* the Iron Law above, stated as
decisions you make at write time, not fixes you make at debug time:

- **Association is `belongs_to` (or
`has_one`)?** Join and preload off the join binding.
Same query, no duplication, no extra round trip.

- **Association is `has_many` (or
`many_to_many`)?** Plain `preload:`, no
join. Accept the second query; refuse the duplicated rows.

If you only ever apply these two, you’ve already fixed the
overwhelming majority of the N+1s that show up in a code review.

## Pattern 3: filtered has_many preloads need the join back

The Iron Law has an escape hatch, and this is it. Sometimes you need
a `has_many` preload but only the *matching* rows —
published comments, not all of them. Plain `preload:` can’t
filter; it fetches the full association. For that you go back to the
join, but you preload off the join binding instead of selecting its
columns directly:

`[](#cb3-1)from p in Post,
[](#cb3-2) join: c in assoc(p, :comments),
[](#cb3-3) where: c.published_at > p.updated_at,
[](#cb3-4) preload: [comments: c]
```

Ecto still does the “one query, then a stitching pass” trick here —
it fetches posts and comments together but reassembles them into the
tree structure, so you get the ergonomics of preload:` with
the filtering power of a join. This is the pattern that trips people up
going the other direction: they see a join and assume it’s always the
row-explosion problem. It’s the *unfiltered, unpreloaded* join
that’s the problem — a filtered join feeding `preload:` is
fine, because you’re back to one row set that Ecto explicitly reshapes
for you.

## Pattern 4: compose the whole tree in one preload, not N calls

If you need posts, their comments, and each comment’s author, don’t
preload posts, then loop and preload comments, then loop again for
authors. Ecto’s `preload:` takes a nested keyword structure
and resolves the entire tree in one composed call, dispatching one query
per *level* of the tree — not per row:

`[](#cb4-1)from p in Post,
[](#cb4-2) preload: [comments: :author]
```

That’s still N+1-shaped in the strict sense — N levels means N
queries — but N is the depth of your association graph, which is small
and fixed, not the row count, which isn’t. That distinction is the whole
point: N+1 isn’t inherently the enemy, N+1 *per row* is.

## Pattern 5–6: composable query modules beat query spaghetti

Once a schema has more than two or three call sites, you’ll be
tempted to write the same
where`/`join`/`preload` combination
slightly differently in each one. [Curiosum’s
write-up on resource-scoped query modules](https://curiosum.com/blog/composable-elixir-ecto-queries-modules) is the cleanest version of
the fix I’ve seen, and it’s worth adopting wholesale rather than
reinventing a worse copy: one module per resource, small composable
functions, piped together at the call site.

`[](#cb5-1)defmodule Blog.PostQuery do
[](#cb5-2) import Ecto.Query
[](#cb5-3)
[](#cb5-4) def base, do: from(p in Post, as: :post)
[](#cb5-5)
[](#cb5-6) def published(query \\ base()) do
[](#cb5-7) where(query, [post: p], p.published)
[](#cb5-8) end
[](#cb5-9)
[](#cb5-10) def with_author(query \\ base()) do
[](#cb5-11) join(query, :inner, [post: p],
[](#cb5-12) u in assoc(p, :author), as: :author
[](#cb5-13) )
[](#cb5-14) end
[](#cb5-15)end
```

That’s pattern five: **extract the filters, not the whole
query, into named functions on a module scoped to the schema.**
Each function takes and returns a queryable, so they compose with the
pipe operator instead of copy-pasting where` clauses across
contexts.

Pattern six is where the preload decision above actually lives once
you’ve adopted this: **preload at the tail of the pipeline, in the
context boundary function, never inside the composable query functions
themselves.**

`[](#cb6-1)Blog.PostQuery.published()
[](#cb6-2)|> Blog.PostQuery.with_author()
[](#cb6-3)|> preload([post: p, author: u], author: u)
[](#cb6-4)|> Repo.all()
```

Keeping preload out of the composable functions means
published/1` and `with_author/1` stay reusable in
contexts that don’t want the association loaded at all — a count query,
an existence check, an admin export with different fields. Bake the
preload into the module function and every caller pays for a join they
might not need.

## Pattern 7: Dataloader, when you don’t own the query shape

Everything above assumes you control the query at the call site. You
don’t, in two common cases: an Absinthe GraphQL resolver, where the
client picks which fields (and therefore which associations) get
requested per query; and a LiveView tree of independent function
components, each rendering off the same list without knowing what its
siblings already loaded.

[Dataloader.Ecto](https://dataloader.hexdocs.pm/Dataloader.Ecto.html)
solves this by batching *within a single request*, deferring
actual execution until every resolver or component has had a chance to
register what it needs:

`[](#cb7-1)source = Dataloader.Ecto.new(MyApp.Repo)
[](#cb7-2)
[](#cb7-3)loader =
[](#cb7-4) Dataloader.new()
[](#cb7-5) |> Dataloader.add_source(:db, source)
[](#cb7-6) |> Dataloader.load_many(
[](#cb7-7) :db, :comments, posts
[](#cb7-8) )
[](#cb7-9) |> Dataloader.run()
[](#cb7-10)
[](#cb7-11)Dataloader.get_many(loader, :db, :comments, posts)
```

The decision rule: reach for Dataloader when the *caller*
doesn’t control the query, not as a default replacement for
preload:`. If you own the function signature, plain
composable Ecto is simpler to read, test, and debug. Dataloader earns
its complexity specifically at the boundary where multiple independent
callers need to share one batched fetch without knowing about each
other.

## Pattern 8: subqueries and lateral joins for “latest N per group”

The pattern that plain preload genuinely cannot express: “the three
most recent comments per post,” not all of them. This is where a
correlated subquery with `parent_as` and a lateral join is
the honest tool, not a workaround:

`[](#cb8-1)recent_comments =
[](#cb8-2) from c in Comment,
[](#cb8-3) where: c.post_id == parent_as(:post).id,
[](#cb8-4) order_by: [desc: c.inserted_at],
[](#cb8-5) limit: 3
[](#cb8-6)
[](#cb8-7)from p in Post, as: :post,
[](#cb8-8) inner_lateral_join: c in subquery(recent_comments),
[](#cb8-9) preload: [comments: c]
```

This is genuinely a different problem from the has_many-vs-belongs_to
split — it’s not about avoiding row explosion, it’s about pushing a
per-row limit down into the database instead of over-fetching everything
and truncating it in Elixir. If you find yourself writing
Enum.take/2` on a preloaded association after the fact, this
pattern is almost always the fix.

## Pattern 9: denormalize before you optimize the join

Not every N+1-shaped problem should be solved with a smarter query.
If the only thing a view needs from a `has_many` is a count —
comment count, order count, active-session count — you don’t need to
load the association at all, filtered or otherwise. A counter cache
column, updated by the same transaction that inserts the child row,
turns an association load into a column read:

`[](#cb9-1)from p in Post, select: p.comments_count
```

This is the pattern people skip because it feels like giving up on
“doing it properly” in Ecto. It’s the opposite — it’s recognizing that
the fastest query is the one you don’t run. If the number is read far
more often than the underlying rows are actually displayed, a maintained
aggregate column beats every join pattern above it on both latency and
plan complexity.

## Pattern 10: raw SQL is not a defeat

Sometimes the query you need — a window function, a recursive CTE, a
LATERAL` with logic Ecto’s DSL doesn’t model cleanly — is
more honestly expressed as SQL than as a wall of `fragment/1`
calls dressed up to look like Elixir. I’ve written the specifics of when
and how elsewhere on [executing raw SQL with
Ecto](https://sublimecoding.com/blog/executing-raw-sql-with-ecto); the short version here is the decision rule, not the syntax:
if you’re spending more time fighting the query DSL into producing the
SQL you already know you want than you’d spend just writing that SQL,
stop fighting it. `Ecto.Adapters.SQL.query/4` behind a named
function in your context module is not a lesser pattern than the nine
above it — it’s the one that admits Ecto’s query DSL, like any DSL, has
an edge, and past that edge the honest move is the tool built for the
job.

## The two you’ll hit but shouldn’t fix here

Two contexts turn ordinary N+1 mistakes into ones that are easy to
miss in code review, because the loop isn’t visible in the query itself
— it’s in the surrounding architecture. I’ve written each up on its own,
since the fix is specific to the context rather than to Ecto:

- **LiveView.** A `handle_event` or a function
component that reloads an association per row, per event, instead of
batching before assign. The patterns for avoiding this live in [12 Phoenix LiveView
Patterns for Production](https://sublimecoding.com/blog/phoenix-liveview-patterns-production).

- **Oban.** A worker that preloads inside a per-item loop
across a batch job, instead of preloading the batch once before
iterating. Covered in [Oban
as a Durable AI Agent Runtime in Elixir](https://sublimecoding.com/blog/oban-durable-ai-agent-runtime).

Both are the same Iron Law from the top of this post, applied
somewhere the query isn’t the first thing you read.

## Pick the rule, not the recipe

If you take one thing from this: stop treating N+1 fixes as a single
recipe you paste in wherever a slow query shows up.
`has_many` gets a separate query, `belongs_to`
gets a join — that’s the rule that should be automatic. Everything past
it — filtered joins, nested preloads, composable query modules,
Dataloader, lateral subqueries, denormalized counters, raw SQL — is a
decision about which cost you’re willing to pay for a specific shape of
data, made deliberately, not discovered in a slow-query log three weeks
after ship. Ecto rewards that kind of upfront judgment more than almost
any other part of a Phoenix codebase; it’s one of the reasons the
language holds up well past prototype scale, a case I’ve made in full in
[Why We’d Pick Elixir
for an AI Startup Backend](https://sublimecoding.com/blog/why-elixir-for-ai-startup-backend). If you’re assembling the rest of your
toolchain around it, [15 Elixir Libraries I
Reach For in 2026](https://sublimecoding.com/blog/elixir-libraries-i-reach-for-2026) covers where Dataloader and its neighbors fit into
a production stack.

If you’re staffing or auditing an Elixir/Phoenix team and query
discipline like this isn’t yet a shared, written-down convention, that’s
exactly the kind of gap a fractional engineering lead closes early
rather than after the slow-query alerts start. [Happy to talk through what that looks like](https://sublimecoding.com/consulting).

---

## Instrumenting LLM Calls in Phoenix with Telemetry

URL: https://sublimecoding.com/blog/phoenix-llm-telemetry
Published: 2026-07-30
Tags: Elixir, AI, engineering

>
**TL;DR:** ReqLLM — the [Req](https://hexdocs.pm/req)-based LLM client for Elixir —
already emits `:telemetry` events with token counts,
calculated cost, and request duration on every call, no wrapper code
required. The gap most Phoenix teams have isn’t visibility, it’s five
lines of `Telemetry.Metrics` connecting an event that already
exists to a dashboard that already ships with Phoenix. This post is the
practitioner path: what ReqLLM emits and where I verified it, how to
wire it into LiveDashboard for local dev, how to ship the same numbers
to Prometheus with a PromEx plugin for production, the minimal
hand-rolled version if you’re on a raw HTTP client instead, and the two
numbers actually worth paging someone over.

## The “what if I don’t want to buy Helicone” question

I wrote [the
case for treating LLM telemetry as a first-class layer, not an
afterthought](https://sublimecoding.com/blog/your-ai-product-needs-telemetry-before-better-model) — the argument that most “the model is bad” complaints
are actually “we can’t see what the model is doing” problems, and that a
telemetry layer usually fixes more than a model upgrade does. That post
is deliberately stack-agnostic and points at hosted options like
Helicone and Langfuse, because most teams asking “how do I see what my
LLM calls are doing” are on Python and a hosted proxy is the fastest
path.

If you’re running Phoenix, you already have two other options that a
Python shop doesn’t: a BEAM-native dashboard that ships in every
`phx.new` app, and a Prometheus/Grafana story that’s one
dependency away. This post is the “okay, but I don’t want to route every
LLM call through a third-party proxy” answer for that stack. It assumes
you’ve already read (or don’t need) the argument for *why*
telemetry matters, and just want the wiring.

One more adjacent post worth knowing about first: [testing AI agent outputs
with ExUnit](https://sublimecoding.com/blog/testing-ai-agent-outputs-exunit) covers
`:telemetry.attach`/`:telemetry.execute` as
primitives, using a test-time event that asserts on tool-call order. I
won’t re-explain what those functions do here — if you haven’t used
`:telemetry` before, read that post’s first section, then
come back. This is the production sibling of that same event system:
instead of asserting on an event in a test, you’re aggregating it into
cost-per-day and p95 latency. And once the events are flowing, [turning them into a per-customer invoice line](https://sublimecoding.com/blog/metering-llm-usage-phoenix) is a different problem with a much lower tolerance for undercounting.

## What ReqLLM already gives you for free

If your Phoenix app talks to an LLM through [ReqLLM](https://github.com/agentjido/req_llm) — the
`Req`/`Finch`-based client from the Jido
ecosystem, covering Anthropic, OpenAI, Google, Bedrock, and a dozen
other providers behind one interface — you don’t need to write a
`:telemetry.span` around your LLM calls. ReqLLM already does
it. I verified the exact event shape against the [library’s telemetry
guide](https://req-llm.hexdocs.pm/telemetry.html) and the `ReqLLM.Telemetry` source rather than trust
the README summary, because getting an event name or a measurement key
wrong here means a wired-up dashboard that silently reads zeros.

ReqLLM emits two families of events:

**The request span — `[:req_llm, :request, :start]`
/ `:stop` / `:exception`.** This is what
the docs now recommend for new integrations, because the
`:stop` event carries both duration and the full request
context in one place. Measurements on `:stop` are
`%{duration: native_time, system_time: ...}`.
`duration` is in native monotonic units, the same convention
Phoenix’s own endpoint/request telemetry uses, so you convert it with
`unit: {:native, :millisecond}` wherever you consume it
rather than doing the math by hand. Metadata on `:stop`
includes `request_id`, `operation`,
`mode`, `provider`, `model` (an
`LLMDB.Model` struct, not a string),
`http_status`, `finish_reason`, and the field that
matters for cost: `usage`, a map with token counts and cost
fields.

**The compat event —
`[:req_llm, :token_usage]`.** A narrower event fired
specifically for cost/token aggregation, and the one most tutorials will
show you first because it’s simpler. Its measurements map nests the raw
usage numbers under a `tokens` key
(`%{tokens: %{input_tokens: _, output_tokens: _, total_tokens: _}, cost: _, total_cost: _, ...}`)
rather than flattening them — worth an `IO.inspect` in
`iex` against your installed version before you write metric
definitions against it, because compat-layer field shapes are exactly
the kind of thing that shifts between minor releases without a changelog
headline.

Either event gets you cost and tokens without touching your LLM call
sites. That’s the actual finding here: the boilerplate you’d expect to
write — wrap every provider call in `:telemetry.span`, thread
through model name and token counts by hand — doesn’t need writing if
ReqLLM is already your client.

## Wiring it into LiveDashboard in dev

Phoenix’s `Telemetry.Metrics` module is the layer both
LiveDashboard and PromEx consume — you define metrics once, against
event names, and both reporters subscribe to the same list. Every
`phx.new --live` app already has a
`YourAppWeb.Telemetry` module with a `metrics/0`
function full of Phoenix and Ecto metrics. Add ReqLLM’s events to that
same list:

`[](#cb1-1)# lib/my_app_web/telemetry.ex
[](#cb1-2)defp req_llm_metrics do
[](#cb1-3) [
[](#cb1-4) distribution(
[](#cb1-5) "req_llm.request.duration",
[](#cb1-6) event_name: [:req_llm, :request, :stop],
[](#cb1-7) measurement: :duration,
[](#cb1-8) unit: {:native, :millisecond},
[](#cb1-9) tags: [:provider, :model],
[](#cb1-10) tag_values: fn meta ->
[](#cb1-11) %{
[](#cb1-12) provider: meta[:provider],
[](#cb1-13) model: meta[:model] && meta[:model].id
[](#cb1-14) }
[](#cb1-15) end,
[](#cb1-16) description: "LLM request latency"
[](#cb1-17) ),
[](#cb1-18) sum(
[](#cb1-19) "req_llm.tokens.total",
[](#cb1-20) event_name: [:req_llm, :token_usage],
[](#cb1-21) measurement: fn m ->
[](#cb1-22) get_in(m, [:tokens, :total_tokens]) || 0
[](#cb1-23) end,
[](#cb1-24) tags: [:provider],
[](#cb1-25) tag_values: &%{provider: &1[:provider]},
[](#cb1-26) description: "Tokens consumed, cumulative"
[](#cb1-27) ),
[](#cb1-28) sum(
[](#cb1-29) "req_llm.cost.total",
[](#cb1-30) event_name: [:req_llm, :token_usage],
[](#cb1-31) measurement: fn m -> m[:total_cost] || 0 end,
[](#cb1-32) tags: [:provider, :model],
[](#cb1-33) tag_values: fn meta ->
[](#cb1-34) %{
[](#cb1-35) provider: meta[:provider],
[](#cb1-36) model: meta[:model] && meta[:model].id
[](#cb1-37) }
[](#cb1-38) end,
[](#cb1-39) description: "Calculated USD cost, cumulative"
[](#cb1-40) ),
[](#cb1-41) counter(
[](#cb1-42) "req_llm.request.count",
[](#cb1-43) event_name: [:req_llm, :request, :stop],
[](#cb1-44) tags: [:provider, :finish_reason]
[](#cb1-45) )
[](#cb1-46) ]
[](#cb1-47)end
```

Then splice req_llm_metrics()` into the existing
`metrics/0` list. That’s the whole integration — no attach
calls, no GenServer to hold running totals. [`Telemetry.Metrics`](https://hexdocs.pm/telemetry_metrics)
handles the subscription; LiveDashboard’s Metrics tab picks up anything
in that list automatically the moment you
`PORT=4001 mix phx.server` and make a call.
`distribution` gives you a live latency histogram per
provider/model, `sum` gives you a running token and cost
counter, `counter` gives you call volume broken out by
`finish_reason` — which is the fastest way to notice a
provider silently truncating responses before a user complains.

The `tag_values` function matters more than it looks:
`model` in ReqLLM’s metadata is an `LLMDB.Model`
struct, and neither LiveDashboard nor Prometheus wants a struct as a
label value, so pull `.id` out before it hits the tag
list.

## Shipping the same numbers to Prometheus with PromEx

LiveDashboard is a great local and staging tool, but it only shows
you the current node’s in-memory state — it doesn’t survive a restart or
aggregate across a fleet. For anything you’d actually alert on, ship the
same `Telemetry.Metrics` definitions to Prometheus via [PromEx](https://github.com/akoutmos/prom_ex), which is a plugin
over the exact same event/metric model — you’re not learning a second
API, you’re pointing the same concepts at a different reporter.

A custom PromEx plugin for ReqLLM, following the [plugin-writing
guide](https://github.com/akoutmos/prom_ex/blob/master/guides/howtos/Writing%20PromEx%20Plugins.md):

`[](#cb2-1)defmodule MyApp.PromEx.Plugins.ReqLLM do
[](#cb2-2) use PromEx.Plugin
[](#cb2-3)
[](#cb2-4) @impl true
[](#cb2-5) def event_metrics(_opts) do
[](#cb2-6) Event.build(
[](#cb2-7) :req_llm_event_metrics,
[](#cb2-8) [
[](#cb2-9) distribution(
[](#cb2-10) [:req_llm, :request, :duration, :milliseconds],
[](#cb2-11) event_name: [:req_llm, :request, :stop],
[](#cb2-12) measurement: :duration,
[](#cb2-13) unit: {:native, :millisecond},
[](#cb2-14) tags: [:provider],
[](#cb2-15) tag_values: fn meta ->
[](#cb2-16) %{provider: meta[:provider]}
[](#cb2-17) end,
[](#cb2-18) reporter_options: [
[](#cb2-19) buckets: [100, 500, 1_000, 3_000, 10_000]
[](#cb2-20) ]
[](#cb2-21) ),
[](#cb2-22) sum(
[](#cb2-23) [:req_llm, :cost, :usd, :total],
[](#cb2-24) event_name: [:req_llm, :token_usage],
[](#cb2-25) measurement: fn m -> m[:total_cost] || 0 end,
[](#cb2-26) tags: [:provider]
[](#cb2-27) )
[](#cb2-28) ]
[](#cb2-29) )
[](#cb2-30) end
[](#cb2-31)end
```

Register it alongside the built-in plugins in your app’s PromEx
module:

[](#cb3-1)defmodule MyApp.PromEx do
[](#cb3-2) use PromEx, otp_app: :my_app
[](#cb3-3)
[](#cb3-4) @impl true
[](#cb3-5) def plugins do
[](#cb3-6) [
[](#cb3-7) PromEx.Plugins.Application,
[](#cb3-8) PromEx.Plugins.Beam,
[](#cb3-9) PromEx.Plugins.Phoenix,
[](#cb3-10) {MyApp.PromEx.Plugins.ReqLLM, []}
[](#cb3-11) ]
[](#cb3-12) end
[](#cb3-13)end
```

From there it’s standard PromEx: mix prom_ex.gen.config`
if you haven’t set up a datasource yet, and the metrics show up at
`/metrics` for Prometheus to scrape. The buckets on the
duration histogram are the one thing worth tuning by hand. LLM calls run
in the hundreds-of-milliseconds to tens-of-seconds range, not the
single-digit milliseconds Phoenix’s own request histogram defaults
assume, so the stock bucket boundaries will bucket almost every LLM call
into the same “overflow” bucket if you don’t override them.

## If you’re not on ReqLLM: the minimal hand-rolled span

If your app calls a provider’s SDK directly, or does raw
`Req`/`Finch` requests without ReqLLM, you don’t
get the events above for free — but you also don’t need much code to get
equivalent ones. `:telemetry.span/3` does the
start/stop/exception bookkeeping for you:

`[](#cb4-1)defmodule MyApp.LLM do
[](#cb4-2) def complete(prompt, opts \\ []) do
[](#cb4-3) metadata = %{
[](#cb4-4) provider: opts[:provider] || :openai,
[](#cb4-5) model: opts[:model]
[](#cb4-6) }
[](#cb4-7)
[](#cb4-8) :telemetry.span(
[](#cb4-9) [:my_app, :llm, :request],
[](#cb4-10) metadata,
[](#cb4-11) fn ->
[](#cb4-12) response = call_provider(prompt, opts)
[](#cb4-13) usage = extract_usage(response)
[](#cb4-14)
[](#cb4-15) result_meta =
[](#cb4-16) Map.merge(metadata, %{
[](#cb4-17) input_tokens: usage.input_tokens,
[](#cb4-18) output_tokens: usage.output_tokens,
[](#cb4-19) cost: usage.cost
[](#cb4-20) })
[](#cb4-21)
[](#cb4-22) {response, result_meta}
[](#cb4-23) end
[](#cb4-24) )
[](#cb4-25) end
[](#cb4-26)end
```

:telemetry.span/3` fires
`[:my_app, :llm, :request, :start]` before the function runs
and `..., :stop]` after, with `duration` computed
for you — and `..., :exception]` if the function raises,
which matters here because a provider timeout is exactly the failure
mode you want in the dashboard, not just the successes. The map your
function returns becomes the `:stop` event’s metadata — which
is why the code merges the token counts into `metadata` by
hand before returning it; that’s how they make it into the event without
a second `:telemetry.execute` call. Everything downstream —
the `Telemetry.Metrics` definitions, LiveDashboard, PromEx —
is identical to the ReqLLM path; only the event name and the code that
produces it changes.

## The two numbers worth alerting on

Once the metrics exist, resist the urge to build a dashboard with
fifteen panels nobody opens. Two numbers actually change what you
do:

**Cost per day, trending.** A `sum` metric on
cost, viewed daily, catches the failure mode that burns money silently:
a prompt template that ballooned after an edit, a retry loop with no
backoff hammering the most expensive model on every failure, a feature
that got popular faster than anyone expected. This is the same instinct
behind checking inference unit economics in [technical due
diligence](https://sublimecoding.com/blog/surviving-technical-due-diligence-ai-founder) — the number that’s fine at ten calls a day and ruinous at
ten thousand needs a trend line, not a spot check.

**p95 latency per provider.** Not the average — the
average hides the calls that are actually annoying a user right now. A
`distribution` metric with tags on provider (and model, if
you route between them) tells you when a specific provider degrades
before your support inbox does. If you’re running an agent loop that
retries on failure — the kind of durable, idempotent loop [Oban makes a good runtime
for](https://sublimecoding.com/blog/oban-durable-ai-agent-runtime) — a latency spike on one provider is also the leading indicator
that your retry budget is about to get expensive, because every retry is
another billed call on top of the original.

The boilerplate you’d expect to write for LLM telemetry usually
doesn’t need writing — the event already exists, and the gap is five
lines of `Telemetry.Metrics`.

Everything else — token counts by endpoint, breakdown by finish
reason, per-model histograms — is worth having in the dashboard for
debugging, but it’s not worth an alert. If you’re already streaming
tokens to the client with [LiveView’s async
APIs](https://sublimecoding.com/blog/streaming-llm-tokens-liveview-2026), the same PubSub-friendly instinct applies here: don’t build
more real-time surface area than someone is actually going to look
at.

## One more option worth knowing about

If you want tracing and not just metrics — spans that follow a
request through an agent’s tool calls, not just a single provider call —
[`agent_obs`](https://github.com/lostbean/agent_obs)
is a small Elixir library built specifically for this, with
`trace_agent/3` and `trace_llm/3` helpers that
emit OpenTelemetry-compatible spans (with an Arize Phoenix integration
via OpenInference conventions). It’s a different shape of problem than
what this post covers — cost/latency dashboards versus distributed
tracing — but if the next question after this post is “now I need to see
the whole agent run, not just one call,” that’s where I’d look
first.

## Where this fits

None of this requires choosing between “buy Helicone” and “build it
yourself” as an all-or-nothing decision — plenty of teams run both, a
hosted tool for the LLM-specific views (prompt diffing, eval scoring)
and Telemetry.Metrics for the operational numbers that live next to the
rest of their Phoenix metrics anyway. What it does mean is that if
you’re already on the BEAM — and [there are real reasons to
be](https://sublimecoding.com/blog/why-elixir-for-ai-startup-backend), not just aesthetic ones — the telemetry story for LLM calls is
closer to “wire up what’s already there” than “adopt a new
platform.”

If you’re building out an AI product’s operational layer and want a
second set of eyes on what’s actually load-bearing versus what’s
dashboard theater, that’s a conversation I have often as a [fractional CTO for AI startups](https://sublimecoding.com/consulting) — happy to look at
what you’ve got.

---

## Phoenix Secrets: runtime.exs, Fly.io, and LLM Keys

URL: https://sublimecoding.com/blog/phoenix-secrets-runtime-fly
Published: 2026-08-10
Tags: Elixir, security, engineering

>
**TL;DR:** `config.exs` compiles into your
release; `runtime.exs` runs when the release boots. Put a
secret in the wrong one and it either bakes an old value into your
Docker image forever, or vanishes the moment you deploy. The fix is
mechanical once you see it: read every secret in
`config/runtime.exs` with `System.fetch_env!/1`,
not `System.get_env/2` with a fallback — a missing LLM key
should crash your boot at 2pm during a deploy, not surface as a silent
500 to a customer at 2am. `fly secrets set` restarts your
machines to apply it; `--stage` lets you land a secret before
the code that reads it ships. Dotenvy gives dev the same
`.env`-file convenience without touching
`System.get_env` at all, which is exactly where the existing
guides on this stop short. And an Anthropic or OpenAI key is not “just
another env var” — it’s a bearer credential against your billing
account, and it deserves narrower handling than your database URL.

## The trap: config.exs bakes, runtime.exs boots

Every Phoenix app since 1.6 ships two configuration entry points, and
the difference between them is the single most common way a new engineer
breaks a deploy. [`config/config.exs`](https://hexdocs.pm/phoenix/deployment.html)
(and its environment siblings,
`dev.exs`/`prod.exs`) runs at **compile
time**, when `mix release` builds the artifact.
Whatever value `System.get_env("ANTHROPIC_API_KEY")` returns
*on the build machine, at build time* gets frozen into the
release’s compiled config. If you build on CI where that variable
happens to be unset, you get `nil` baked in — permanently,
until the next rebuild — no matter what you later set in the deployed
environment.

`config/runtime.exs` is different by design: [Phoenix executes it
after compilation, right before the application starts](https://hexdocs.pm/phoenix/releases.html), on every
boot, in every environment including a running release on a production
machine. That’s the only place `System.get_env` reads the
*actual* environment the process is running in. This is why the
generated `runtime.exs` already handles
`SECRET_KEY_BASE` this way — it’s not a style choice, it’s
the only correct place to read a value that has to differ between your
laptop and the fly machine that’s actually serving traffic.

The trap is subtle because it doesn’t fail loudly. You put an LLM key
in `config.exs` because that’s the file you had open,
`mix phx.server` works fine locally (your shell has the env
var, and dev doesn’t build a release), CI passes, the deploy succeeds —
and then the first production request that calls the model gets a 401
from a key that was `nil` at build time. Nothing in that
chain complains until a real user hits it.

## Fail at boot, not at 2am: `fetch_env!` over `get_env`

Once a secret is correctly in `runtime.exs`, there’s a
second decision that matters just as much: what happens when it’s
*missing*.

`[](#cb1-1)# config/runtime.exs — wrong instinct: silently proceed
[](#cb1-2)anthropic_key = System.get_env("ANTHROPIC_API_KEY", "")
```

This “works.” The app boots. Every request to your agent endpoint
returns a vague provider error, or worse, silently degrades to a stub
response, and you find out from a support ticket instead of a deploy
log.

[](#cb2-1)# config/runtime.exs — crash the boot instead
[](#cb2-2)anthropic_key = System.fetch_env!("ANTHROPIC_API_KEY")
[](#cb2-3)
[](#cb2-4)config :sublime_coding, :anthropic, api_key: anthropic_key
```

System.fetch_env!/1` raises `ArgumentError`
immediately if the variable isn’t set, which means the release refuses
to start at all. On Fly, a release that crashes on boot fails the
deploy’s health check and the rollout stops — you see it in
`fly logs` and `fly status` before a single
customer request lands on the bad machine, because [Fly won’t route traffic to a
machine that never comes up healthy](https://fly.io/docs/launch/deploy/). That is the failure mode you
want: a loud, deploy-time crash you catch in your terminal, not a quiet,
request-time failure someone else catches for you.

This generalizes past LLM keys. Any secret without which the app is
*meaningfully* broken — the database URL,
`SECRET_KEY_BASE`, an LLM key that gates a core feature —
should use `fetch_env!`. Reserve
`System.get_env/2` with a real default for genuinely optional
config: an analytics ID, a feature flag, a port number with a sane
fallback. This repo’s own `runtime.exs` draws exactly that
line — `PORT` gets
`System.get_env("PORT", "4000")` because a default port is
harmless, while `SECRET_KEY_BASE` gets a `raise`
with a message telling you how to generate one, because a missing
signing key is not something you want silently defaulted in
production.

A missing secret should be a deploy-time crash you catch in your
terminal, not a request-time failure your customer catches for you.

## `fly secrets set`, staged secrets, and what actually restarts

Fly.io’s secrets are the thing `runtime.exs` reads at boot
— they’re injected as process environment variables on the machine,
nothing more exotic. The workflow:

`[](#cb3-1)fly secrets set ANTHROPIC_API_KEY=sk-ant-... \
[](#cb3-2) OPENAI_API_KEY=sk-proj-...
```

[This does two things](https://fly.io/docs/apps/secrets/):
it writes the secret into the app’s encrypted vault, then immediately
restarts every machine in the app to pick it up — which also resets each
machine’s ephemeral filesystem, worth knowing if you keep any scratch
state outside a volume. That restart-on-set behavior is why you don’t
want to fly secrets set` casually against a busy production
app mid-incident; it’s a real deploy event, not a config tweak.

Two situations where the default behavior is wrong for you:

**Setting a secret before the code that reads it
ships.** If you’re adding a new `ANTHROPIC_API_KEY`
read in `runtime.exs` in the same release that needs the
secret, setting it normally works fine — the restart-and-the-deploy can
happen in either order. But if you want to stage the secret
*ahead* of a deploy without triggering an extra restart cycle
right now, `--stage` sets the value in the vault without
touching running machines:
`fly secrets set ANTHROPIC_API_KEY=sk-... --stage`. [The staged value applies the
next time a machine starts or updates](https://fly.io/docs/apps/secrets/) — your next
`fly deploy` picks it up naturally, or you can force it early
with `fly secrets deploy`, which redeploys the current
release with the staged secrets without rebuilding the image.

**Rotating a leaked or expiring key.** Same command, no
`--stage` — you want the restart immediately:
`fly secrets set ANTHROPIC_API_KEY=sk-new-...`. The old key
stops being read the moment the new machines come up. This is also the
moment `fetch_env!` earns its keep on the other end: if you
typo the new key name, the next deploy crashes at boot instead of
quietly running every LLM call against a key that no longer exists.

`fly secrets list` shows names, digests, and set-times —
never plaintext values, by design. If you need to confirm *what*
a secret’s value is, you kept it somewhere else (a password manager, not
Fly) or you’re regenerating it, not reading it back.

## Dev: Dotenvy gives you `.env` files without touching `System.get_env`

Everything above covers the release. Locally, you don’t want to
`export` five LLM keys into your shell every session, and you
don’t want a `.env` file that quietly leaks into
`System.get_env` and gets read the same way production
secrets are — because then a stray `git add .env` is a
production-shaped incident. This is precisely the gap the existing
Phoenix-secrets writeups leave: they cover the compile/runtime split,
then wave at “use a `.env` file in dev” without saying how to
keep that convenience from bleeding into your runtime read path.

[Dotenvy](https://dotenvy.hexdocs.pm/getting_started.html)
closes it cleanly because it’s read-only by design — it never calls
`System.put_env/2`, so `.env`-sourced values are
invisible to plain `System.get_env` or
`System.fetch_env!` calls elsewhere in your app. You read
them only through Dotenvy’s own `env!/2`, which forces you to
declare the type and the requirement inline:

`[](#cb4-1)# config/runtime.exs
[](#cb4-2)import Config
[](#cb4-3)import Dotenvy
[](#cb4-4)
[](#cb4-5)env_dir = System.get_env("RELEASE_ROOT") || Path.expand(".")
[](#cb4-6)
[](#cb4-7)source!([
[](#cb4-8) Path.absname(".env", env_dir),
[](#cb4-9) Path.absname(".#{config_env()}.env", env_dir),
[](#cb4-10) System.get_env()
[](#cb4-11)])
[](#cb4-12)
[](#cb4-13)config :sublime_coding, :anthropic,
[](#cb4-14) api_key: env!("ANTHROPIC_API_KEY", :string!)
```

source!/1` layers files in order — a shared
`.env`, then an env-specific `.dev.env` or
`.test.env` overriding it, then the real
`System.get_env()` last so production’s actual environment
always wins over anything a stray `.env` file might contain.
`:string!` (the bang type, not the bang function) enforces
non-empty, same intent as `fetch_env!` — missing or blank
crashes the boot instead of handing you `nil`. In prod
there’s typically no `.env` file on disk at all, so
`source!/1` layers in nothing but
`System.get_env()`, and every value still flows through the
exact same `env!` call — one code path for both environments,
which is the actual point of doing it this way instead of branching on
`config_env()` yourself.

`.env` and `.dev.env` still belong in
`.gitignore`, same as they always did — Dotenvy changes how
the value gets *read*, not whether the file belongs in git.

## LLM keys specifically: narrower rules than a database URL

Everything so far applies to any secret. LLM provider keys earn extra
caution for two reasons that don’t apply to, say, your Postgres
password.

**They’re bearer credentials against a metered bill, not a
fixed resource.** A leaked database URL gets you rows; a leaked
Anthropic or OpenAI key gets an attacker your billing account. When a
corporate-tier key leaks, [threat
actors run high-volume inference workloads against it and drain the
account before anyone notices](https://www.vicarius.io/articles/8-000-chatgpt-api-keys-exposed-across-github-production-sites) — there’s no row count to bound the
damage, only your provider’s rate limit and however long detection
takes. GitGuardian’s most recent scan of public GitHub found AI-service
secrets growing 81% year over year, faster than almost every other
credential category they track, which tells you the scanning bots are
tuned for exactly this key shape now — [`sk-ant-`
and `sk-proj-` prefixes are as fingerprintable as AWS’s
`AKIA`](https://blog.gitguardian.com/the-state-of-secrets-sprawl-2026/).

**Never in `config.exs`, full stop — including for
local convenience.** The compile-time trap above is annoying when
it’s `SECRET_KEY_BASE`; it’s actively dangerous for an LLM
key, because a key baked into a compiled release artifact ships with
every copy of that artifact — any Docker image, any build cache, any
layer you push to a registry. Read it in `runtime.exs` — a
direct `fetch_env!` in prod or `env!` behind
Dotenvy in dev.

**Scope the key to what actually needs it, and rotate on a
schedule you keep.** Most providers let you name and scope API
keys per project or per use case — use that, so a leaked key from your
staging agent isn’t also your production billing key. And rotate
proactively rather than reactively; the providers’ own detection catches
a lot of leaks fast, [OpenAI’s
partner program with GitHub revokes recognized key patterns within
minutes of detection](https://safeguard.sh/resources/blog/openai-api-key-leakage-on-github-at-scale), but that safety net only covers keys that
match a known pattern in a place GitHub scans — it does nothing for a
key sitting in your Fly secrets vault that an ex-contractor still has a
local copy of.

This post is about the boring, load-bearing layer: getting the key
into the *process* safely. It doesn’t cover what happens after
your application holds the key — whether the model itself ever sees it,
whether a tool call can leak it back out through a crafted prompt. I’ve
written that half separately: [secrets management for AI
agents](https://sublimecoding.com/blog/ai-agent-secrets-management) covers the broker pattern that keeps a credential scoped to
the tool that needs it and out of the model’s context entirely.
Different layer, same root discipline — this post gets the key from Fly
into your BEAM process correctly; that one governs what your code does
with it once it’s there.

## The full picture, in order

Put together, the sequence for adding a new LLM key to a Phoenix app
on Fly looks like this:

- Add `System.fetch_env!("ANTHROPIC_API_KEY")` (or
`env!` via Dotenvy in dev) to `config/runtime.exs`
— never `config.exs`.

- `fly secrets set ANTHROPIC_API_KEY=sk-ant-...` before or
alongside the deploy that reads it; use `--stage` if you want
to land the secret ahead of the code without an extra restart.

- Deploy. If the key is missing or mistyped, the release fails to boot
and the rollout stops — check `fly logs` before assuming it’s
a code problem.

- Scope the key narrowly at the provider, note it in whatever runbook
tracks what’s set where, and put its rotation on the same calendar as
your other production credentials.

None of this is exotic Elixir. It’s the same discipline that governs
`SECRET_KEY_BASE` in every `mix phx.new` app,
applied on purpose to the credential that’s most likely to be the one
you added under deadline pressure without thinking about where it goes.
If you’re standing up an AI feature on Phoenix and want a second pair of
eyes on the security posture around it — this layer and the ones above
it, like the [AI-native
security stack](https://sublimecoding.com/blog/running-security-at-an-ai-native-company-2026) or the [50-line agent loop](https://sublimecoding.com/blog/build-ai-agent-loop-50-lines-elixir)
this key usually feeds — [that’s a conversation
worth having before the first deploy, not after the first leak](https://sublimecoding.com/consulting).

---

## Multi-Tenant AI SaaS in Phoenix: Isolation That Holds

URL: https://sublimecoding.com/blog/multi-tenant-ai-saas-phoenix
Published: 2026-08-06
Tags: Elixir, AI startups, security

>
**TL;DR:** Row-level tenant isolation in Postgres is a
solved problem — a `tenant_id` column, an index, and
discipline. What’s not solved, because almost nobody has written it
down, is what happens to that isolation once every request also flows
through an LLM call and a retrieval query. Three places break in ways
ordinary Ecto hygiene doesn’t catch: your provider API keys (one shared
key means one shared blast radius and one shared rate-limit pool), the
context-building code that assembles a prompt (easy to leave an unscoped
call site if scoping isn’t structural), and — the sharp one — a RAG
retrieval query that’s flawless Ecto and still returns another tenant’s
documents, because storage was scoped correctly and the read wasn’t.
None of this shows up as an error. It shows up as tenant A’s onboarding
contract answering tenant B’s support question, with a clean log and a
passing test suite that never tested the thing that mattered.

## The question a diligence reviewer already asks you

If you’ve read [what enterprise
security reviews actually probe on an AI startup](https://sublimecoding.com/blog/enterprise-security-reviews-ai-startups), you’ve seen the
line that stops more deals than any other item on the questionnaire:
*does one customer’s data ever influence another customer’s
outputs?* It’s a fair question and a specific one, and “we use
Postgres row-level scoping, we’re fine” is not actually an answer to it
— it’s an answer to a different, easier question. This post is the
implementation answer: what tenant isolation has to look like once the
data doesn’t just sit in a table, it flows through a prompt and a vector
search on its way to another tenant’s chat window.

I’m going to spend one paragraph on the storage-layer decision and
then leave it, because better posts already own that ground. Everything
after that paragraph is the part that’s specific to AI products, and
it’s the part I don’t see written up anywhere with actual code.

## Schema-per-tenant vs. shared schema: not this post

You have two mainstream options for where tenant data physically
lives in Postgres: a shared schema with a `tenant_id` column
on every table (cheap to run, cheap to migrate, isolation enforced by
your queries), or schema-per-tenant using Postgres’s native schema
namespaces and Ecto’s `prefix` option (stronger blast-radius
containment, more expensive migrations, a ceiling on how many tenants
you can comfortably run). [Curiosum’s
multi-tenancy guide](https://curiosum.com/blog/multitenancy-in-elixir) and [Ecto’s
own query-prefixes howto](https://hexdocs.pm/ecto/multi-tenancy-with-query-prefixes.html) cover that tradeoff well and I’m not going
to re-litigate it. Pick shared-schema unless you have a specific reason
not to — a small number of large regulated tenants, usually — and move
on. The rest of this post assumes
shared-schema-plus-`tenant_id`, because that’s what most AI
SaaS products actually run, and because the interesting failures live
one layer up from that decision either way.

## Per-tenant provider keys vs. one shared key

The first AI-specific isolation decision most teams make by accident:
one API key to OpenAI or Anthropic, shared across every tenant, with
usage attributed after the fact in your own logs. It’s the obvious
starting point — one secret to provision, one bill to reconcile — and
it’s fine for a while. It stops being fine at a predictable point, and
it’s worth deciding on purpose instead of drifting into it.

The core tradeoff is blast radius versus operational overhead:

Shared key, attribution in your logs
Per-tenant key (or project)

Blast radius of a leaked key
Every tenant’s traffic and spend
One tenant’s traffic and spend

Rate-limit isolation
One noisy tenant throttles everyone
Each tenant hits its own ceiling

Ops overhead
One secret to rotate
N secrets to provision, rotate, monitor

Usage visibility
You build the attribution layer
Provider dashboard does it per key/project

Best fit
Early stage, low tenant count, uniform usage
Regulated or high-usage tenants, noisy-neighbor risk

The rate-limit line is the one people get wrong, so it’s worth being
precise. [OpenAI
enforces rate limits at the organization and project level, not per raw
API key](https://developers.openai.com/api/docs/guides/rate-limits) — a project carries its own token and request-rate
allocation, separate from other projects in the same org. So “per-tenant
key” isolation in practice means giving a tenant (or a tenant tier —
enterprise tenants get their own project, everyone else shares one) its
own project, not just a distinct-looking secret pointed at the same
shared pool. Do that with a shared key instead, and one tenant running a
batch job at 2am throttles every other tenant’s live traffic through the
same 429s, and you’ll spend the incident review explaining why “we have
rate limiting” didn’t stop it.

You don’t need per-tenant keys from day one. You need to know which
side of that line you’re on, and you need it to be a decision, not the
accidental consequence of copying `OPENAI_API_KEY` into one
`.env` file and never revisiting it. The trigger to move off
shared is usually the first tenant whose contract asks about it
directly, or the first noisy-neighbor incident — whichever comes
first.

## Pin tenant_id where the prompt gets built, not at every call site

The second failure mode is more insidious because it doesn’t look
like a security bug — it looks like a normal Ecto query that someone
wrote slightly wrong. The pattern I’ve seen repeated across teams:
`tenant_id` scoping gets added correctly at some call sites
and quietly dropped at others, because scoping is enforced by
*convention* — “remember to filter by tenant” — instead of by the
shape of the code.

The fix is the same discipline I laid out for [the has_many/belongs_to
split in Ecto](https://sublimecoding.com/blog/ecto-query-patterns-n-plus-one): put the rule where a call site can’t accidentally
skip it, not in a comment reminding the next engineer to remember. For
tenant scoping specifically, that means the tenant-scoped queryable is
the *only* entry point into the context — there is no unscoped
function to reach for by mistake:

`[](#cb1-1)defmodule MyApp.RAG.Query do
[](#cb1-2) import Ecto.Query
[](#cb1-3) import Pgvector.Ecto.Query
[](#cb1-4)
[](#cb1-5) # The only way to get a Chunk queryable.
[](#cb1-6) # There is no unscoped version to reach
[](#cb1-7) # for by accident.
[](#cb1-8) def for_tenant(tenant_id) do
[](#cb1-9) from c in MyApp.RAG.Chunk,
[](#cb1-10) where: c.tenant_id == ^tenant_id
[](#cb1-11) end
[](#cb1-12)
[](#cb1-13) def nearest(tenant_id, embedding, k \\ 5) do
[](#cb1-14) for_tenant(tenant_id)
[](#cb1-15) |> order_by(
[](#cb1-16) [c],
[](#cb1-17) cosine_distance(c.embedding, ^embedding)
[](#cb1-18) )
[](#cb1-19) |> limit(^k)
[](#cb1-20) end
[](#cb1-21)end
```

Every function that touches Chunk` starts from
`for_tenant/1`, so `tenant_id` isn’t a
`WHERE` clause someone remembers to add — it’s a parameter
you can’t call the module without providing. Context-boundary functions
that assemble the actual LLM prompt — pulling retrieved chunks, prior
messages, tenant-specific system prompt fragments — should take the
authenticated tenant from *your* code (the current session, never
a model- or client-supplied argument) and pass it straight into
`for_tenant/1`. That’s the same rule from [prompt injection
defense for AI startups](https://sublimecoding.com/blog/prompt-injection-defense-ai-startup): scope every tool and every query to the
session’s user, taken from code you control, not from anything the model
or the request body hands you.

## The bug that passes code review: storage scoped, retrieval isn’t

Here’s the failure mode that’s specific to RAG, and it’s the one that
scares me most because it’s invisible in a diff. Say your ingestion
pipeline is disciplined — every chunk is written with
`tenant_id` set, there’s a foreign key, there’s an index,
code review would catch a write that skipped it. And say your retrieval
function is *also* well-written Ecto: correct
`cosine_distance` call, correct `limit`, no N+1,
nothing a linter or a reviewer would flag.

`[](#cb2-1)# Clean Ecto. Wrong result. Nobody catches
[](#cb2-2)# this in review because nothing about it
[](#cb2-3)# looks wrong.
[](#cb2-4)def nearest_unscoped(embedding, k \\ 5) do
[](#cb2-5) from(c in MyApp.RAG.Chunk,
[](#cb2-6) order_by: cosine_distance(
[](#cb2-7) c.embedding, ^embedding
[](#cb2-8) ),
[](#cb2-9) limit: ^k
[](#cb2-10) )
[](#cb2-11) |> MyApp.Repo.all()
[](#cb2-12)end
```

This function is missing exactly one where` clause, and
that’s the entire bug. It will never error, because top-k cosine
similarity across the whole `chunks` table always returns
*something* — it just returns the nearest chunks from every
tenant, not the calling tenant. Storage is scoped perfectly. The write
path has a foreign key and an index proving it. The read path — the one
query that decides what actually lands in tenant B’s context window —
has none of that, because nothing about a missing `WHERE`
shows up as a schema violation, a type error, or a slow query. It shows
up as tenant A’s onboarding contract or support transcript answering
tenant B’s question, word for word, in a response your product renders
as if the model just knew the answer.

A query that’s flawless Ecto and completely wrong policy is the
failure mode multi-tenant RAG hides best — nothing about it looks broken
until you go looking for the tenant that isn’t supposed to be in the
result set.

This is the same architecture I described in the [hand-rolled pgvector post](https://sublimecoding.com/blog/phoenix-rag-pgvector-arcana):
retrieval is one Ecto query, which is the whole appeal — composable,
`WHERE`-able, no separate metadata-filter DSL to fight. That
composability is exactly why the failure is so cheap to introduce:
adding a filter is a one-line change, so is *forgetting* one, and
the query that forgot looks identical in a code review to the one that
didn’t, unless the reviewer is specifically checking for a
`tenant_id` clause every single time. At scale, “check for it
every time” is not a control. A structural rule — one queryable, no
unscoped path in — is.

## Prove it: the test that has to exist

If tenant isolation on the retrieval path matters — and per the
security-questionnaire framing above, it’s often the single question
that decides whether a regulated buyer signs — then it needs a test that
fails the moment someone reintroduces `nearest_unscoped/2`,
adds a join that drops the `where`, or refactors the query
module and loses the guarantee. Not a manual QA pass before launch. A
test that runs on every commit, forever.

`[](#cb3-1)defmodule MyApp.RAG.QueryTest do
[](#cb3-2) use MyApp.DataCase, async: true
[](#cb3-3)
[](#cb3-4) alias MyApp.RAG.Query
[](#cb3-5)
[](#cb3-6) test "retrieval never crosses tenants" do
[](#cb3-7) tenant_a = insert(:tenant)
[](#cb3-8) tenant_b = insert(:tenant)
[](#cb3-9)
[](#cb3-10) insert(:chunk,
[](#cb3-11) tenant_id: tenant_a.id,
[](#cb3-12) body: "tenant a's onboarding contract"
[](#cb3-13) )
[](#cb3-14) insert(:chunk,
[](#cb3-15) tenant_id: tenant_b.id,
[](#cb3-16) body: "tenant b's support policy"
[](#cb3-17) )
[](#cb3-18)
[](#cb3-19) [embedding] = embed!(["onboarding"])
[](#cb3-20)
[](#cb3-21) leaked =
[](#cb3-22) Query.nearest(tenant_b.id, embedding, 10)
[](#cb3-23) |> MyApp.Repo.all()
[](#cb3-24) |> Enum.filter(
[](#cb3-25) &(&1.tenant_id == tenant_a.id)
[](#cb3-26) )
[](#cb3-27)
[](#cb3-28) assert leaked == []
[](#cb3-29) end
[](#cb3-30)end
```

Stub embed!/1` to a deterministic fixture so this doesn’t
hit a real embeddings API in CI — the assertion doesn’t depend on which
embedding you get back, only on whether the result set ever contains a
row from `tenant_a`. That’s the whole test. It’s cheap to
write, it runs in milliseconds, and it’s the one piece of evidence that
turns “we scope by tenant” from a claim in a security questionnaire into
something you can point at. A reviewer who asks the tenant-isolation
question and gets shown this test, passing, is having a very different
conversation than one who gets shown an architecture diagram.

## Where this actually gets tested is the interview, not the demo

The demo will never expose this bug, because a demo has one tenant,
or a handful of friendly ones with nothing sensitive in the same corpus.
It surfaces in production, at the exact moment two real customers’
documents happen to be semantically close enough that the top-k result
set for one includes the other — and the person who finds it first is
either your customer’s security team during an enterprise deal, or,
worse, the customer themselves, reading an answer in their chat window
that quotes language they’ve never seen before and don’t recognize.

Cross-tenant retrieval leakage is also a sharper problem than the [prompt-injection
trifecta](https://sublimecoding.com/blog/prompt-injection-defense-ai-startup) I’ve written about elsewhere, precisely because it needs no
attacker. [Simon
Willison’s framing](https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/) — private data, untrusted content, and external
communication converging in one context — describes an adversary
exploiting a gap. This doesn’t require anyone malicious. It requires one
missing `WHERE` clause and a corpus large enough that the gap
gets exercised by ordinary traffic. That makes it a baseline hygiene
problem you solve before you get anywhere near adversarial threat
modeling, not an advanced case you defer until you have a security
hire.

## What this buys you in the room that matters

Go back to the questionnaire line this post opened with: *does one
customer’s data ever influence another customer’s outputs?* Most AI
startups answer it with an architecture diagram and a sentence about
Postgres row-level scoping, and a reviewer who’s read a few of these
knows that answer describes the write path and says nothing about the
read path — which is exactly where the gap in this post lives. The
better answer is structural: one scoped queryable per tenant-touching
resource, no unscoped path into it, and a test in CI that fails the day
someone adds one. That’s not a bigger engineering lift than the naive
version — it’s roughly the same code, written so a call site can’t skip
the part that matters, plus one test file.

If you’re staffing this now, or you’ve got an enterprise security
review on the calendar and you’re not confident your retrieval path
would survive the tenant-isolation question asked directly, that’s
exactly the kind of architecture review I run as a fractional CTO for AI
startups. [Let’s talk about what that looks like
for your stack](https://sublimecoding.com/consulting) before the reviewer asks the question you haven’t
tested the answer to.

---

## Structured LLM Output in Elixir: Which Library, When

URL: https://sublimecoding.com/blog/structured-llm-output-elixir
Published: 2026-08-17
Tags: Elixir, AI, AI tools

>
**TL;DR:** You have four real options for getting an LLM
to return a validated Elixir struct instead of a string you regex apart,
and the SERP for this query is HexDocs and GitHub — nobody’s scored them
against each other. [InstructorEx](https://github.com/thmsmlr/instructor_ex) is the
well-known name (781 stars) but its last commit was over a year ago and
its latest Hex release predates that. [InstructorLite](https://github.com/martosaur/instructor_lite)
is the fork that’s actually shipping — its repo was last pushed July 9,
2026. [ReqLLM](https://github.com/agentjido/req_llm)’s
`generate_object/4` needs no extra dependency if you’re
already using it as your LLM client, but I read the source and it does
**not** retry the model on a validation failure the way
both Instructor libraries do — it locally coerces types and gives up,
which most people evaluating it are not expecting. And the fourth option
is the one I already pointed at in [15 Elixir Libraries I
Reach For in 2026](https://sublimecoding.com/blog/elixir-libraries-i-reach-for-2026): Req plus an Ecto changeset plus a hand-rolled
retry loop, in about 50 lines, with zero new dependencies. Here’s the
scoring, with the receipts.

## This isn’t a tutorial on using Instructor

If you searched “elixir structured llm output” to find out how to
call `Instructor.chat_completion/1`, HexDocs already owns
that — the [quickstart](https://github.com/thmsmlr/instructor_ex/blob/main/pages/quickstart.livemd)
walks the basic Ecto-schema-plus-completion-call pattern better than I’m
going to re-explain it here. This post answers the question one layer
up, the one nobody’s written because it requires actually opening four
repos and comparing them instead of writing up whichever one you already
knew about: **which of these should a new Elixir project actually
depend on, and when should it depend on none of them?**

That question matters more than the tutorial does, because the wrong
answer here isn’t “slightly slower code” — it’s a dependency that stops
getting patched while your product’s core reliability mechanism (does
the model’s output match the shape my database expects) sits on top of
it.

## What I actually checked before scoring anything

Every claim below is pulled from the GitHub API
(`pushed_at`, star count, open issue count), Hex.pm’s package
pages, the libraries’ own READMEs, and — for the one claim that
surprised me — the actual `generate_object` source in
ReqLLM’s `lib/`. Nothing here is recalled from a training
cutoff; library maintenance state moves fast enough in this ecosystem
that a six-month-old memory is actively misleading. Every date and count
below was checked on July 13, 2026 — re-verify before you commit to a
dependency; that’s the whole point.

## Option 1: InstructorEx — the name everyone knows, the one that’s stalled

[`thmsmlr/instructor_ex`](https://github.com/thmsmlr/instructor_ex),
published to Hex as [`instructor`](https://hex.pm/packages/instructor),
is the library [my
libraries roundup](https://sublimecoding.com/blog/elixir-libraries-i-reach-for-2026) called “early-stage” a few weeks ago. Rechecking
it now: 781 stars, but the GitHub API’s `pushed_at` reads
**2025-06-07** — over a year of no commits — and the latest
tagged Hex release is `0.1.0`, shipped
**2025-02-09**. GitHub’s own issue count (open issues plus
PRs, its convention) sits at 40. That’s not “quiet because it’s done”
the way [Mox](https://hex.pm/packages/mox) is quiet; it’s a
maintainer who’s moved on, with a real backlog nobody’s triaging.

The odd part: it still pulls **214,720 total downloads**
and **7,160 in the last 7 days** on Hex. That’s not new
adoption — it’s every existing project’s `mix.lock` still
resolving to it on `mix deps.get`. Download count is a
lagging indicator here, not a health signal.

Providers, per the README: OpenAI, Anthropic, Groq, Ollama, Gemini,
vLLM, llama.cpp. The API is the one you’ve probably seen — an
`Ecto.Schema` with a `use Instructor.Validator`, a
`validate_changeset/1` callback for cross-field checks, and a
`max_retries` option that re-prompts the model with the
validation errors on failure:

`[](#cb1-1)defmodule SpamPrediction do
[](#cb1-2) use Ecto.Schema
[](#cb1-3) use Instructor.Validator
[](#cb1-4)
[](#cb1-5) @primary_key false
[](#cb1-6) embedded_schema do
[](#cb1-7) field(:class, Ecto.Enum,
[](#cb1-8) values: [:spam, :not_spam])
[](#cb1-9) field(:reason, :string)
[](#cb1-10) field(:score, :float)
[](#cb1-11) end
[](#cb1-12)
[](#cb1-13) @impl true
[](#cb1-14) def validate_changeset(changeset) do
[](#cb1-15) Ecto.Changeset.validate_number(
[](#cb1-16) changeset, :score,
[](#cb1-17) greater_than_or_equal_to: 0.0,
[](#cb1-18) less_than_or_equal_to: 1.0
[](#cb1-19) )
[](#cb1-20) end
[](#cb1-21)end
[](#cb1-22)
[](#cb1-23)Instructor.chat_completion(
[](#cb1-24) model: "gpt-4o-mini",
[](#cb1-25) response_model: SpamPrediction,
[](#cb1-26) max_retries: 3,
[](#cb1-27) messages: [
[](#cb1-28) %{role: "user", content: email_text}
[](#cb1-29) ]
[](#cb1-30))
```

That retry behavior is genuinely good design — verified, working,
feeding the changeset’s own errors back into the next prompt. The
problem isn’t the API. It’s that you’d be starting a new project in
August 2026 on a dependency whose maintainer hasn’t touched it since
before this post’s own site existed.

## Option 2: InstructorLite — the fork that’s actually shipping

[martosaur/instructor_lite`](https://github.com/martosaur/instructor_lite)
describes itself, in its own README, as “a fork, spiritual successor,
and almost an entire rewrite” of InstructorEx — and the maintenance
numbers back that framing up. GitHub’s `pushed_at`:
**2026-07-09**, four days before I started researching this
post. The [Hex page](https://hex.pm/packages/instructor_lite)
shows the latest release, `v1.2.0`, landed **February
1, 2026**, on a cadence of roughly every one to three months back
to its `v0.1.0` in September 2024. 140 stars,
**zero** open issues. Total downloads are smaller — 62,935
— but that’s a younger, narrower rewrite against an incumbent with a
two-year head start, not a signal of thin adoption.

Provider coverage is narrower and explicit rather than aspirational:
OpenAI, Anthropic, Gemini, and “any Chat Completions-compatible API”
(which covers Grok and most self-hosted OpenAI-shaped endpoints). The
README states plainly that it “facilitates generating prompts, calling
LLMs, casting and validating responses, including retrying prompts when
validation fails” — so the retry-with-feedback behavior InstructorEx
has, InstructorLite keeps.

`[](#cb2-1)defmodule UserInfo do
[](#cb2-2) use Ecto.Schema
[](#cb2-3) use InstructorLite.Instruction
[](#cb2-4)
[](#cb2-5) @primary_key false
[](#cb2-6) embedded_schema do
[](#cb2-7) field(:name, :string)
[](#cb2-8) field(:age, :integer)
[](#cb2-9) end
[](#cb2-10)end
[](#cb2-11)
[](#cb2-12)InstructorLite.instruct(
[](#cb2-13) %{input: [
[](#cb2-14) %{role: "user", content: text}
[](#cb2-15) ]},
[](#cb2-16) response_model: UserInfo,
[](#cb2-17) adapter_context: [
[](#cb2-18) api_key: System.fetch_env!("OPENAI_KEY")
[](#cb2-19) ]
[](#cb2-20))
```

What I’d actually flag for anyone evaluating it: the project’s own
positioning is “lean,” “composable,” “magic-free” — its docs explicitly
say it does little enough that “it makes you question if you should just
write your own version.” That’s an unusually honest thing for a library
to say about itself, and it’s the right instinct for exactly the reason
[the libraries
roundup](https://sublimecoding.com/blog/elixir-libraries-i-reach-for-2026) argues for wrapping any Instructor variant behind your own
boundary module in the first place — a thin, composable library is
cheaper to swap out from behind that boundary than a thick, opinionated
one.

## Option 3: ReqLLM’s generate_object` — free if you’re already on it, with a catch

[`agentjido/req_llm`](https://github.com/agentjido/req_llm)
is the library behind [the
telemetry events I wrote up separately](https://sublimecoding.com/blog/phoenix-llm-telemetry) — if you’re already sending
LLM calls through it for provider abstraction or streaming, ReqLLM
already gives you telemetry too, and structured output is a function
call away with no new dependency. Provider abstraction is the real purchase here: absorbing protocol churn on your behalf is [the recurring tax on Elixir tooling](https://sublimecoding.com/blog/elixir-coding-harness) that talks to model APIs directly. Its numbers are the strongest of the
four: **1.17.1** released **July 6, 2026**,
`pushed_at` on the repo is *today*, 544 stars, and per
its own docs it covers **21 providers exposing 1,205
models** — more raw surface area than either Instructor
variant.

`generate_object/4` takes a model spec, a prompt (or
messages), and a schema — but the schema is a [NimbleOptions](https://hexdocs.pm/nimble_options) keyword list
or a raw JSON Schema map, not an `Ecto.Schema`. That’s a real
design difference, not a detail: if your codebase already validates
everything at the boundary with changesets (the pattern [the libraries roundup
argues for](https://sublimecoding.com/blog/elixir-libraries-i-reach-for-2026)), ReqLLM’s structured output speaks a different
validation dialect than the rest of your app.

`[](#cb3-1)schema = [
[](#cb3-2) name: [type: :string, required: true],
[](#cb3-3) age: [type: :pos_integer, required: true]
[](#cb3-4)]
[](#cb3-5)
[](#cb3-6)ReqLLM.generate_object!(
[](#cb3-7) "anthropic:claude-sonnet-5",
[](#cb3-8) "Extract: John Doe is 42",
[](#cb3-9) schema
[](#cb3-10))
[](#cb3-11)#=> %{name: "John Doe", age: 42}
```

Here’s the part worth slowing down for. Both Instructor libraries
re-prompt the model on a validation failure — that’s the headline
feature of the whole category. I went and read ReqLLM’s
generate_object` implementation directly rather than trust a
README summary, because getting this wrong in either direction (assuming
retry that isn’t there, or missing retry that is) is exactly the kind of
thing that ships a bug six weeks later. What it actually does on a type
mismatch is attempt local coercion — and if coercion fails, the code
path returns `{:ok, coerced_data}` anyway rather than looping
back to the model with the error.

ReqLLM’s `generate_object` doesn’t re-prompt the model on
a validation failure — it coerces locally and, if that fails too, hands
you back the malformed data as a success.

That’s not a bug, it’s a different design goal — ReqLLM is a
general-purpose provider client with structured output as one feature
among many, not a library purpose-built around the retry-on-invalid
workflow. But it means if you pick ReqLLM for structured output
specifically *because* it’s the library you’re already using, you
need to add your own outer retry loop for the validation-failure case,
the same one you’d write for option four below. Skipping that step is
the single most likely way this option burns someone.

## Option 4: raw JSON mode + Ecto changeset + a retry loop — the ~50-line escape hatch

This is the paragraph from [the libraries roundup](https://sublimecoding.com/blog/elixir-libraries-i-reach-for-2026)
grown up: *“If you’d rather not take the dependency, the pattern is
reproducible in ~50 lines with Req + Ecto changesets + a retry
loop.”* Here’s that pattern, in full, using nothing that isn’t
already a default dependency on this site’s own stack — [Req](https://hex.pm/packages/req) and [Ecto](https://hex.pm/packages/ecto). The shape is close to [the 50-line agent
loop](https://sublimecoding.com/blog/build-ai-agent-loop-50-lines-elixir)’s own guardrail pattern: a recursive function, an explicit step
counter, and a hard stop.

`[](#cb4-1)defmodule LLM.Structured do
[](#cb4-2) @max_retries 3
[](#cb4-3)
[](#cb4-4) def extract(schema_mod, prompt) do
[](#cb4-5) loop(schema_mod, prompt, nil, 0)
[](#cb4-6) end
[](#cb4-7)
[](#cb4-8) defp loop(schema_mod, prompt, err, n)
[](#cb4-9) when n < @max_retries do
[](#cb4-10) text = call(schema_mod, prompt, err)
[](#cb4-11)
[](#cb4-12) case Jason.decode(text) do
[](#cb4-13) {:ok, data} ->
[](#cb4-14) cs = schema_mod.changeset(
[](#cb4-15) struct(schema_mod), data
[](#cb4-16) )
[](#cb4-17)
[](#cb4-18) if cs.valid? do
[](#cb4-19) {:ok, Ecto.Changeset.apply_changes(cs)}
[](#cb4-20) else
[](#cb4-21) loop(schema_mod, prompt,
[](#cb4-22) format_errors(cs), n + 1)
[](#cb4-23) end
[](#cb4-24)
[](#cb4-25) {:error, reason} ->
[](#cb4-26) loop(schema_mod, prompt,
[](#cb4-27) inspect(reason), n + 1)
[](#cb4-28) end
[](#cb4-29) end
[](#cb4-30)
[](#cb4-31) defp loop(_schema_mod, _prompt, err, _n),
[](#cb4-32) do: {:error, {:max_retries, err}}
[](#cb4-33)
[](#cb4-34) defp call(schema_mod, prompt, nil) do
[](#cb4-35) request(schema_mod, prompt)
[](#cb4-36) end
[](#cb4-37)
[](#cb4-38) defp call(schema_mod, prompt, err) do
[](#cb4-39) request(
[](#cb4-40) schema_mod,
[](#cb4-41) prompt <> "\nFix: " <> err
[](#cb4-42) )
[](#cb4-43) end
[](#cb4-44)
[](#cb4-45) defp request(schema_mod, prompt) do
[](#cb4-46) {:ok, %{body: body}} =
[](#cb4-47) Req.post(
[](#cb4-48) "https://api.anthropic.com/v1/messages",
[](#cb4-49) json: %{
[](#cb4-50) model: "claude-sonnet-4-5",
[](#cb4-51) max_tokens: 1024,
[](#cb4-52) system: "Return only JSON matching "
[](#cb4-53) <> schema_mod.json_hint(),
[](#cb4-54) messages: [
[](#cb4-55) %{role: "user", content: prompt}
[](#cb4-56) ]
[](#cb4-57) },
[](#cb4-58) headers: [
[](#cb4-59) {"x-api-key", api_key()},
[](#cb4-60) {"anthropic-version", "2023-06-01"}
[](#cb4-61) ]
[](#cb4-62) )
[](#cb4-63)
[](#cb4-64) body["content"]
[](#cb4-65) |> Enum.find(&(&1["type"] == "text"))
[](#cb4-66) |> Map.get("text")
[](#cb4-67) end
[](#cb4-68)
[](#cb4-69) defp format_errors(cs) do
[](#cb4-70) Ecto.Changeset.traverse_errors(
[](#cb4-71) cs,
[](#cb4-72) fn {msg, _} -> msg end
[](#cb4-73) )
[](#cb4-74) |> inspect()
[](#cb4-75) end
[](#cb4-76)
[](#cb4-77) defp api_key,
[](#cb4-78) do: System.fetch_env!("ANTHROPIC_API_KEY")
[](#cb4-79)end
```

schema_mod` is any Ecto schema module exposing
`changeset/2` and a `json_hint/0` you write once
(a plain string describing the expected JSON shape — no macro magic, no
library-specific DSL). Everything here is Elixir you already know how to
debug at 2am: `Req.post`, a changeset, a recursive function
with a counter. No community release cadence to track, no
provider-coverage gap to hit, no dependency to audit in the next
security review. The cost is the one you’d expect — you own the
retry-prompt wording, the JSON-mode reliability quirks per provider, and
every edge case the two Instructor libraries have already had community
bug reports fix for them. [Testing this boundary](https://sublimecoding.com/blog/testing-ai-agent-outputs-exunit)
works exactly like testing the agent loop’s LLM call: stub
`Req.Test` in place of the real endpoint and script the
validation-failure-then-success transcript directly.

## The decision table

Option
Last shipped
Providers
Retries on invalid output
New deps
Escape hatch

InstructorEx
2025-02-09 (Hex); repo quiet since 2025-06-07
OpenAI, Anthropic, Groq, Ollama, Gemini, vLLM, llama.cpp
Yes, built-in (`max_retries`)
1 (Ecto already assumed)
Fork or vendor it — no active upstream to lean on

InstructorLite
2026-02-01, active cadence
OpenAI, Anthropic, Gemini, Chat Completions-compatible
Yes, built-in
1 (Ecto already assumed)
Explicitly designed to be read and overridden

ReqLLM `generate_object`
2026-07-06 (repo pushed today)
21 providers / 1,205 models
**No** — coerces, doesn’t re-prompt
0 if ReqLLM’s already your client
You write the outer retry loop yourself

Raw JSON + Ecto + retry
You control it
Whatever you wire up
Yes — you write the loop
0
It *is* the escape hatch

## When to skip all four

**Skip the libraries, write the loop, when:** you have
one provider, one or two schemas, and a team that already lives in Ecto
changesets daily. Fifty lines you own outright beats a dependency for a
problem this small, and it’s the choice [the libraries roundup’s
own rubric](https://sublimecoding.com/blog/elixir-libraries-i-reach-for-2026) points to — a dependency earns its place by doing one
thing you can’t trivially replicate; this is one you can.

**Reach for InstructorLite when:** you want the
re-prompt-on-failure behavior without writing it yourself, you’re
starting fresh, and you’re comfortable being an early adopter of a
library that’s a few months into its second life as a rewrite. Wrap it
behind a boundary module regardless — not because it’s shaky, but
because that’s the discipline that makes swapping it out later free.

**Reach for ReqLLM’s `generate_object`
when:** you’re already using ReqLLM as your provider client for
its actual selling point — one interface across 21 providers, streaming,
and the telemetry events [I’ve
already wired into LiveDashboard and PromEx](https://sublimecoding.com/blog/phoenix-llm-telemetry) — and you’d rather not
add a second HTTP-and-schema library just for the structured-output
slice. Budget the outer retry loop yourself; don’t assume it’s
there.

**Don’t reach for InstructorEx on a new project.** Its
API is genuinely good and its README is still the best-written
quickstart of the four, but a 14-month-old commit and a 40-item
unattended backlog are exactly the maintenance-status red flags [the libraries roundup’s
rubric](https://sublimecoding.com/blog/elixir-libraries-i-reach-for-2026) tells you to screen for before you `mix deps.get`
anything. If you’ve already got it in production, that’s a different,
lower-urgency conversation — wrap it behind a boundary module now, and
let the boundary decide whether InstructorLite or the raw loop replaces
it later, on your schedule instead of an incident’s.

None of this changes the one thing that was already true before any
of these four options existed: the payoff of the BEAM for AI workloads
was never any single library, it’s the runtime underneath all of them —
supervision, cheap concurrency, `:telemetry` as a first-class
primitive instead of a bolted-on SDK. I’ve made [the fuller case for
that](https://sublimecoding.com/blog/why-elixir-for-ai-startup-backend) elsewhere; this post is just the one narrow decision inside
it.

## Where to go from here

Whichever option you pick, the failure mode that actually costs you
in production isn’t “the library is missing a feature” — it’s shipping
structured output with no test coverage on what happens when the model
returns something your schema rejects. [The ExUnit patterns for
agent output](https://sublimecoding.com/blog/testing-ai-agent-outputs-exunit) apply directly here: stub the HTTP boundary with
`Req.Test`, script a validation-failure-then-retry
transcript, and assert on the retry actually happening — not just on the
final struct looking right.

If you’re mid-decision on this for a real product and want a second
set of eyes on which of these fits your actual constraints — team size,
provider lock-in tolerance, how much of your validation logic already
lives in Ecto — [that’s a conversation worth
having](https://sublimecoding.com/consulting).

---

## Metering LLM Usage Per Customer in Phoenix

URL: https://sublimecoding.com/blog/metering-llm-usage-phoenix
Published: 2026-08-13
Tags: Elixir, AI, engineering

>
**TL;DR:** [Wiring
ReqLLM’s telemetry events into LiveDashboard and PromEx](https://sublimecoding.com/blog/phoenix-llm-telemetry) answers
“what is my LLM spend doing right now” — an ops question. This post
answers a different one: “what do I invoice tenant X for last month, and
can I defend that number in a support ticket.” That’s a billing-grade
problem, not a dashboard problem, and it has a different centerpiece: an
append-only Postgres ledger, integer math for every dollar (never
`:float`), tenant attribution baked into the call site
instead of bolted on after, and idempotent writes so a retried job can’t
double-bill a customer. Vendors like Lago, OpenMeter, and Stripe’s
Billing Meters exist for exactly this — they’re the right off-ramp once
rating, tax, and dunning outgrow a table you own, not a reason to avoid
building the ledger in the first place.

## Telemetry tells you what happened; a ledger tells you what to bill

[The telemetry post](https://sublimecoding.com/blog/phoenix-llm-telemetry) covers
ReqLLM’s `[:req_llm, :token_usage]` event in detail — I won’t
re-explain the event shape here beyond the one thing that matters for
what follows: its measurements nest token counts under a
`:tokens` key, with cost fields like `total_cost`
sitting flat at the top level. That event is genuinely enough to build a
live cost dashboard, and if that’s all you need, stop there.

It is not enough to send an invoice. A dashboard tolerates an
undercount — nobody’s paged when yesterday’s total is off by three
cents. An invoice line item is a claim you’re making to a paying
customer, and the bar for “an LLM call happened, I saw the number go by
in LiveDashboard” is nowhere near the bar for “I can show you the row,
the tenant it belongs to, and the exact cents.” That’s the gap this post
closes: the same telemetry event, but written into a structure built to
survive an audit instead of a glance.

This is also the argument I made in [why
telemetry has to exist before you reach for a better model](https://sublimecoding.com/blog/your-ai-product-needs-telemetry-before-better-model) taken one
step further — once you can see usage, the very next question a finance
person or a founder asks is “so what does this cost *per
customer*,” and that question doesn’t get answered by a metric, it
gets answered by a ledger.

## Build vs. buy, up front

Before any schema: don’t build this if you don’t have to. [Stripe’s
Billing Meters](https://docs.stripe.com/billing/subscriptions/usage-based) are a reasonable first move if you already run Stripe
for payments and your pricing is a single metered dimension — you report
usage, Stripe rates and invoices it. [OpenMeter](https://openmeter.io/) and [Lago](https://github.com/getlago/lago) go further:
multi-dimensional pricing, credits, entitlements, a customer-facing
usage dashboard, and — for Lago specifically — payment-provider
independence, since it’s open source and talks to Stripe, Adyen, or
GoCardless behind the same API. All three exist because rating, tax,
dunning, and self-serve plan changes are a real product surface, not a
weekend project, and there is no glory in reinventing Stripe’s invoice
engine badly.

What none of them do for you is the part that’s actually specific to
your product: deciding *what a billable unit is* for an
LLM-backed feature, and getting the raw usage event out of your
application and into a durable record before you hand it to anyone.
That’s the build side, and it’s smaller than it looks — a table, a write
path, and an aggregation query. It also happens to be a small lift
specifically because you’re on this stack: an append-only Postgres table
and a job queue you already run aren’t new infrastructure, they’re the
same durable-by-default primitives [the case for Elixir at an
AI startup](https://sublimecoding.com/blog/why-elixir-for-ai-startup-backend) argues you get close to for free. The decision isn’t
“build or buy,” it’s “build the ledger, and decide later whether a
vendor should own rating and invoicing on top of it.” I’ve walked
through this exact build-vs-buy tradeoff, just for a different function,
in [the vCISO math post](https://sublimecoding.com/blog/vciso-math-for-ai-founders) —
the shape of the reasoning is identical: pay for the parts that are
genuinely someone else’s core competency (tax jurisdictions, dunning
logic, PCI scope), build the parts that are actually your product’s
domain logic (what counts as usage, who it belongs to). If you’re not
sure which side of that line your team is on, that’s the conversation I
have as a [fractional CTO for AI startups](https://sublimecoding.com/consulting) more
often than almost any other.

## The ledger: append-only, integer cents, never `:float`

Here’s the Iron Law this post exists to make concrete: **money
is never a `:float`.** Not “usually,” not “unless it’s
small” — never. A float can’t represent most decimal fractions exactly,
and LLM pricing makes this worse than typical SaaS billing does, because
the per-unit price is fractional by construction: a $3-per-million-token
input rate costs $0.000003 per token, and summing thousands of
float-rounded per-call costs across a billing period compounds an error
that’s invisible on any single row and very visible on an invoice
total.

The fix isn’t `Decimal` here, though `Decimal`
is the right call for currency amounts users type into a form. It’s
smaller than that: store cost in **integer micro-dollars**
— one micro-dollar is $0.000001 — and round to cents exactly once, at
invoice time, not once per event. Anthropic’s current published rate for
Claude Sonnet 4.6 is [$3 per
million input tokens, $15 per million output tokens](https://platform.claude.com/docs/en/about-claude/pricing), which happens
to convert to a clean integer: 3 micro-dollars per input token, 15
micro-dollars per output token. Store the price table in the same unit
as the ledger and every multiplication stays an integer, no rounding
until the very last step.

The migration:

`[](#cb1-1)create table(:usage_events,
[](#cb1-2) primary_key: false) do
[](#cb1-3) add :id, :binary_id, primary_key: true
[](#cb1-4) add :tenant_id, :binary_id, null: false
[](#cb1-5) add :customer_id, :binary_id, null: false
[](#cb1-6) add :provider, :string, null: false
[](#cb1-7) add :model, :string, null: false
[](#cb1-8) add :input_tokens, :integer, null: false
[](#cb1-9) add :output_tokens, :integer, null: false
[](#cb1-10) # Integer micro-dollars. 1 = $0.000001.
[](#cb1-11) # NEVER :float for this column.
[](#cb1-12) add :cost_micros, :integer, null: false
[](#cb1-13) add :request_id, :string, null: false
[](#cb1-14)
[](#cb1-15) timestamps(updated_at: false)
[](#cb1-16)end
[](#cb1-17)
[](#cb1-18)create unique_index(
[](#cb1-19) :usage_events, [:request_id]
[](#cb1-20))
[](#cb1-21)create index(
[](#cb1-22) :usage_events, [:tenant_id, :inserted_at]
[](#cb1-23))
```

usage_events` is append-only on purpose — no
`update`, ever. If a call needs a correction, you insert a
compensating row with a negative `cost_micros`, the same
discipline a real accounting ledger uses, instead of mutating history
that a past invoice already referenced. The `unique_index` on
`request_id` is the idempotency guarantee: populate it from
the `id` ReqLLM already stamps on every response, and a
retried write can’t create a second billable row for one API call.

The schema, with the Iron Law spelled out where a future reader will
actually see it:

`[](#cb2-1)defmodule MyApp.Billing.UsageEvent do
[](#cb2-2) use Ecto.Schema
[](#cb2-3) import Ecto.Changeset
[](#cb2-4)
[](#cb2-5) @primary_key {:id, :binary_id,
[](#cb2-6) autogenerate: true}
[](#cb2-7) schema "usage_events" do
[](#cb2-8) field :tenant_id, Ecto.UUID
[](#cb2-9) field :customer_id, Ecto.UUID
[](#cb2-10) field :provider, :string
[](#cb2-11) field :model, :string
[](#cb2-12) field :input_tokens, :integer
[](#cb2-13) field :output_tokens, :integer
[](#cb2-14) field :cost_micros, :integer
[](#cb2-15) field :request_id, :string
[](#cb2-16)
[](#cb2-17) timestamps(updated_at: false)
[](#cb2-18) end
[](#cb2-19)
[](#cb2-20) @fields ~w(tenant_id customer_id provider
[](#cb2-21) model input_tokens output_tokens
[](#cb2-22) cost_micros request_id)a
[](#cb2-23)
[](#cb2-24) def changeset(event, attrs) do
[](#cb2-25) event
[](#cb2-26) |> cast(attrs, @fields)
[](#cb2-27) |> validate_required(@fields)
[](#cb2-28) |> validate_number(:input_tokens,
[](#cb2-29) greater_than_or_equal_to: 0)
[](#cb2-30) |> validate_number(:output_tokens,
[](#cb2-31) greater_than_or_equal_to: 0)
[](#cb2-32) |> unique_constraint(:request_id)
[](#cb2-33) end
[](#cb2-34)end
```

## Attribution at the call site, not retrofitted onto the event

Here’s the failure mode that actually loses money: a call goes out to
the provider with no tenant attached, and there is no honest way to
reconstruct whose bill it belongs to after the fact. ReqLLM’s
[:req_llm, :token_usage]` event carries
`provider` and `model` in its metadata because
those are things ReqLLM itself knows. It doesn’t and can’t know your
`tenant_id` — that’s your application’s concept, not the
client library’s, and trying to smuggle it in after the fact (global
process state, a `Logger.metadata` read inside a shared
`:telemetry.attach` handler) is fragile exactly where
fragility is most expensive: it works in every test and silently drops
tenant on some request path you didn’t think to check.

The reliable fix is structural, not clever: wrap every LLM call in a
project-owned module whose function signature *requires* a
tenant, and emit your own billing event from inside that wrapper — where
`tenant_id` is already a bound variable, not something you’re
trying to recover later.

`[](#cb3-1)defmodule MyApp.Billing.LLM do
[](#cb3-2) @moduledoc """
[](#cb3-3) The only sanctioned way to call an LLM provider
[](#cb3-4) in this app. tenant_id is a required argument —
[](#cb3-5) there is no code path that reaches ReqLLM without
[](#cb3-6) a tenant already attached.
[](#cb3-7) """
[](#cb3-8)
[](#cb3-9) @event [:my_app, :usage, :billed]
[](#cb3-10)
[](#cb3-11) # cents-per-million rate table, in micro-dollars
[](#cb3-12) # per token — see /blog/phoenix-llm-telemetry for
[](#cb3-13) # the raw ReqLLM event this wraps.
[](#cb3-14) @rates %{
[](#cb3-15) "anthropic:claude-sonnet-4-6" =>
[](#cb3-16) %{input: 3, output: 15}
[](#cb3-17) }
[](#cb3-18)
[](#cb3-19) def generate(tenant_id, customer_id, model,
[](#cb3-20) prompt, opts \\ [])
[](#cb3-21) when is_binary(tenant_id) and
[](#cb3-22) is_binary(customer_id) do
[](#cb3-23) {:ok, response} =
[](#cb3-24) ReqLLM.generate_text(
[](#cb3-25) model, prompt, opts
[](#cb3-26) )
[](#cb3-27)
[](#cb3-28) usage = response.usage
[](#cb3-29) rate = Map.fetch!(@rates, model)
[](#cb3-30)
[](#cb3-31) [provider, _] =
[](#cb3-32) String.split(model, ":", parts: 2)
[](#cb3-33)
[](#cb3-34) cost_micros =
[](#cb3-35) usage.input_tokens * rate.input +
[](#cb3-36) usage.output_tokens * rate.output
[](#cb3-37)
[](#cb3-38) :telemetry.execute(
[](#cb3-39) @event,
[](#cb3-40) %{
[](#cb3-41) input_tokens: usage.input_tokens,
[](#cb3-42) output_tokens: usage.output_tokens,
[](#cb3-43) cost_micros: cost_micros
[](#cb3-44) },
[](#cb3-45) %{
[](#cb3-46) tenant_id: tenant_id,
[](#cb3-47) customer_id: customer_id,
[](#cb3-48) provider: provider,
[](#cb3-49) model: model,
[](#cb3-50) request_id: response.id
[](#cb3-51) }
[](#cb3-52) )
[](#cb3-53)
[](#cb3-54) {:ok, response}
[](#cb3-55) end
[](#cb3-56)end
```

Two things worth noticing. First, this computes
cost_micros` itself from token counts and a price table it
owns, rather than trusting ReqLLM’s `total_cost` field
verbatim — that keeps the number you bill on inside your own integer
math the whole way through, instead of inheriting whatever float
precision the provider’s SDK used upstream. Second, this is exactly the
pattern the Iron Law “wrap third-party library APIs behind a
project-owned module” is for: `ReqLLM.generate_text/3` never
appears anywhere else in the codebase, which means “no tenant, no call”
isn’t a convention anyone can forget — it’s the only function that
exists.

A handler attached once, at boot, turns that event into a ledger
row:

`[](#cb4-1):telemetry.attach(
[](#cb4-2) "usage-ledger",
[](#cb4-3) [:my_app, :usage, :billed],
[](#cb4-4) &MyApp.Billing.Ledger.record/4,
[](#cb4-5) nil
[](#cb4-6))
```

[](#cb5-1)defmodule MyApp.Billing.Ledger do
[](#cb5-2) alias MyApp.Repo
[](#cb5-3) alias MyApp.Billing.UsageEvent
[](#cb5-4)
[](#cb5-5) def record(_event, measurements,
[](#cb5-6) metadata, _config) do
[](#cb5-7) attrs = Map.merge(
[](#cb5-8) metadata,
[](#cb5-9) measurements
[](#cb5-10) )
[](#cb5-11)
[](#cb5-12) %UsageEvent{}
[](#cb5-13) |> UsageEvent.changeset(attrs)
[](#cb5-14) |> Repo.insert(
[](#cb5-15) on_conflict: :nothing,
[](#cb5-16) conflict_target: :request_id
[](#cb5-17) )
[](#cb5-18) end
[](#cb5-19)end
```

on_conflict: :nothing` paired with the
`request_id` unique index means the same event firing twice —
a retried job, a duplicate `:telemetry.execute` from a bug —
writes the row once. That’s the idempotency guarantee at the data layer;
it’s the same discipline [the Oban post](https://sublimecoding.com/blog/oban-durable-ai-agent-runtime) argues for
at the job layer, applied here to a plain insert instead of a queued
job.

## Aggregation: turning rows into an invoice line

Once events are landing, the invoice query is a straight aggregate —
this isn’t the `has_many`-vs-`belongs_to` decision
[the Ecto patterns
post](https://sublimecoding.com/blog/ecto-query-patterns-n-plus-one) covers, it’s the simpler cousin: `GROUP BY` and
`sum/1`, with the rounding done exactly once, after the
sum:

`[](#cb6-1)def invoice_lines(tenant_id, period) do
[](#cb6-2) {start_at, end_at} = period
[](#cb6-3)
[](#cb6-4) from(e in UsageEvent,
[](#cb6-5) where: e.tenant_id == ^tenant_id,
[](#cb6-6) where: e.inserted_at >= ^start_at,
[](#cb6-7) where: e.inserted_at < ^end_at,
[](#cb6-8) group_by: e.customer_id,
[](#cb6-9) select: %{
[](#cb6-10) customer_id: e.customer_id,
[](#cb6-11) total_micros: sum(e.cost_micros),
[](#cb6-12) input_tokens: sum(e.input_tokens),
[](#cb6-13) output_tokens: sum(e.output_tokens)
[](#cb6-14) }
[](#cb6-15) )
[](#cb6-16) |> Repo.all()
[](#cb6-17) |> Enum.map(&to_cents/1)
[](#cb6-18)end
[](#cb6-19)
[](#cb6-20)defp to_cents(%{total_micros: micros} = row) do
[](#cb6-21) # Exactly one rounding step, at the end, not
[](#cb6-22) # once per event.
[](#cb6-23) cents = round(micros / 10_000)
[](#cb6-24) Map.put(row, :total_cents, cents)
[](#cb6-25)end
```

Worth running that arithmetic once, by hand, so the “round once”
claim isn’t just asserted. Three calls for one tenant on Claude Sonnet
4.6 at $3/$15 per million tokens: 15,000 input + 3,200 output tokens on
the first call is 15_000 * 3 + 3_200 * 15 = 93,000`
micro-dollars. A second, larger call at 120,000 input + 40,000 output
tokens is `120_000 * 3 + 40_000 * 15 = 960,000`
micro-dollars. A third, tiny call at 500 input + 50 output tokens is
`500 * 3 + 50 * 15 = 2,250` micro-dollars. Summed:
`93,000 + 960,000 + 2,250 = 1,055,250` micro-dollars, which
is `1,055,250 / 10,000 = 105.525` cents — rounds to
**106 cents, $1.06** on the invoice. Every intermediate
value in that chain is an integer; the only place a fraction ever
appears is the final division, and it appears exactly once.

Round once, at invoice time — not once per event. That’s the
difference between an integer ledger and a float that’s merely usually
right.

## Overage handling: soft-cap warnings and hard-cap enforcement as Oban jobs

Aggregation on demand is fine for generating a monthly invoice. It’s
too slow to be the thing that stops a runaway agent loop from burning a
tenant’s month in an afternoon — for that you want a check that runs
close to real time, and [Oban](https://hexdocs.pm/oban) is
the same durable-job substrate [the agent-runtime post](https://sublimecoding.com/blog/oban-durable-ai-agent-runtime)
argues for, applied to spend instead of agent steps. After every ledger
write, enqueue a cap check — uniqued per tenant so a burst of usage
events doesn’t enqueue a burst of redundant checks:

`[](#cb7-1)defmodule MyApp.Billing.CapCheckWorker do
[](#cb7-2) use Oban.Worker,
[](#cb7-3) queue: :billing,
[](#cb7-4) unique: [
[](#cb7-5) period: 60,
[](#cb7-6) fields: [:worker, :args],
[](#cb7-7) keys: [:tenant_id],
[](#cb7-8) states: [
[](#cb7-9) :available, :scheduled, :executing
[](#cb7-10) ]
[](#cb7-11) ]
[](#cb7-12)
[](#cb7-13) alias MyApp.Billing.{Ledger, Tenants}
[](#cb7-14)
[](#cb7-15) @impl Oban.Worker
[](#cb7-16) def perform(%Oban.Job{
[](#cb7-17) args: %{"tenant_id" => tenant_id}
[](#cb7-18) }) do
[](#cb7-19) tenant = Tenants.get!(tenant_id)
[](#cb7-20) spent = Ledger.period_total_micros(tenant_id)
[](#cb7-21)
[](#cb7-22) cond do
[](#cb7-23) spent >= tenant.hard_cap_micros ->
[](#cb7-24) Tenants.suspend(tenant, spent)
[](#cb7-25)
[](#cb7-26) spent >= tenant.soft_cap_micros and
[](#cb7-27) not tenant.soft_cap_warned ->
[](#cb7-28) Tenants.mark_warned(tenant)
[](#cb7-29) MyApp.Billing.Notifier.soft_cap(tenant)
[](#cb7-30)
[](#cb7-31) true ->
[](#cb7-32) :ok
[](#cb7-33) end
[](#cb7-34) end
[](#cb7-35)end
```

The unique` block does real work here, not just tidiness:
`period: 60` collapses every cap check enqueued for one
tenant within a rolling minute into a single job, which matters because
a busy tenant can generate dozens of usage events a minute and you don’t
need — or want — a cap check to run once per event.
`Tenants.suspend/2` and `Tenants.mark_warned/1`
need to be idempotent in their own right too, the same way the Oban
post’s payment example needed an idempotency key at the tool layer: a
job that reruns because Oban retried it after a timeout must not re-send
the warning email or re-flip an already-suspended tenant into some worse
state. A boolean flag checked before the side effect —
`not tenant.soft_cap_warned` above — is enough; you don’t
need anything fancier than the guard clause already sitting in
`perform/1`.

Enqueue it from the same place the ledger write happens:

`[](#cb8-1)def record(_event, measurements,
[](#cb8-2) metadata, _config) do
[](#cb8-3) attrs = Map.merge(metadata, measurements)
[](#cb8-4)
[](#cb8-5) with {:ok, event} <-
[](#cb8-6) %UsageEvent{}
[](#cb8-7) |> UsageEvent.changeset(attrs)
[](#cb8-8) |> Repo.insert(
[](#cb8-9) on_conflict: :nothing,
[](#cb8-10) conflict_target: :request_id
[](#cb8-11) ) do
[](#cb8-12) %{tenant_id: metadata.tenant_id}
[](#cb8-13) |> MyApp.Billing.CapCheckWorker.new()
[](#cb8-14) |> Oban.insert()
[](#cb8-15)
[](#cb8-16) {:ok, event}
[](#cb8-17) end
[](#cb8-18)end
```

## When to hand off to a vendor

The ledger above is genuinely small — one table, one wrapper module,
one aggregation query, one Oban worker. It’ll carry a startup a long
way: correct invoice totals, tenant attribution that can’t silently
drop, caps that actually stop a runaway bill. What it doesn’t do, and
what you shouldn’t build yourself past a certain point, is the part
that’s a real second product: tax calculation across jurisdictions,
dunning and retry logic on failed payments, a self-serve customer-facing
usage dashboard, multi-currency, proration on mid-cycle plan changes,
revenue recognition reporting for your finance team. That’s the moment
to point your invoice_lines/2` query at [Stripe’s
usage-based billing](https://docs.stripe.com/billing/subscriptions/usage-based), [OpenMeter](https://openmeter.io/),
or [Lago](https://github.com/getlago/lago) instead of writing
an invoicing engine from scratch — you keep the ledger (it’s still the
source of truth and the thing an auditor asks for), and let the vendor
own rating and collection on top of it.

The honest signal that it’s time: when the *billing* problem
starts eating more engineering time per week than the *product*
problem. If you’re not sure where you are on that curve, that
build-vs-buy read is exactly the kind of call I help AI startups make as
a [fractional CTO](https://sublimecoding.com/consulting) — happy to look at what
you’ve got before you sink a quarter into either direction.

---

## Building an LLM Audit Trail That Passes SOC 2

URL: https://sublimecoding.com/blog/llm-audit-trail-soc2
Published: 2026-09-01
Tags: security, Elixir, AI

>
**TL;DR:** Every SOC 2 guide that mentions LLMs stops at
policy language — “log access to AI systems” — and never shows you the
table. Here’s the table. An append-only `llm_audit_events`
schema in Ecto, the exact fields that make an entry defensible in an
auditor’s sample versus decorative, and the field most teams skip that
ends up saving them in the room: the provider’s own request ID. Also
covered honestly: whether you have to log the raw prompt (usually yes,
with a redaction step, not a blanket no), and how long to keep any of it
before “audit trail” becomes “unmanaged PII warehouse.”

## The instrumentation point is the same call, different consumer

[Metering LLM usage per
customer in Phoenix](https://sublimecoding.com/blog/metering-llm-usage-phoenix) covers the append-only ledger you build to bill
a tenant for token usage — integer micro-dollars, idempotent writes
keyed on a provider request ID, `never :float`. This post
instruments the *same* outbound call, but the record it produces
answers a different question and lives under different rules. A billing
ledger exists to defend an invoice line for as long as you keep
customers; it can be thin — tenant, tokens, cost, done. An audit trail
exists to defend a *claim about control*, to a third party you
didn’t choose, on a schedule you don’t control, and it has to survive
scrutiny of who called what, with what input, under whose authority, for
as long as your retention policy says it does. Same
`:telemetry` hook. Different table, different columns,
different retention clock.

There’s a third consumer of that same call worth separating out. [The agent evidence pack
for a security review](https://sublimecoding.com/blog/agent-evidence-pack-security-review) specifies the tool-call record as a
*reviewer-facing artifact* — the field set you hand someone
evaluating your product, and how to generate it so it can’t drift from
the code. This post is the storage side of that record: the schema it
lives in, the retention clocks it runs on, and what an auditor pulls out
of it. Read that one for what to show; read this one for what to
build.

If you haven’t wired the underlying telemetry yet, [the ReqLLM integration
walkthrough](https://sublimecoding.com/blog/phoenix-llm-telemetry) covers the event shapes this post assumes — specifically
`[:req_llm, :request, :stop]`, whose metadata already carries
`request_id`, `provider`, `model`, and
`finish_reason` before you’ve written a line of your own
code.

## What SOC 2 actually asks you to log

The reframe I give founders heading into their first SOC 2 is that
there is no “AI” criterion to satisfy. There are criteria that predate
LLMs by a decade and apply to them without modification, and knowing
which ones is what separates a defensible answer from a guess in the
audit room. [SOC 2 as a
revenue tool rather than a security project](https://sublimecoding.com/blog/soc-2-is-a-revenue-tool-not-a-security-tool) is why you’re in the
room at all; this is the artifact that has to survive once you’re
there.

**CC7.2 (System Operations)** requires the entity to
monitor system components for anomalies and analyze them to determine
whether they’re security events — this is the criterion an LLM call log
most directly satisfies, because a prompt-injection attempt or a call
from a deprovisioned service account *is* the anomaly it’s asking
you to be able to detect. The criterion’s own words are “monitors system
components and the operation of those components for anomalies that are
indicative of malicious acts, natural disasters, and errors” ([AICPA,
TSP Section 100, 2017 Trust Services Criteria](https://www.aicpa-cima.com/resources/download/2017-trust-services-criteria-with-revised-points-of-focus-2022)). **CC6.1 and
CC6.3 (Logical and Physical Access Controls)** govern who’s
authorized to trigger the call in the first place, which is why “calling
user or tenant” is a required column, not an optional one — an audit log
with no actor attached can’t prove access was restricted to authorized
parties. **CC7.3 (System Operations)** picks up where CC7.2
stops: it covers evaluating the security events CC7.2 surfaced to
determine whether they rise to *security incidents* — a failure
to meet your objectives — and taking action if they do. That evaluation
only works if the log entry carries enough context (input, output,
timestamp, actor) to judge against.

None of that is AI-specific. What’s new is the volume and the surface
area: a REST API you called twelve times a day now gets called on every
user keystroke, through an SDK whose retry behavior you don’t fully
control, against a vendor whose model version changes under you. The
criteria didn’t change. The number of events that need to satisfy them
did. Controlling what you send in the first place is the other half of that: [throttling against the provider’s real TPM and RPM ceilings](https://sublimecoding.com/blog/throttling-llm-calls-elixir-429) keeps the 429 from happening at all.

## An append-only Ecto schema for LLM audit events

Same discipline as the billing ledger — no `update`, ever,
corrections are new rows — but a wider column set, because the auditor
isn’t sampling for a dollar amount, they’re sampling for a story: who
did what, to which model, with what result.

`[](#cb1-1)create table(:llm_audit_events,
[](#cb1-2) primary_key: false) do
[](#cb1-3) add :id, :binary_id, primary_key: true
[](#cb1-4) add :occurred_at, :utc_datetime_usec,
[](#cb1-5) null: false
[](#cb1-6) add :actor_id, :binary_id, null: false
[](#cb1-7) add :actor_type, :string, null: false
[](#cb1-8) add :tenant_id, :binary_id, null: false
[](#cb1-9) add :provider, :string, null: false
[](#cb1-10) add :model, :string, null: false
[](#cb1-11) add :provider_request_id, :string,
[](#cb1-12) null: false
[](#cb1-13) add :input_tokens, :integer, null: false
[](#cb1-14) add :output_tokens, :integer, null: false
[](#cb1-15) add :latency_ms, :integer, null: false
[](#cb1-16) add :tool_calls, {:array, :map},
[](#cb1-17) default: []
[](#cb1-18) add :finish_reason, :string
[](#cb1-19) add :prompt_digest, :string, null: false
[](#cb1-20) add :prompt_redacted, :text
[](#cb1-21) add :outcome, :string, null: false
[](#cb1-22)
[](#cb1-23) timestamps(updated_at: false)
[](#cb1-24)end
[](#cb1-25)
[](#cb1-26)create unique_index(
[](#cb1-27) :llm_audit_events,
[](#cb1-28) [:provider, :provider_request_id],
[](#cb1-29) name: :llm_audit_events_provider_request_idx
[](#cb1-30))
[](#cb1-31)create index(
[](#cb1-32) :llm_audit_events,
[](#cb1-33) [:tenant_id, :occurred_at]
[](#cb1-34))
[](#cb1-35)create index(
[](#cb1-36) :llm_audit_events, [:actor_id]
[](#cb1-37))
```

prompt_digest` (a SHA-256 of the raw prompt) always gets
written, even when `prompt_redacted` doesn’t — it lets you
prove *which* prompt produced a given output, and lets you
re-associate a customer complaint with an exact call, without keeping
the raw text around by default. More on that split below.

The Ecto schema, with the invariant a future reader needs written
where they’ll see it:

`[](#cb2-1)defmodule MyApp.Audit.LLMEvent do
[](#cb2-2) use Ecto.Schema
[](#cb2-3) import Ecto.Changeset
[](#cb2-4)
[](#cb2-5) @primary_key {:id, :binary_id,
[](#cb2-6) autogenerate: true}
[](#cb2-7) schema "llm_audit_events" do
[](#cb2-8) field :occurred_at, :utc_datetime_usec
[](#cb2-9) field :actor_id, Ecto.UUID
[](#cb2-10) field :actor_type, :string
[](#cb2-11) field :tenant_id, Ecto.UUID
[](#cb2-12) field :provider, :string
[](#cb2-13) field :model, :string
[](#cb2-14) field :provider_request_id, :string
[](#cb2-15) field :input_tokens, :integer
[](#cb2-16) field :output_tokens, :integer
[](#cb2-17) field :latency_ms, :integer
[](#cb2-18) field :tool_calls, {:array, :map},
[](#cb2-19) default: []
[](#cb2-20) field :finish_reason, :string
[](#cb2-21) field :prompt_digest, :string
[](#cb2-22) field :prompt_redacted, :string
[](#cb2-23) field :outcome, :string
[](#cb2-24)
[](#cb2-25) timestamps(updated_at: false)
[](#cb2-26) end
[](#cb2-27)
[](#cb2-28) # append-only: no update_changeset/2 exists
[](#cb2-29) # on this module. Corrections are new rows
[](#cb2-30) # with :outcome set to "superseded".
[](#cb2-31)
[](#cb2-32) @fields ~w(occurred_at actor_id actor_type
[](#cb2-33) tenant_id provider model
[](#cb2-34) provider_request_id input_tokens
[](#cb2-35) output_tokens latency_ms
[](#cb2-36) tool_calls finish_reason
[](#cb2-37) prompt_digest prompt_redacted
[](#cb2-38) outcome)a
[](#cb2-39)
[](#cb2-40) def changeset(event, attrs) do
[](#cb2-41) event
[](#cb2-42) |> cast(attrs, @fields)
[](#cb2-43) |> validate_required(@fields -- [
[](#cb2-44) :prompt_redacted, :finish_reason
[](#cb2-45) ])
[](#cb2-46) |> unique_constraint(
[](#cb2-47) :provider_request_id,
[](#cb2-48) name: :llm_audit_events_provider_request_idx
[](#cb2-49) )
[](#cb2-50) end
[](#cb2-51)end
```

Writing this from inside the same call wrapper the billing ledger
uses is deliberate, not laziness — one project-owned
MyApp.LLM.call/3` that requires `actor_id` and
`tenant_id` as arguments, emits one billing row and one audit
row from the same `:telemetry` handler, off the same event.
Two consumers, one instrumentation point, no code path that reaches the
provider without both attached.

## Why the provider request ID is the field that saves you

The provider request ID is the join key that lets you prove your log
matches the vendor’s log, which is the difference between an entry an
auditor believes and one they don’t. Every major provider stamps a
request ID on its response — `x-request-id` from OpenAI,
`request-id` from Anthropic, `x-amzn-requestid`
from Bedrock — and ReqLLM already surfaces it in the
`[:req_llm, :request, :stop]` metadata as
`request_id`, so capturing it costs you nothing beyond a
column. One gotcha carried over from that post: `model`
arrives as an `LLMDB.Model` struct rather than a string, so
take `.id` before it reaches the changeset.

Here’s the failure mode it prevents. An auditor doing Type 2 testing
doesn’t take your word for a control’s operating effectiveness across
the review period — they pull a sample of events and check each one has
proof. If their sample includes an LLM call and your row has no way to
independently verify it happened as described, you’re asking them to
trust your application’s own account of itself, which is close to the
definition of an unauditable control. A `provider_request_id`
gives a reviewer something to check that you didn’t author. How strong
that check is depends on the provider. On Bedrock it’s the best case: [model
invocation logging](https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html) writes each call’s `requestId`,
calling principal ARN, and input/output token counts to CloudWatch Logs
or S3 — though it is disabled by default, so turn it on before you need
it rather than after. With OpenAI and Anthropic the path is narrower:
the ID is what their support resolves a disputed call against, and your
token counts still reconcile against the usage they invoice you for.
Either way your row stops being a claim and becomes a claim with a
witness.

A row with a provider request ID is a claim with a witness. A row
without one is just your application’s word for what happened.

It’s also the field that makes incident response tractable under
CC7.2/CC7.3: “this model call misbehaved” becomes “here’s the exact
provider-side request, escalate it to their support with this ID”
instead of “here’s an application log we hope corresponds to
something.”

## Do you have to log the prompt?

Usually yes, but not the raw text by default — you need enough to
reconstruct what happened without warehousing customer PII you have no
retention plan for. The `prompt_digest` /
`prompt_redacted` split is the practical answer: hash every
prompt unconditionally (cheap, reversible only with the original text,
and enough to prove which input produced which output), and only store
redacted or truncated prompt text when your data-handling policy
explicitly allows it for that tenant and that data class.

Redaction in practice means running a PII scrubber (structured field
masking for known formats — emails, SSNs, card numbers — plus a
conservative regex pass) on the prompt before it’s written to
`prompt_redacted`, and treating anything you can’t
confidently classify as “don’t store it, keep the digest.” This is a
judgment call your data classification policy should make explicit, not
one an engineer should default on ad hoc per feature — which is the same
governance gap [running
security at an AI-native company](https://sublimecoding.com/blog/running-security-at-an-ai-native-company-2026) argues costs teams the most when
it’s left implicit.

The trap to avoid is the binary version of this decision: “we log
everything” turns your audit trail into the biggest unencrypted PII
surface in the company, and “we log nothing” turns it into an audit
trail an auditor can’t actually use. The
digest-plus-conditional-redaction split is what lets both things be true
— you can prove exactly which prompt produced a disputed output when
you’re authorized to look, and you’re not carrying raw customer text you
never needed for anything but that one dispute.

## Retention: the tension nobody writes down

Retain audit records only as long as your documented policy says, and
no policy should say “forever” — SOC 2 doesn’t mandate a duration. If
your report includes the Confidentiality or Privacy categories, C1.1 and
P4.2 put you on the hook to define a retention period and dispose on it,
and an auditor will check your documented period against what’s actually
in the table. If you scoped Security only — which most startups do —
nothing forces the question, which is exactly why teams never answer it
and end up with a five-year-old prompt table nobody owns. A common,
defensible split: `prompt_digest`, token counts, actor, and
`provider_request_id` (the audit-value columns) live for the
length of your audit window plus a buffer — 13 months covers a Type 2
period with room to spare — while `prompt_redacted` gets a
shorter clock, often 30–90 days, because raw text is the
highest-liability column in the table and the one with the least ongoing
audit value once the immediate incident-response window has closed.

Build the shorter clock as an actual job, not a policy document
nobody automates:

`[](#cb3-1)defmodule MyApp.Audit.PurgeRedactedPrompts do
[](#cb3-2) use Oban.Worker, queue: :maintenance
[](#cb3-3)
[](#cb3-4) import Ecto.Query
[](#cb3-5)
[](#cb3-6) @retention_days 60
[](#cb3-7)
[](#cb3-8) @impl Oban.Worker
[](#cb3-9) def perform(_job) do
[](#cb3-10) cutoff = DateTime.utc_now()
[](#cb3-11) |> DateTime.add(-@retention_days, :day)
[](#cb3-12)
[](#cb3-13) from(e in MyApp.Audit.LLMEvent,
[](#cb3-14) where: e.occurred_at < ^cutoff,
[](#cb3-15) where: not is_nil(e.prompt_redacted)
[](#cb3-16) )
[](#cb3-17) |> MyApp.Repo.update_all(
[](#cb3-18) set: [prompt_redacted: nil]
[](#cb3-19) )
[](#cb3-20)
[](#cb3-21) :ok
[](#cb3-22) end
[](#cb3-23)end
```

Notice this nulls prompt_redacted` and leaves everything
else in the row — the digest, the actor, the request ID, the token
counts all survive, because those are the columns that carry audit value
across the full retention window. The record isn’t deleted; the
liability is.

This is the one sanctioned mutation of an otherwise append-only
table, and it’s worth writing down as an invariant rather than
rediscovering it during an audit: the purge job may null
`prompt_redacted` and nothing else. No column an auditor
samples is ever rewritten. If you want the database holding that line
instead of code review, a column-level grant or a Postgres rule will do
it.

## What does an auditor actually ask for?

A sample of events across the review period, and for each one: proof
the actor was authorized, proof the call happened as logged, and proof
an anomaly (if any occurred) was evaluated per your documented process.
What makes a sampled row pass or fail comes down to whether each field
is independently checkable or just asserted.

Field
Why it’s captured
What it proves to an auditor

`actor_id` / `actor_type`
Ties every call to an authorized identity, satisfying
CC6.1/CC6.3
Access was restricted to authorized parties, not anonymous or shared
credentials

`provider_request_id`
Vendor-issued, independently verifiable
The call happened as described — not just your application’s
word

`occurred_at` (usec precision)
Correlates with vendor logs, rate-limit windows, and incident
timelines
The sequence of events is reconstructable, not approximate

`tenant_id`
Scopes the call to a customer/data boundary
No cross-tenant data exposure in a multi-tenant system

`prompt_digest`
Hash of the exact input, always written
Which input produced which output, without storing raw text by
default

`finish_reason` / `outcome`
Flags truncation, refusal, or error states
Anomalies were captured, not silently dropped (CC7.2)

`input_tokens` / `output_tokens`
Sizes the interaction
Corroborates cost and volume against provider billing, an
independent cross-check

A row missing `provider_request_id` or
`actor_id` isn’t wrong, exactly — it’s unfalsifiable, and an
auditor’s job is to reject unfalsifiable evidence.

## What makes an entry defensible instead of decorative

The short version: every field a stranger could independently check,
versus every field that only your own application vouches for. A log
statement that says `"LLM call succeeded"` in your Sentry
breadcrumbs is decorative — it’s true only because your code says it’s
true. A row with a provider request ID, an authenticated actor, a token
count that reconciles against the vendor’s own invoice, and a timestamp
precise enough to sequence against a rate-limit event is defensible,
because three of those four fields can be checked against a source you
don’t control.

This is the same distinction [security controls that
assume a team you don’t have](https://sublimecoding.com/blog/security-controls-assume-a-team) makes about controls generally — a
policy document is decorative until something makes it operational, and
an audit trail is the specific place that gap shows up first, because
it’s the artifact the auditor actually opens. If you’re heading into a
SOC 2 process for the first time and want a second set of eyes on which
controls in your stack are load-bearing versus theater before the
auditor tells you, that’s exactly the kind of gap review I do as a [fractional CTO / vCISO for AI startups](https://sublimecoding.com/consulting) — cheaper
to find in a working session than in a finding.

## Read this next

If SOC 2 readiness for an AI product is new territory generally, [surviving
enterprise security reviews as an AI startup](https://sublimecoding.com/blog/enterprise-security-reviews-ai-startups) covers the broader
review a prospect’s security team runs before signing, of which an audit
trail like this one is a single, load-bearing piece.

---

## Wiring LiveView Uploads to Claude Vision

URL: https://sublimecoding.com/blog/liveview-uploads-claude-vision
Published: 2026-08-20
Tags: Elixir, Phoenix, AI

>
**TL;DR:** Every tutorial that wires a file upload to a
vision model teaches the happy path: pick a file, send it, print the
answer. None of them teach the part that actually costs money —
validating the file *before* it burns tokens, not after.
LiveView’s `allow_upload/3` gives you that boundary for free:
type and size limits enforced client-side and re-checked server-side,
before a single byte reaches your API key. This post builds the full
loop — `allow_upload` with real limits,
`consume_uploaded_entries` reading the file off disk, a
page-count gate for PDFs that no vendor doc will tell you to add, both
the image and PDF request shapes for Claude, two distinct loading states
(uploading vs. analyzing), and what each request actually costs in
tokens, cited against the current platform limits.

## The boundary nobody enforces

Search “LiveView file upload” and you’ll land on tutorials from 2021
— `allow_upload`, drag a file, save it to disk, done. Search
“Claude vision API” and you’ll land on Anthropic’s own docs, which are
excellent at showing you the request shape and correctly silent on
everything downstream of it, because the vendor doc has no reason to
care what you spend. Nobody has written the connective tissue: what
happens when you wire the two together and a user drops a 40-page PDF
into a field you built for logo screenshots.

The fix isn’t clever. It’s enforcing limits at the boundary — in
`allow_upload/3`, before the file is even fully uploaded —
instead of checking after you’ve already paid for the model call. Here’s
the config, with every option doing real work:

`[](#cb1-1)def mount(_params, _session, socket) do
[](#cb1-2) socket =
[](#cb1-3) socket
[](#cb1-4) |> assign(:stage, :idle)
[](#cb1-5) |> assign(:analysis, AsyncResult.ok(:idle))
[](#cb1-6) |> allow_upload(:doc,
[](#cb1-7) accept: ~w(.jpg .jpeg .png .webp .pdf),
[](#cb1-8) max_entries: 1,
[](#cb1-9) # Claude's base64 image cap is 10MB, and base64
[](#cb1-10) # inflates the raw bytes by ~33%. Cap the raw
[](#cb1-11) # upload well under that so encoding never pushes
[](#cb1-12) # a "valid" file over the API's real limit.
[](#cb1-13) max_file_size: 7_500_000,
[](#cb1-14) auto_upload: true,
[](#cb1-15) progress: &handle_progress/3
[](#cb1-16) )
[](#cb1-17)
[](#cb1-18) {:ok, socket}
[](#cb1-19)end
```

That comment is the whole point of this post in one place.
max_file_size` isn’t an arbitrary round number — it’s
derived from a real constraint. Claude’s Messages API accepts
base64-encoded images up to 10MB, and base64 encoding inflates raw bytes
by roughly 4/3 ([Anthropic’s
vision limits](https://platform.claude.com/docs/en/build-with-claude/vision#request-limits)). A 9.5MB JPEG passes a naive
`max_file_size: 10_000_000` check and then gets rejected by
the API after upload finishes, chunk cost paid and user staring at an
error. `allow_upload`’s own docs are explicit that
`max_file_size` defaults to 8MB and is enforced as chunks
arrive, not just at selection time ([`Phoenix.LiveView.allow_upload/3`](https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html#allow_upload/3))
— set it to the number that matches your downstream API’s
*actual* ceiling, not LiveView’s default.

Validate at the boundary. The generic tutorials skip this because
vendor docs have no reason to care what you spend.

`accept` does the same job for file type — reject a
`.docx` at the file picker, not inside your Claude wrapper
after it’s already on disk. That’s necessary, but for PDFs it isn’t
sufficient: a 50-page PDF can be well under 7.5MB and still blow through
Claude’s page limits and burn thousands of tokens you didn’t budget for.
That gate has to happen after the file lands but before you call the API
— more on it below.

## Two upload paths, one shared entry point

Claude handles images and PDFs through two different content-block
shapes, but both start the same way in LiveView: a single
`allow_upload` config accepting both extensions, and
`consume_uploaded_entries` reading whichever one landed. The
branch happens once, at the API call, not in the upload plumbing.

The template needs `live_file_input/1` for the picker and
a per-entry progress bar — `entry.progress` updates
automatically as chunks arrive, no polling required:

`[](#cb2-1)<div class="uploader" phx-drop-target={@uploads.doc.ref}>
[](#cb2-2) <.live_file_input upload={@uploads.doc} />
[](#cb2-3)
[](#cb2-4) <div :for={entry <- @uploads.doc.entries}>
[](#cb2-5) <p>{entry.client_name}</p>
[](#cb2-6) <progress value={entry.progress} max="100" />
[](#cb2-7) <span>{entry.progress}%</span>
[](#cb2-8)
[](#cb2-9) <p :for={err <- upload_errors(@uploads.doc, entry)}>
[](#cb2-10) {upload_error_to_string(err)}
[](#cb2-11) </p>
[](#cb2-12) </div>
[](#cb2-13)
[](#cb2-14) <p :for={err <- upload_errors(@uploads.doc)}>
[](#cb2-15) {upload_error_to_string(err)}
[](#cb2-16) </p>
[](#cb2-17)</div>
```

upload_errors/2` (per-entry) and
`upload_errors/1` (upload-wide) surface exactly the failures
`allow_upload`’s limits produce — `:too_large` and
`:not_accepted` are the two you’ll see from the config above
([`upload_errors/2`](https://hexdocs.pm/phoenix_live_view/Phoenix.Component.html#upload_errors/2)).
Render them. A silently-rejected file with no feedback is worse than no
limit at all, because the user just tries again with the same file.

## Consuming the entry: two states, not one

Here’s the detail almost every demo gets wrong: “uploading” and
“analyzing” are different states with different failure modes, and
collapsing them into one spinner hides which half of the pipeline
actually broke. A stalled upload bar means the network is slow. A stuck
“analyzing” spinner means Claude is slow, or down, or the file confused
it. Users — and you, debugging a support ticket — need to know
which.

I’m pairing `auto_upload: true` with a
`progress` callback, which is the pattern LiveView’s own docs
recommend for consuming files the moment they finish rather than waiting
on a form submit ([`allow_upload/3`
progress + auto_upload example](https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html#allow_upload/3)):

`[](#cb3-1)defp handle_progress(:doc, entry, socket) do
[](#cb3-2) if entry.done? do
[](#cb3-3) {:noreply, start_analysis(socket, entry)}
[](#cb3-4) else
[](#cb3-5) {:noreply, socket}
[](#cb3-6) end
[](#cb3-7)end
```

start_analysis/2` is where the upload state ends and the
analysis state begins — the seam where the two loading UIs need to
diverge:

`[](#cb4-1)defp start_analysis(socket, entry) do
[](#cb4-2) {binary, media_type} =
[](#cb4-3) consume_uploaded_entry(socket, entry, fn meta ->
[](#cb4-4) {:ok, {File.read!(meta.path), entry.client_type}}
[](#cb4-5) end)
[](#cb4-6)
[](#cb4-7) case boundary_check(binary, media_type) do
[](#cb4-8) :ok ->
[](#cb4-9) socket
[](#cb4-10) |> assign(:stage, :analyzing)
[](#cb4-11) |> assign(:analysis, AsyncResult.loading())
[](#cb4-12) |> start_async(:analysis, fn ->
[](#cb4-13) MyApp.ClaudeVision.analyze(binary, media_type)
[](#cb4-14) end)
[](#cb4-15)
[](#cb4-16) {:error, reason} ->
[](#cb4-17) socket
[](#cb4-18) |> assign(:stage, :rejected)
[](#cb4-19) |> put_flash(:error, reason)
[](#cb4-20) end
[](#cb4-21)end
```

consume_uploaded_entry/3` hands back whatever your
callback returns and removes the entry from the upload config; LiveView
guarantees it only runs once the entry is fully written to disk ([`consume_uploaded_entry/3`](https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html#consume_uploaded_entry/3)).
Note the callback returns the file’s *bytes*, not
`meta.path` — the temp file is cleaned up once the entry is
consumed, so a path smuggled out of the callback points at nothing by
the time an async task gets around to reading it. The template reads
`@stage` to pick between “Uploading… 42%” and “Analyzing…”
instead of one generic loading blob:

`[](#cb5-1)<%= case @stage do %>
[](#cb5-2) <% :analyzing -> %>
[](#cb5-3) <p class="analyzing">
[](#cb5-4) <span class="cursor">▌</span> Analyzing…
[](#cb5-5) </p>
[](#cb5-6) <% :rejected -> %>
[](#cb5-7) <p class="error">File rejected before analysis.</p>
[](#cb5-8) <% _ -> %>
[](#cb5-9)<% end %>
[](#cb5-10)
[](#cb5-11)<.async_result :let={text} assign={@analysis}>
[](#cb5-12) <:loading>Thinking…</:loading>
[](#cb5-13) <:failed :let={_reason}>
[](#cb5-14) <p>The model couldn't read that file.</p>
[](#cb5-15) </:failed>
[](#cb5-16) <p>{text}</p>
[](#cb5-17)</.async_result>
```

handle_async/3` closes the loop, same
two-terminal-outcomes shape I used for [streaming LLM tokens in
LiveView](https://sublimecoding.com/blog/streaming-llm-tokens-liveview-2026) — a clean `AsyncResult.ok` or a
`AsyncResult.failed`, nothing left hanging:

`[](#cb6-1)def handle_async(:analysis, {:ok, text}, socket) do
[](#cb6-2) {:noreply,
[](#cb6-3) socket
[](#cb6-4) |> assign(:stage, :done)
[](#cb6-5) |> assign(:analysis,
[](#cb6-6) AsyncResult.ok(socket.assigns.analysis, text))}
[](#cb6-7)end
[](#cb6-8)
[](#cb6-9)def handle_async(:analysis, {:exit, reason}, socket) do
[](#cb6-10) {:noreply,
[](#cb6-11) assign(socket, :analysis,
[](#cb6-12) AsyncResult.failed(socket.assigns.analysis, reason))}
[](#cb6-13)end
```

## The page-count gate: the boundary check no vendor doc writes for you

allow_upload`’s limits stop a file that’s the wrong
*type* or the wrong *size*. Neither stops a file that’s
the wrong *shape* — a PDF that’s technically 6MB but has 80 pages
of dense text, which is exactly the kind of file a real user uploads to
a “summarize this contract” feature. Claude’s own PDF limits cap
requests at 600 pages, or 100 pages when the request’s context window is
under 1M tokens — which is the default for most models you’ll actually
be calling ([Anthropic
PDF support, page and size limits](https://platform.claude.com/docs/en/build-with-claude/pdf-support#check-pdf-requirements)). A file under every
`allow_upload` limit can still land well past that.

Getting an exact page count means parsing the PDF’s object structure,
which is more machinery than this feature needs. A byte-scan for page
objects is a cheap, honest approximation — cheap enough to run before
the API call, honest enough to catch the case that matters (an 80-page
file, not an off-by-one on a 99-page one):

`[](#cb7-1)defmodule MyApp.PdfInspect do
[](#cb7-2) # ponytail: byte-scan heuristic, not a real parser.
[](#cb7-3) # Swap for a proper PDF library if you need exact
[](#cb7-4) # counts on malformed or linearized PDFs.
[](#cb7-5) def count_pages(binary) when is_binary(binary) do
[](#cb7-6) ~r{/Type\s*/Page[^s]}
[](#cb7-7) |> Regex.scan(binary)
[](#cb7-8) |> length()
[](#cb7-9) end
[](#cb7-10)end
[](#cb7-11)
[](#cb7-12)defp boundary_check(binary, "application/pdf") do
[](#cb7-13) pages = MyApp.PdfInspect.count_pages(binary)
[](#cb7-14)
[](#cb7-15) if pages <= 100 do
[](#cb7-16) :ok
[](#cb7-17) else
[](#cb7-18) {:error, "PDF has #{pages} pages; 100 max."}
[](#cb7-19) end
[](#cb7-20)end
[](#cb7-21)
[](#cb7-22)defp boundary_check(_binary, _media_type), do: :ok
```

This is the gate the brief for this post exists to make: the check
runs on bytes already in memory (read once, inside the consume callback)
but strictly before start_async` fires the network call. A
rejected 200-page PDF costs one disk read and a regex scan. An accepted
one costs a real API call and every token on every page.

## What the request actually looks like — image and PDF

Both paths funnel through one wrapper, because a LiveView should
never hold an HTTP client directly against a third-party API — same rule
I used for the streaming post’s `MyApp.LLM` module. The only
branch is the content-block `type`: `"image"` for
pictures, `"document"` for PDFs, both carrying the same
`base64` source shape:

`[](#cb8-1)defmodule MyApp.ClaudeVision do
[](#cb8-2) @moduledoc "Boundary around Claude's vision/PDF API."
[](#cb8-3)
[](#cb8-4) @endpoint "https://api.anthropic.com/v1/messages"
[](#cb8-5) @model "claude-sonnet-5"
[](#cb8-6)
[](#cb8-7) def analyze(binary, media_type) do
[](#cb8-8) block_type =
[](#cb8-9) if media_type == "application/pdf",
[](#cb8-10) do: "document",
[](#cb8-11) else: "image"
[](#cb8-12)
[](#cb8-13) data = Base.encode64(binary)
[](#cb8-14)
[](#cb8-15) body = %{
[](#cb8-16) model: @model,
[](#cb8-17) max_tokens: 1024,
[](#cb8-18) messages: [
[](#cb8-19) %{
[](#cb8-20) role: "user",
[](#cb8-21) content: [
[](#cb8-22) %{
[](#cb8-23) type: block_type,
[](#cb8-24) source: %{
[](#cb8-25) type: "base64",
[](#cb8-26) media_type: media_type,
[](#cb8-27) data: data
[](#cb8-28) }
[](#cb8-29) },
[](#cb8-30) %{type: "text", text: prompt_for(block_type)}
[](#cb8-31) ]
[](#cb8-32) }
[](#cb8-33) ]
[](#cb8-34) }
[](#cb8-35)
[](#cb8-36) resp =
[](#cb8-37) Req.post!(@endpoint,
[](#cb8-38) headers: [
[](#cb8-39) {"x-api-key", api_key()},
[](#cb8-40) {"anthropic-version", "2023-06-01"}
[](#cb8-41) ],
[](#cb8-42) json: body,
[](#cb8-43) receive_timeout: 60_000
[](#cb8-44) )
[](#cb8-45)
[](#cb8-46) resp.body["content"]
[](#cb8-47) |> Enum.map_join("", & &1["text"])
[](#cb8-48) end
[](#cb8-49)
[](#cb8-50) defp prompt_for("document"),
[](#cb8-51) do: "Summarize this document's key points."
[](#cb8-52)
[](#cb8-53) defp prompt_for("image"),
[](#cb8-54) do: "Describe what's in this image."
[](#cb8-55)
[](#cb8-56) defp api_key, do: System.fetch_env!("ANTHROPIC_API_KEY")
[](#cb8-57)end
```

That request shape — type: "image"`
vs. `type: "document"`, both under
`source: {type: "base64", media_type: ..., data: ...}` —
comes straight from Anthropic’s Messages API reference for [vision](https://platform.claude.com/docs/en/build-with-claude/vision#base64-encoded-image-example)
and [PDF
support](https://platform.claude.com/docs/en/build-with-claude/pdf-support#option-2-base64-encoded-pdf-document) respectively. Under the hood, a PDF isn’t parsed as
text-only: each page is rendered to an image and paired with its
extracted text, so Claude reasons over the layout, charts, and tables
the same way it reasons over a screenshot ([how
PDF support works](https://platform.claude.com/docs/en/build-with-claude/pdf-support#how-pdf-support-works)). That’s the meaningful difference from a RAG
pipeline: [pgvector-backed
RAG in Phoenix](https://sublimecoding.com/blog/phoenix-rag-pgvector-arcana) chunks and embeds *text*, blind to layout;
this path reasons over the document *visually*, which is what you
want for a form, a scanned contract, or a chart nobody transcribed.

## Inline base64 vs. the Files API

Every example above sends the file inline as base64, which is the
right default for this feature: one upload, one analysis, one request.
Reach for the [Files
API](https://platform.claude.com/docs/en/build-with-claude/files) instead when either of two things is true. First, you’re near
the ceiling — a PDF pushing toward the 32MB total request-size limit, or
an image close to the 10MB base64 cap, where the encoding overhead
itself becomes the constraint. Second, the same file gets referenced
across multiple turns — a chat-style feature where the user asks three
follow-up questions about the document they uploaded once. Anthropic’s
own guidance is explicit here: resending base64 data on every turn of a
multi-turn conversation grows your payload every turn even though the
file never changed; uploading once and referencing a
`file_id` keeps it flat ([Files
API for images](https://platform.claude.com/docs/en/build-with-claude/vision#files-api-image-example)). For the single-shot “drop a file, get an analysis”
flow this post builds, inline base64 is the simpler correct answer —
don’t add the extra round trip until a real multi-turn feature earns
it.

## Streaming the analysis back

Everything above waits for the full `AsyncResult.ok`
before rendering anything — fine for a one- or two-sentence description,
worse for a long document summary where the user is staring at
“Thinking…” for ten seconds with nothing to read. If you want the
response to stream token-by-token instead, the plumbing doesn’t change:
swap the single `AsyncResult.ok` call for Claude’s streaming
response wired into the same `start_async` task via
`send/2` and `handle_info/2`. That’s the exact
pattern — cancellation-safe, backpressure-aware — I built out in full in
[Streaming LLM Tokens
in LiveView, the 2026 Way](https://sublimecoding.com/blog/streaming-llm-tokens-liveview-2026); nothing about the upload, the boundary
check, or the process lifecycle here changes when you add it.

## What this actually costs

The page-count gate above isn’t paranoia — vision requests are priced
in visual tokens, and they add up faster than text. Claude tiles an
image into 28×28-pixel patches, so cost is
`⌈width / 28⌉ × ⌈height / 28⌉` tokens: a 1000×1000px image
runs 1,296 tokens on any tier, and models with high-resolution support
(Sonnet 5 and Opus 4.8 among them) can process up to a 2576px long edge
at roughly 4,784 tokens before downscaling caps it further ([resolution
and token cost](https://platform.claude.com/docs/en/build-with-claude/vision#resolution-and-token-cost)). A single screenshot is cheap. A batch of them adds
up exactly like any other per-token cost.

PDFs cost more per unit, because each page pays twice: roughly
1,500–3,000 text tokens per page for the extracted content, plus the
same image-token cost as a rendered screenshot of that page, since PDF
support is built on the vision pipeline ([PDF
cost estimation](https://platform.claude.com/docs/en/build-with-claude/pdf-support#estimate-your-costs)). A ten-page contract can run 20,000+ tokens before
the model writes a single word back. That’s the real reason the
page-count gate belongs in the request path and not in a monitoring
dashboard you check after the bill arrives — same principle as
validating a prompt won’t blow the context window before you ever open a
socket to the model.

## Where the process boundary matters

One more thing worth naming, because it’s easy to build this feature
and never think about it: every piece of this — the upload, the boundary
check, the Claude call, the async result — lives inside one user’s
LiveView process. If it crashes mid-analysis, no other user’s upload is
affected, and the `Req` call inside `start_async`
dies with it, which means you’re not paying for a request nobody’s
waiting on anymore. That’s not an accident of this design, it’s the
reason to build vision features in Phoenix in the first place — the same
per-request process isolation I laid out in [Elixir’s BEAM Is the Runtime AI
Agents Want](https://sublimecoding.com/blog/elixir-beam-ai-agents). If this single-shot analysis grows into something that
loops — ask a follow-up, call a tool, re-analyze with more context — the
shape to reach for is the same GenServer loop from [Build an AI Agent Loop
in 50 Lines of Elixir](https://sublimecoding.com/blog/build-ai-agent-loop-50-lines-elixir), not a bigger LiveView.

## When not to build it this way

If the files are large, numerous, or need to be processed unattended
— a nightly batch that re-analyzes every document uploaded that day —
don’t run it through a live user-facing socket at all. Use Anthropic’s
[Message
Batches API](https://platform.claude.com/docs/en/build-with-claude/pdf-support#process-document-batches) and an Oban job, and show the user a “processing” state
that updates when the job finishes rather than holding a LiveView
connection open for a long-running batch. And if the upload target is
genuinely enormous — video, multi-gigabyte archives — you’re outside
what any of Claude’s vision or document endpoints are built for; that’s
a different pipeline entirely.

For the common case — a user drops a screenshot or a short PDF and
wants an answer in seconds — this is the whole shape: validate at the
boundary before it costs a token, consume the entry off disk, split the
upload state from the analysis state in the UI, and let the process die
cleanly when either the user or the model gives up.

If you’re building this kind of feature into an early-stage product
and want a second set of eyes on where the cost and security boundaries
actually belong, that’s exactly the kind of conversation a [fractional CTO engagement](https://sublimecoding.com/consulting) starts with.

---

## The Claude Code Resource Bible: 46 Tools Worth Knowing in 2026

URL: https://sublimecoding.com/blog/claude-code-resource-bible
Published: 2026-05-07
Tags: AI tools, agents, developer workflows, mcp

>
**TL;DR:** 46 tools across the Claude Code ecosystem, organized by category (official, directories, MCP servers, skills, multiplexers, agent frameworks, automation), with picks for what's actually worth your time and what to skip. If you read one section, read the Recommended Starter Stack at the bottom.

Claude Code went from a CLI wrapper to a real ecosystem in under a year. The reference MCP server repo has eighty-five thousand stars. There are at least five different "awesome Claude skills" lists, and the smallest one has more entries than I can remember. Half a dozen terminal multiplexers exist purely to manage parallel Claude sessions. The ecosystem is real, and most of what's in it is noise.

This post is a curated map. I've installed, broken, and uninstalled most of these tools myself over the last several months. The goal is to save you the half-day of comparison-shopping that I already spent. Where I have an opinion, I'll give it. Where a tool is famous but mid, I'll say so. Where something genuinely changed how I work, it gets a Sublime pick.

What this post is not: a sponsored list, a roundup of every GitHub repo with "claude" in the name, or a vibes-based ranking. If a tool isn't here, it's either because the URL didn't verify, the project looked abandoned, or I couldn't articulate why anyone should install it. There are also a handful of tools you'd expect to see based on aggregated infographics floating around X (you know the ones) that turned out to be either misnamed or non-existent. I dropped them rather than dressing up a 404.

A note on count. The earlier draft of this list had fifty-five entries. After verification, ten of them either pointed at the wrong repo, didn't exist, or duplicated something now bundled into Claude Code itself. I'm not going to round up. The honest number is forty-six.

## How to use this list

If you're new to Claude Code, start at the Official Anthropic section and don't skip ahead until your CLI is set up. If you've been running Claude Code for a few weeks and want integrations, jump to MCP Servers. If you're already managing multiple agents and the bottleneck is coordination, Agent Frameworks is where you'll spend your time.

A quick taxonomy, because the ecosystem invents new words for old things. **MCP servers** are network-accessible tool providers. They expose a database, an API, or a service over a standard protocol so Claude can call them. **Skills** are scoped instruction packs that ride alongside the system prompt. They're roughly "how to do X well, with the right files and commands." **Plugins** are bundles that may include skills, slash commands, hooks, and MCP wiring at once. **Marketplaces** are install registries for any of the above.

That's the whole conceptual model. If a tool description seems wordier than that, it's marketing.

One more rule of thumb: prefer fewer, sharper integrations over a kitchen-sink install. Each MCP server costs context window and adds another thing to debug when an agent goes sideways. The temptation, especially when reading a list like this, is to install everything that looks interesting. Don't. Pick the three or four that map to systems you actually use today. You can always add more later.

## Official Anthropic

Start here. If you're new, the Anthropic docs and reference servers will save you a week of misinformation collected from screenshots on X.

**Claude Code Docs.** [code.claude.com/docs](https://code.claude.com/docs) is the source of truth for the CLI, slash commands, hooks, settings.json, and the plugin system. When a third-party tutorial contradicts the docs, trust the docs.

**Anthropic Partner Network.** Announced via the [Partner Network post](https://www.anthropic.com/news/claude-partner-network), this is Anthropic's formal program for systems integrators and consultancies. Useful if you're trying to land enterprise work on top of Claude. Less useful day-to-day.

**Claude Certified Architect.** Delivered through the Partner Portal under "Claude Certified Architect, Foundations." There's no standalone public page, so don't expect to enroll like it's AWS Cloud Practitioner. Your company has to be in the partner network.

**Official MCP Servers Repo.** [github.com/modelcontextprotocol/servers](https://github.com/modelcontextprotocol/servers) hosts the seven actively-maintained reference servers (Everything, Fetch, Filesystem, Git, Memory, Sequential Thinking, Time). Older reference servers (Postgres, Slack, GitHub) have been moved to [servers-archived](https://github.com/modelcontextprotocol/servers) and superseded by vendor-maintained versions. Always check the official org first before pulling a fork.

>
**Sublime pick:** The Official MCP Servers Repo. When you're putting MCPs in front of production code, "trustworthy provenance" is the only feature that matters.

## Directories and mega lists

These are bookmark-worthy, not install-worthy. Use them to discover, then go install from the source.

**Everything Claude Code (ecc.tools).** [ecc.tools](https://ecc.tools) is the product face of [affaan-m/everything-claude-code](https://ecc.tools), a community library of skills, agents, and slash commands. The site indexes 181 skills, 47 agents, and 79 commands at the time of writing. Good for browsing; quality varies.

**awesome-claude-code (hesreallyhim).** [github.com/hesreallyhim/awesome-claude-code](https://github.com/hesreallyhim/awesome-claude-code) is the de-facto mega list, with 42.8k stars. It's currently mid-reorganization, so the structure may shift, but the link surface is the broadest you'll find.

**Awesome MCP Servers (punkpeye).** [github.com/punkpeye/awesome-mcp-servers](https://github.com/punkpeye/awesome-mcp-servers) sits at 86.4k stars and is the canonical MCP discovery list. Categorized, well-maintained, and updated weekly.

**50+ Best MCP Addons (claudefa.st).** [claudefa.st/blog/tools/mcp-extensions/best-addons](https://claudefa.st/blog/tools/mcp-extensions/best-addons) is a more editorial take, last updated yesterday relative to publication. Useful when you want a human to have already culled the duds.

>
**Sublime pick:** Awesome MCP Servers (punkpeye). It's the only list I open more than once a month. Bookmark it; don't try to install everything in it.

## MCP servers, the integration layer

This is where Claude Code stops being a chat that writes code and starts being an agent that touches your real systems. Pick servers based on the systems you actually use, not the ones that look cool in a demo.

**Playwright MCP.** [github.com/microsoft/playwright-mcp](https://github.com/microsoft/playwright-mcp) is Microsoft's official browser automation server. 32.1k stars. If you want Claude to open a browser, click around, and verify the result, this is the one. The accessibility-tree mode is faster than screenshot mode for most testing.

`claude mcp add playwright npx @playwright/mcp@latest
```

**Firecrawl MCP.** [github.com/firecrawl/firecrawl-mcp-server](https://github.com/firecrawl/firecrawl-mcp-server) is the official Firecrawl scraping and search server. If you've ever asked Claude to "go read this URL" and watched it hallucinate the contents, Firecrawl fixes that. You'll need a Firecrawl API key.

**Supabase MCP.** [github.com/supabase-community/supabase-mcp](https://github.com/supabase-community/supabase-mcp) is the community-maintained server (note the supabase-community/` namespace, not `supabase/`). Apache 2.0, actively released. Frame this in your head as a community project that Supabase blesses, not a first-party product.

**crystaldba/postgres-mcp.** The original modelcontextprotocol/servers Postgres reference is archived. [github.com/crystaldba/postgres-mcp](https://github.com/crystaldba/postgres-mcp) is the maintained replacement I'd grab today. It exposes schema introspection plus query tools without you having to write a custom server.

**GitHub MCP.** [github.com/github/github-mcp-server](https://github.com/github/github-mcp-server) is the official GitHub server, currently at v1.0.3. PRs, issues, code search, and file operations all in one place. If you're using Claude Code in a real repo, install this immediately.

**Context7.** [github.com/upstash/context7](https://github.com/upstash/context7) is Upstash's live-docs server. 54.7k stars, dozens of releases, and the only sane answer to "Claude is using a deprecated API from 2023." It pulls current library docs at request time. The single biggest unforced-error eliminator I've installed this year.

**Sentry MCP.** [github.com/getsentry/sentry-mcp](https://github.com/getsentry/sentry-mcp) is the official Sentry server, also hosted at [mcp.sentry.dev](https://mcp.sentry.dev). Wire this up and Claude can pull live errors, trace events, and triage issues. For production debugging it's a step-change.

**Dokploy MCP.** [github.com/Dokploy/mcp](https://github.com/Dokploy/mcp) is Dokploy's official package. If you self-host with Dokploy, this gives Claude direct access to your deployment surface.

**Slack MCP.** The official reference at [npmjs.com/package/@modelcontextprotocol/server-slack](https://www.npmjs.com/package/@modelcontextprotocol/server-slack) was archived from the main MCP repo. Slack also runs an official remote server documented at [docs.slack.dev/ai/slack-mcp-server](https://docs.slack.dev/ai/slack-mcp-server). For new projects, use the remote.

**Notion MCP.** [github.com/makenotion/notion-mcp-server](https://github.com/makenotion/notion-mcp-server) is the official local server, currently v2.0.0. Notion is migrating users to a remote MCP, and the local server may eventually sunset. If you're starting fresh, plan for the remote.

**Figma MCP.** Setup instructions live at [help.figma.com](https://help.figma.com/hc/en-us/articles/39888612464151-Claude-Code-and-Figma-Set-up-the-MCP-server). The recommended path is the remote at `mcp.figma.com`, not a local install. For implementing a designer's mockup in code, this is the cleanest handoff I've seen.

**Linear MCP.** [linear.app/docs/mcp](https://linear.app/docs/mcp) documents the official remote MCP at `https://mcp.linear.app/mcp`, OAuth 2.1. There's no public source repo; it's a hosted service. Pair it with the GitHub MCP and "ship the linear ticket" becomes one prompt. Add it like this:

`claude mcp add --transport http linear https://mcp.linear.app/mcp
```

A practical note on layering MCPs: I keep my most-used four (GitHub, Context7, Sentry, Linear) in user-scope ~/.claude.json`, and project-specific ones (Playwright for a frontend repo, Postgres for a service repo) in the project's `.mcp.json`. That way I don't pay the context cost of Playwright on a Go service that has nothing to render.

>
**Sublime pick:** Context7. It kills the stale-docs problem instantly and you don't have to remember to mention it. Once it's installed, Claude reaches for it automatically when a library version matters.

**Honorable mention:** Sentry MCP. The first time Claude pulled a live trace and proposed the fix without me copy-pasting an error, I knew this category was going to swallow my error-investigation workflow.

## Articles and blog posts worth reading

Quick annotations, not deep dives.

[10 CLI Tools for AI Coding (Starmorph)](https://blog.starmorph.com/blog/10-cli-tools-for-ai-coding) by Dylan Boudro. Solid intro to the broader CLI agent space if you're trying to compare Claude Code to Codex, Aider, and friends. Useful if you're picking a primary CLI rather than committing to Claude Code, or if you need to argue with a teammate who's already committed to a different one.

[Best MCP Servers for Devs (Firecrawl)](https://www.firecrawl.dev/blog/best-mcp-servers-for-developers). The 2026 edition. Short, opinionated, sensible picks. Some overlap with this post; their angle is more "what does Firecrawl recommend pairing with Firecrawl." Worth a read for a second opinion on the same canon.

[Claude Code MCP Servers: How to Connect, Configure, and Use Them (Builder.io)](https://www.builder.io/blog/claude-code-mcp-servers). The actual config walkthrough I'd hand a teammate who's never wired up an MCP. Covers `.mcp.json`, scope, transports, and OAuth flows. March 2026, still accurate.

[Agentmaxxing (Vibecoding)](https://vibecoding.app/blog/agentmaxxing). Provocative essay on running many agents in parallel. Read it for the framing, not the prescriptions. The author leans further into autonomy than I do, but the piece is the cleanest articulation I've seen of why the next bottleneck for solo devs is coordination, not capability.

## Skills and extensions

Skills are the muscle memory of Claude Code. A good skill is the difference between Claude flailing and Claude executing a workflow you'd recognize as "how a senior would do it."

**Superpowers (obra).** [github.com/obra/superpowers](https://github.com/obra/superpowers) by Jesse Vincent. v5.1.0, very large following. Brainstorm, plan, and execute slash commands plus twenty-plus battle-tested skills covering TDD, debugging, code review, and more. It works across Claude Code, Codex, Gemini CLI, and OpenCode. If you install nothing else from this section, install this.

**Taste Skill.** [tasteskill.dev](https://www.tasteskill.dev) (source: [github.com/Leonxlnx/taste-skill](https://github.com/Leonxlnx/taste-skill)). An anti-slop frontend skill. The pitch: Claude generates UI that doesn't immediately read as AI-generated. Mileage varies by stack, but the underlying instructions are worth reading even if you don't install the skill verbatim.

**Chrome CDP.** [github.com/pasky/chrome-cdp-skill](https://github.com/pasky/chrome-cdp-skill) wires Claude into a Chrome DevTools Protocol session. Lighter-weight than Playwright when you just need to inspect a running page. Alternatives exist (pengelbrecht's variant, the obra/superpowers-chrome bundle), but pasky's is the one I've reached for.

**Delivery Lifecycle.** [github.com/levnikolaevich/claude-code-skills](https://github.com/levnikolaevich/claude-code-skills) is a full Agile-pipeline plugin suite. Backlog grooming, sprint planning, release notes, the whole choreography. If your team genuinely runs Agile, this maps to that ritual cleanly.

**Codex Settings.** [github.com/fcakyon/claude-codex-settings](https://github.com/fcakyon/claude-codex-settings) is a personal Claude Code plus Codex setup pack. Worth a read even if you don't adopt it: it shows what a thoughtful settings.json plus skills layout looks like.

**CLUI.** [github.com/lcoutodemos/clui-cc](https://github.com/lcoutodemos/clui-cc) is a macOS-only TUI overlay for Claude Code. Niche but lovely if you live in the terminal and want a slightly nicer surface than the raw CLI.

**Clawd Cursor.** [clawdcursor.com](https://clawdcursor.com), v0.8.8. Desktop-control via MCP. It lets Claude move your mouse and read your screen. Powerful and slightly unsettling. Worth experimenting with on a sandbox machine before pointing it at your real one. (Note: don't confuse this with `rullerzhou-afk/clawd-on-desk`, which is an unrelated pixel-art desktop pet.)

>
**Sublime pick:** Superpowers. Install it today, thank me later. The brainstorm/plan/execute trio alone is worth the install. Once you have the muscle memory of `/brainstorm` before any non-trivial change, you won't go back.

A note on installation hygiene for skills: read the `SKILL.md` before you install. A skill is essentially "extra system prompt the agent will follow." Treat it with the same scrutiny you'd give a dependency in production code. If the description is vague or the maintainer doesn't explain the intent, that's a signal to skip.

## Terminal multiplexers

When one Claude session isn't enough. The whole category exists because git worktrees plus tmux is awkward, and several teams independently decided to fix it.

**cmux.** [cmux.com](https://cmux.com) is a native macOS app, Swift/AppKit, launched February 2026. It manages parallel Claude sessions with a real GUI. If you're on a Mac and don't want to live in tmux, this is the most polished option I've used.

**claude-squad.** [github.com/smtg-ai/claude-squad](https://github.com/smtg-ai/claude-squad), v1.0.17. Tmux plus git worktrees, but the rough edges are sanded down. Spin up N agents on N branches, watch them run, merge what you like. I've been running it for several months and it's the tool I open first when I want to fan out work.

**dmux.** [github.com/standardagents/dmux](https://github.com/standardagents/dmux) (also at dmux.ai). Cross-agent: it manages sessions for Claude Code, Codex, OpenCode, Cline, Gemini CLI, and more. Pick this if you intentionally use multiple agent CLIs and want one harness over all of them.

**mux (coder/mux).** [github.com/coder/mux](https://github.com/coder/mux), v0.24.0. From the Coder team, so it leans toward remote dev environments. Active, well-maintained, but unless you're already a Coder shop, the others are easier starts.

### Multiplexer comparison

cmux
claude-squad
tmux (raw)

**Setup time**
60 seconds (download `.dmg`, open)
5 minutes (`brew install`, plus tmux + git worktrees)
0 (already installed)

**Platform**
macOS only
macOS, Linux
macOS, Linux, BSD, anywhere

**Best for**
Mac devs who want a GUI for parallel agents
Terminal-native devs running 3-10 parallel sessions on branches
Engineers who already have a tmux config they love

**Skip if**
You're on Linux or you actually like the terminal
You don't use git worktrees or you hate tmux
You don't already know tmux; the learning curve isn't worth it just for this

>
**Sublime pick:** claude-squad if you live in the terminal, cmux if you're on a Mac and want a real window manager. Don't bother with raw tmux unless you already love tmux.

A workflow tip that pays off across all three: pair the multiplexer with git worktrees, not branches. Each agent gets its own checkout, no `git stash` dance, no fighting over uncommitted changes. claude-squad does this automatically. cmux makes it a one-click affair. Raw tmux requires you to wire it up yourself, which is most of why the dedicated tools exist.

## Agent frameworks

When you're past one agent and need real coordination. Several of these projects use the word "swarm." Some of them mean it; most don't.

**ClawTeam.** [github.com/HKUDS/ClawTeam](https://github.com/HKUDS/ClawTeam) is HKUDS's multi-agent swarm coordinator. Academic provenance, which means the code is more interesting to read than to deploy, but the patterns are educational.

**CLI-Anything.** [github.com/HKUDS/CLI-Anything](https://github.com/HKUDS/CLI-Anything), also HKUDS. The pitch: generate a CLI for any app. As an experiment in agent-as-codegen it's fun. As a daily driver, treat it as alpha.

**Autoresearch.** [github.com/uditgoenka/autoresearch](https://github.com/uditgoenka/autoresearch) implements Karpathy's autonomous-loop pattern as a skill. If you want Claude to grind on an open-ended research task without a human in the loop, this is the cleanest minimal version I've seen.

**Hermes Agent.** [github.com/NousResearch/hermes-agent](https://github.com/NousResearch/hermes-agent) by Nous Research. Integrates with Claude Code via a `/cc` command. If you're already in the Nous ecosystem (or want to be), this is the bridge.

**CopilotKit.** [github.com/CopilotKit/CopilotKit](https://github.com/CopilotKit/CopilotKit), v1.57.0, 30.9k stars. Less of a Claude-Code-specific tool and more an agent UI framework. Useful if you're shipping agent features inside a product, not just driving Claude from your terminal.

**Ruflo.** [github.com/ruvnet/ruflo](https://github.com/ruvnet/ruflo) (formerly Claude Flow, rebranded around January 2026). This is the multi-agent orchestration layer I actually reach for. Persistent memory, swarm coordination, and a deep MCP surface that exposes everything from observability to security audits. I dispatched a Ruflo swarm earlier today through MCP and the agents coordinated cleanly without the prompt-engineering pep talk I'd usually need. It deserves more than a paragraph, which is why we wrote [the Ruflo deep dive](https://sublimecoding.com/blog/ruflo-claude-flow-multi-agent-deep-dive) as a companion to this post.

>
**Sublime pick:** Ruflo. It's the only framework in this list I'd commit to before shipping a project. Read [our Ruflo breakdown](https://sublimecoding.com/blog/ruflo-claude-flow-multi-agent-deep-dive) for the install, the tradeoffs, and the failure modes.

## Automation and infra

Shipping agents to production. The tools here are less glamorous than the framework section but more important if you're running anything past a hobby project.

**n8n-as-code.** [github.com/EtienneLescot/n8n-as-code](https://github.com/EtienneLescot/n8n-as-code) brings n8n's 537 nodes and seven thousand templates under Claude Code as code. If you've been writing automation glue by hand and feel the urge to graduate, this is the bridge.

**Cloak Browser.** [github.com/CloakHQ/CloakBrowser](https://github.com/CloakHQ/CloakBrowser) is a stealth Chromium project (not Claude-Code-specific). It integrates with Claude Code via the separate `overtimepog/CloakMCP` server. Worth knowing about if your agent work runs into bot-detection walls; not worth installing speculatively.

**OpenLogs.** [github.com/charlietlamb/openlogs](https://github.com/charlietlamb/openlogs). I'm including this with a caveat: it's a small, single-contributor project. The shape is right (open observability for AI agents), but evaluate for yourself before betting infrastructure on it. Treat it as a starting point, not a finished platform.

**Vercel Chat SDK.** [github.com/vercel/ai-chatbot](https://github.com/vercel/ai-chatbot), 20.3k stars. The Next.js AI Chatbot template, commonly called the "Chat SDK." Best path I know for shipping a chat-driven AI feature in a real product without writing the harness yourself.

**Self-hosted infra (Happy Engineering).** [happy.engineering](https://happy.engineering) plus [github.com/slopus/happy](https://github.com/slopus/happy). Mobile and web client for Claude Code with a self-hostable relay. If you want to drive Claude Code from your phone without sending traffic through someone else's server, this is the path.

>
**Sublime pick:** Vercel Chat SDK if you're shipping AI features fast. For observability, OpenLogs is promising but I'd run it side-by-side with whatever you already use until you've stress-tested it.

The under-discussed move in this section is Happy Engineering's self-hosted relay. Most agent infrastructure assumes you're driving from a laptop and never need to step away. If you want to dispatch work from your phone (kick off a long task at the gym, check on a stuck agent at dinner), the relay model is the only realistic way to do that without piping all your traffic through someone else's hosted control plane. Whether you self-host Happy or roll your own, it's worth thinking about the architecture before you need it.

## Sublime's recommended starter stack

If you only install five things, install these. They cover the highest-leverage ground for a working developer who's already past "hello world" with Claude Code.

**1. Superpowers (skill bundle).** Brainstorm, plan, execute, plus the canonical TDD and debugging skills.

`claude plugin install obra/superpowers
```

**2. claude-squad (parallel sessions).** Fan out across worktrees without rebuilding tmux from scratch.

brew install smtg-ai/tap/claude-squad
```

**3. Context7 (live docs MCP).** Stop watching Claude write code against last year's API.

claude mcp add context7 -- npx -y @upstash/context7-mcp
```

**4. Sentry MCP (production debugging).** Wire your error feed into the model that's writing the fix.

claude mcp add --transport http sentry https://mcp.sentry.dev/mcp
```

**5. Ruflo (when you're ready to scale).** When one agent isn't enough and you want orchestration with persistent memory rather than a manual claude-squad` round-robin. The full install and configuration walkthrough is in [the Ruflo deep dive](https://sublimecoding.com/blog/ruflo-claude-flow-multi-agent-deep-dive).

`npm install -g ruflo
```

These five cover the long tail. Skills give you workflow muscle memory, claude-squad gives you parallelism, Context7 stops the hallucination of stale APIs, Sentry MCP gives you a feedback loop with production, and Ruflo is what you reach for when the workflow itself needs to become an agent.

## What we left off and why

Honest cuts, in case you saw a tool elsewhere and wondered why it's not here:

The original outline included a generic "PostgreSQL MCP" entry. The original reference server is archived. I swapped in crystaldba/postgres-mcp` instead because at least one Postgres entry deserves to be in the bible, and that's the maintained one I'd actually point a friend at.

A handful of "awesome" lists got cut because they share a name with four or five other repos. If a list isn't unambiguously the canonical thing, including it just creates more discovery work for you.

A few agent frameworks were dropped for not having a verifiable canonical URL. If I can't link to the project root with confidence, I can't recommend installing it.

"Loop" appeared in the original outline as a third-party skill. The `/loop` command is now bundled in Claude Code itself, documented at [code.claude.com/docs/en/scheduled-tasks](https://code.claude.com/docs/en/scheduled-tasks). You already have it.

A bundle of `nicholasoxford/*` MCP servers were referenced in source material I was working from. They don't exist. The actual Anthropic-maintained reference servers are at [modelcontextprotocol/servers](https://github.com/modelcontextprotocol/servers). When you see an MCP install command from a screenshot, check the org before you run it.

## What's coming next

This is Post 1. Post 2 is the long-form companion: [the Ruflo deep dive](https://sublimecoding.com/blog/ruflo-claude-flow-multi-agent-deep-dive), where we go past the recommendation and into the install, the architecture, the orchestration model, and the specific failure modes I've hit running it on real work.

Sublime Coding is freshly pivoted toward practical, opinionated coverage of AI coding tools and developer workflows. If that's the kind of thing you want in your inbox without filler, the newsletter signup is below. We don't run sponsored picks.

If a tool I've missed deserves a spot in the next revision of this list, drop it in the comments with a one-sentence pitch for why. Especially interested in things you've actually run for more than a week.

## Read this next

- [**How I Prompt Claude as a Staff Engineer (50 Prompts I Actually Use)**](https://sublimecoding.com/blog/staff-engineer-claude-prompts) — The companion piece: the prompts that put these 46 tools to work.

- [**Prompt Skills, Not Claude: Four Rules from Anthropic's Engineers**](https://sublimecoding.com/blog/prompt-skills-not-claude) — The skill-authoring rules behind the best entries in this list.

---

## The Claude Code Plugin Stack I Actually Run

URL: https://sublimecoding.com/blog/claude-code-plugin-stack
Published: 2026-07-06
Tags: AI tools, productivity, engineering

>
**TL;DR:** Plugins bundle skills, hooks, and MCP configs
into one install, and it’s tempting to grab whatever looks useful. I
don’t. A plugin earns a slot in my stack only if it encodes a discipline
I’d otherwise have to reconstruct from memory every session, or a piece
of domain expertise I don’t personally hold. Almost all of that value
lives in one category — process and workflow — and it’s the category
most people skip in favor of flashier domain-specific tooling. The
domain stuff is real too, but only on the projects that actually need
it; I don’t run Elixir-specific plugins against a Python repo just
because they’re installed globally. Below is the actual stack, organized
by why each one is there.

## The question that decides if a plugin earns a slot

I run a lot of different codebases in a given month — my own
projects, client engagements where I’m the fractional CTO or vCISO
walking into someone else’s repo, greenfield prototypes. Every one of
those sessions starts the same way: an agent with no memory of
yesterday, reading whatever context I hand it. Plugins are how I hand it
more than a prompt.

But “more context” isn’t automatically good. A plugin that just wraps
a prompt template I could type myself isn’t worth a permanent install —
it’s clutter that shows up in every session whether the project needs it
or not. The bar I actually use: **does this encode a repeatable
discipline or real domain expertise I’d otherwise have to reconstruct
from scratch, every single time?** If yes, it’s earned a slot. If
it’s just “a thing that exists in the marketplace,” it’s not.

That filter sorts my stack into three very unequal piles. The first
is small in plugin count and enormous in leverage. The other two are
real, but scoped tight.

## Process plugins: where almost all the value lives

These don’t teach the agent anything about a tech stack. They change
*how* the agent and I work together, which means they pay off on
every project, every day, regardless of language or domain. If I could
only keep one category, it’s this one.

**Superpowers’ brainstorming-first workflow.** The
single highest-leverage habit I’ve adopted is refusing to let an agent
write code before it’s asked me clarifying questions. Left alone, a
coding agent will take an ambiguous ask and confidently build
*something* — fast, plausible, and frequently not what I meant. A
brainstorm-first discipline forces a short back-and-forth before the
first line of code: what’s actually broken, who hits this, what does
“done” look like. It costs two minutes. Skipping it costs an afternoon
of reviewing code against the wrong spec. I’ve written before about [prompting skills instead of
prompting the model one sentence at a time](https://sublimecoding.com/blog/prompt-skills-not-claude) — this is the clearest
example of a skill that’s worth more than any prompt, because the
discipline only works if it fires *every* time, unprompted, and a
plugin is how you make “every time” actually happen instead of “when I
remember to ask for it.”

**Superpowers’ systematic-debugging discipline.** The
default failure mode for an agent staring at a bug is the same as the
default failure mode for a junior engineer: jump straight to the first
plausible fix, apply it, declare victory. That’s a symptom patch, not a
root cause fix, and it’s how the same bug comes back three commits later
wearing a different hat. A debugging plugin worth keeping forces the
boring sequence — reproduce, isolate, form a hypothesis, test the
hypothesis, *then* fix — before any edit gets made. I’ve
catalogued the specific bug patterns this catches in [the AI coding agent bugs
I catch every week](https://sublimecoding.com/blog/things-ai-coding-agents-get-wrong); most of them are exactly this shortcut, taken
with confidence.

**The `code-review` skill, run at adjustable
effort.** Not every diff deserves the same scrutiny. A one-line
config change and a new auth boundary should not get the same review
budget, and an agent that reviews everything at maximum depth is as
useless as one that reviews everything at a skim — the first wastes your
afternoon, the second misses the thing that matters. The review plugins
I keep let me dial the pass: quick sanity check on a small diff, full
adversarial read on anything touching money, auth, or a data boundary.
That’s the same judgment I apply running an actual PR review, encoded so
it doesn’t depend on me remembering to ask for it that day.

**Ponytail’s lazy-engineer discipline.** This is the one
that saves the most wall-clock time and gets skipped the most, because
it doesn’t feel like it’s “doing” anything — it’s a plugin whose entire
job is to argue *against* writing code. Reuse what’s already in
the codebase before adding a new file. Reach for the standard library
before a dependency. Treat YAGNI as a reflex, not a suggestion, and push
back out loud when a task is drifting toward an interface with one
implementation or a config knob for a value that never changes. Left
alone, agents over-build — they’ll happily scaffold an abstraction layer
for a feature that will only ever have one caller. The instinct to ship
the smallest working slice and question the rest in the same breath is
the same one I’ve written about for [shipping
complex features as vertical slices](https://sublimecoding.com/blog/smallest-possible-slice-shipping-complex-features); a lazy-engineer plugin is that
instinct running inside the agent instead of just inside my own
head.

**Caveman’s terse-communication style.** Smaller win,
but a real one: a plugin that strips the filler out of an agent’s
responses — no recap of code it just showed you, no essay defending a
simplification that’s already obvious, no “great question!” padding. It
doesn’t change what gets built, but it changes how fast I can read the
output and decide what to do next, and over a full day of agent sessions
that adds up to real time back.

**The `claude-md-management` plugin.** A
project’s CLAUDE.md is executable tribal knowledge — the operational
realities, the gotchas, the “don’t do X, here’s why” that a new session
needs and a stale file quietly stops providing. Left unmaintained, it
rots the same way any onboarding doc rots: it drifts from what’s
actually true, and a new session inherits confidently wrong context. A
plugin that periodically checks the file against the current codebase
and flags what’s stale is cheap insurance against that drift. I’ve made
the underlying case for treating the file this way in [your CLAUDE.md is the onboarding
doc you never wrote](https://sublimecoding.com/blog/claude-md-onboarding-doc).

Every plugin costs something even when it’s not doing anything —
context the agent reads, a config surface that can drift, one more thing
to explain to the next person.

## Domain plugins: deep, but scoped to the project that needs them

The second pile is domain-specific — plugins that encode years of
framework or vertical idiom I don’t want to reconstruct from a blog post
every time I touch that stack. These are real, and they’re deep, but the
discipline here is different: install them *per project*, not
globally. Dragging Elixir-specific tooling into a Python repo doesn’t
help — at best it’s inert weight, at worst it’s an agent trying to apply
the wrong idiom to the wrong runtime.

**The `elixir-phoenix` plugin.** On the
client engagements and personal projects running on the BEAM, this is
the deepest single install I keep. Ash resource patterns, LiveView
idioms (streams versus assigns, async mounts, the
PubSub-subscribe-after-`connected?/1` rule that trips up
every agent that hasn’t seen it), Oban worker conventions, and an N+1
detector that actually understands the difference between a
`has_many` that needs a separate query and a
`belongs_to` that needs a join. I’ve written about a chunk of
this idiom directly — [twelve LiveView
patterns that hold up in production](https://sublimecoding.com/blog/phoenix-liveview-patterns-production) and [running an agent loop on Oban
as a durable runtime](https://sublimecoding.com/blog/oban-durable-ai-agent-runtime) are both things a generic coding agent will get
subtly wrong without this kind of scaffolding, because the failure modes
are BEAM-specific and nowhere near as intuitive as they look from a
Rails or Node background.

**The `claude-seo` plugin.** Technical SEO,
schema markup correctness, content-quality checks, and — new enough that
most people haven’t built a habit around it yet — readability for
AI-search and generative engine results, not just the classic ten blue
links. I keep this scoped to content and marketing-site work
specifically; it has nothing useful to say about a backend service repo,
and installing it everywhere just adds noise to sessions where it
doesn’t apply.

**The `marketing-skills` plugin.**
Copywriting, positioning, launch-plan structure. I reach for this maybe
one week in eight, when the task in front of me is genuinely
go-to-market rather than code — and that’s exactly why it stays out of
the default stack. It’s real expertise I don’t want to reconstruct from
scratch when I need it, but it has no business being active during an
engineering session where it’ll happily suggest a rewrite of a docstring
in marketing voice.

The pattern across all three: depth is the point, and scope is the
discipline. A domain plugin that’s genuinely good encodes something
you’d otherwise have to relearn or reverse-engineer — that’s worth the
install. It’s also worth *uninstalling*, or at minimum not
enabling, the moment you’re not on that kind of project. The failure
mode isn’t picking a bad domain plugin; it’s letting a good one bleed
context into work it wasn’t built for.

## Utility plugins: small, single-purpose, no ceremony

The third pile is the smallest per-item but still earns its keep —
narrow tools that solve one recurring annoyance and then get out of the
way.

**A design-taste plugin like
`design-taste-frontend`.** Left to its own aesthetic
judgment, a coding agent defaults to the same handful of generic
patterns — the same gradient, the same card-with-shadow, the same
spacing scale every AI-assisted landing page seems to land on. A
design-taste plugin that actively blocks those defaults and pushes
toward something with an actual point of view is worth keeping
specifically because the failure mode it prevents is invisible until a
real person looks at the page and it feels like every other AI-built
site.

**The `dataviz` skill.** Similar idea,
narrower scope: default chart output from an agent is functional and
forgettable. A skill that encodes actual data-viz taste — when a bar
chart beats a line chart, when to just use a table instead of forcing a
visualization the data doesn’t support — saves a full review-and-redo
cycle on anything involving a dashboard or report.

**The `commit-commands` plugin.** Not
glamorous, but they save real minutes: consistent commit message
formatting, a PR description scaffold that pulls the actual diff summary
instead of a placeholder, review-comment triage. This is the kind of
thing that’s genuinely one line of value per use, which is exactly why
it should be a saved command rather than something I re-explain every
time.

**The `update-config` skill.** Slightly meta
— a plugin for configuring the harness itself. Useful specifically
because hook and permission configuration is exactly the kind of thing
you set up once, forget the syntax for, and need again three months
later on a different project.

**The `keybindings-help` skill.** Small,
personal, and entirely about shaving friction off the parts of the
workflow that happen hundreds of times a day. Not a discipline, not
domain expertise — just ergonomics, and I don’t pretend it’s more than
that.

**The `obsidian` plugin.** An Obsidian-style
link into a personal knowledge base, so an agent can pull in and write
back durable notes instead of every insight evaporating at the end of
the session. This is the same problem CLAUDE.md solves at the project
level, one layer up — a persistent memory an agent actually reads
instead of a wiki that only humans open. I went deep on the architecture
for doing this well in [building an LLM wiki your
coding agent actually reads](https://sublimecoding.com/blog/build-an-llm-wiki-for-your-agent).

## What doesn’t make the cut

If you want the full field before you start cutting, I catalogued [46 tools worth knowing](https://sublimecoding.com/blog/claude-code-resource-bible)
elsewhere — that post is the breadth survey; this one is the subset that
actually survived the filter above and stayed installed. Notice what’s
not up there: nothing installed because it trended, nothing that
duplicates a discipline a different plugin already covers, nothing
domain-specific running outside the project it was built for. The
failure mode I see most often — in my own stack early on, and in
engineers I’ve worked with since — isn’t under-installing. It’s the
opposite: a plugin folder that’s grown by accretion, full of things that
seemed useful in the moment and now just add noise and context bloat to
every session, including the ones where they don’t apply. Even an unused
plugin isn’t free — it’s tokens spent on context nobody asked for, one
more surface that can quietly drift out of date, one more thing to
explain when a new teammate asks why it’s there.

The process category is where I’d tell anyone starting from zero to
spend their first real effort — a brainstorm-first habit and a real
debugging discipline change the shape of every session you’ll ever run,
on any stack. The domain plugins are worth going deep on, but only
exactly as deep as the projects you’re actually running, and only on
those projects. Everything else is quality-of-life, worth having, not
worth confusing for either of the two things above it.

If you’re standing up this kind of workflow across a team rather than
just for yourself — deciding which disciplines get codified, which stay
person-specific, and where the line is between a real productivity
investment and tool sprawl — that’s a conversation I have often in [fractional engineering leadership
engagements](https://sublimecoding.com/consulting).

---

## AIOS: An AI Operating System in Plain Markdown

URL: https://sublimecoding.com/blog/aios-ai-operating-system-markdown
Published: 2026-07-09
Tags: AI tools, agents, productivity

>
**TL;DR:** I open-sourced the vault I actually run — [AIOS](https://github.com/sublimecoder/aios), MIT-licensed,
plain markdown and shell scripts, no database, no lock-in. Clone it, run
`/aios-bootstrap`, answer questions for ten minutes, and you
have a vault your coding agent reads at the start of every session and
writes back to at the end. Wire a repo to it with one command and the
loop closes: code → digest → nightly ingest → brain → next session
starts smarter than the last one. It works with Claude Code today
because that’s what I use; the canonical behavior is markdown, so
swapping the tool later doesn’t cost you the vault.

## Why I open-sourced it

I’ve written the theory here before. [Building an AI memory that
outlives any single agent](https://sublimecoding.com/blog/ai-memory-outlives-any-agent) made the portability argument: your
accumulated context is worthless the day you switch tools if it only
lives in one vendor’s chat history or config format. [Build an LLM wiki your
coding agent actually reads](https://sublimecoding.com/blog/build-an-llm-wiki-for-your-agent) laid out the architecture — three layers
(immutable sources, model-written synthesis, a read-first index), three
operations (ingest, query, lint), two rules that stop it rotting. Both
posts describe a shape. Neither one hands you the thing.

AIOS is the thing. It’s the actual vault I run across every repo I
touch, templated so you can clone it and have your own running in about
fifteen minutes. The v2 rewrite went further and [made scope the first path segment](https://sublimecoding.com/blog/aios-v2-scope-is-the-filesystem), so the write guard became prefix validation instead of four coordinated config edits.

The design bet underneath it is an inversion of how most people think
about agent memory. The usual fixes — a bigger context window, a vector
store, a `CLAUDE.md` that grows until nobody reads it — all
treat memory as a *retrieval* problem: throw everything in,
search it at query time. AIOS treats it as an **editorial**
problem instead. Sessions produce raw signal — commits, diffs,
decisions. A periodic ingest pass compounds that signal into durable,
hand-shaped project notes. A read-first index — the Knowledge Map —
points at them, one line each, cheap enough to load whole.

Why that editorial pass beats a bigger context window or a vector
store is the whole case the wiki post makes — AIOS is that architecture
shipped, not re-argued here.

## Setup: clone it, re-init git, and let it interview you

The whole quickstart:

`[](#cb1-1)git clone https://github.com/sublimecoder/aios ~/code/aios
[](#cb1-2)cd ~/code/aios
[](#cb1-3)rm -rf .git && git init # fresh history — see below
[](#cb1-4)claude
```

The rm -rf .git && git init` step isn’t cosmetic.
You’re about to fill this repo with identity files, project history, and
decisions you made under deadline pressure — the exact things you don’t
want sitting in a public template’s commit graph, discoverable by anyone
who forks upstream before you scrub it. Re-initializing gives you a
clean history from commit one, and git history is forever in the
direction that matters: a later fix can’t un-share something an earlier
commit exposed. If any part of what you put in this vault is private,
the repo needs to be private too — that’s a decision to make before the
first commit, not after.

Once you’re in a Claude Code session inside the fresh clone, run:

`/aios-bootstrap
```

That command just reads BOOTSTRAP.md` and executes it as
a prompt — the whole bootstrap process is itself a markdown file, which
is the pattern repeating at the meta level. It interviews you for about
ten minutes, mostly you answering questions, and writes the vault around
your actual answers instead of plausible-sounding defaults. A few things
it’s strict about:

- **Interview first, write second.** It won’t create a
file until it has real answers, asked in small batches instead of one
long form.

- **Never invents biography.** If it doesn’t know
something about you, it asks. An identity file full of
plausible-but-wrong facts is worse than an empty one.

- **Default to less.** One scope, no wall, no optional
subsystems, unless your answers actually demand more.

The first real question is about **scopes** — a scope is
a bucket of context: a job, a side project, a client. Its whole job is
routing, keeping one project’s accumulated memory out of an unrelated
one and telling the agent which identity file governs the current
session. And here’s the honest part most tools with this feature won’t
tell you: **one scope is the right answer for most
people.** The bootstrap won’t talk you into more. If you hesitate
when it asks how many contexts you need, the correct answer is one —
adding a second scope later is a two-line edit to a TSV file, not a
redesign.

## Wiring repos: the loop that makes it compound

A vault that only holds what you type into it by hand is just notes.
The part that makes AIOS actually compound is wiring your code repos
into it, so sessions feed the vault automatically:

`[](#cb3-1)scripts/aios-wire-repo.sh ~/code/acme-api work
[](#cb3-2)scripts/aios-install-nightly.sh # drain queue at 03:00
```

The loop it closes looks like this:

 code session in any repo
 │
 │ SessionStart → aios-context.sh
 │ loads the project's brain into the session
 ▼
 you work
 │
 │ Stop → aios-digest.sh
 │ queues a signal-only digest in the vault
 │ (branch, commits, diff-stat)
 ▼
 /aios-ingest (nightly, or by hand)
 │
 │ synthesizes the queue into durable facts
 ▼
 AIOS/Projects/<scope>/<project>.md + Knowledge Map
 │
 └────► next session starts here, knowing more
```

aios-wire-repo.sh` does exactly three things, all
idempotent, all scoped to the vault and the one repo you point it
at:

- Adds a row to `AIOS/Systems/repo-layers.tsv` — the
manifest.

- Merges two hooks into *that repo’s*
`.claude/settings.json`: `SessionStart` →
`aios-context.sh`, `Stop` →
`aios-digest.sh`.

- Drops a git-ignored `CLAUDE.local.md` in the repo, naming
the scope and pointing back at the vault. That file is a durable pointer
for the contexts the hook doesn’t reach — subagents,
`claude -p` runs, other editors — and it deliberately doesn’t
duplicate the brain content the hook already injects.

Both hooks write **only to the vault**, never to the
invoking repo. And the manifest is never guessed: a repo you haven’t
wired is a silent no-op — no context loaded, no digest written. That’s a
deliberate design choice over the alternative, which is a tool that
infers your scope from directory names and occasionally infers wrong in
a way you don’t notice until a client’s context leaks into a portfolio
project.

You can verify any of it without touching a real session:

`[](#cb5-1)scripts/aios-wire-repo.sh --check ~/code/acme-api
```

This is where CLAUDE.local.md` matters beyond the pointer
function — it’s the piece I already wrote about as [the onboarding doc your agent
reads first](https://sublimecoding.com/blog/claude-md-onboarding-doc). Claude Code [loads memory files in
order and concatenates them](https://code.claude.com/docs/en/memory): policy, then your global
`~/.claude/CLAUDE.md`, then the repo’s own
`CLAUDE.md`, then `CLAUDE.local.md` last. Local
instructions ride alongside the repo’s committed rules instead of
overriding them, which is exactly the property you want — the repo’s own
onboarding doc stays authoritative, and the vault pointer just tells the
agent where the *rest* of what it knows lives.

## The nightly ingest: unattended, and scoped so it can’t leak anything

Digests pile up in a queue and do nothing until they’re ingested.
Schedule that once — launchd on macOS, cron everywhere else, 03:00 local
by default:

`[](#cb6-1)scripts/aios-install-nightly.sh # 03:00 local
[](#cb6-2)scripts/aios-install-nightly.sh --status
[](#cb6-3)scripts/aios-install-nightly.sh --dry-run
```

What actually runs, unattended, every night:
claude -p "/aios-ingest"`, but with a scoped tool allowlist
instead of `--dangerously-skip-permissions`:

`--allowedTools \
 "Read,Write,Edit,Grep,Glob,Bash,Task,TodoWrite"
```

That’s the detail I’d flag if you read nothing else in this section.
Network and browser tools are denied by omission, so an agent running
unattended, on a cron job, at 3am, with nobody watching, has no
exfiltration surface even if something in the queue tried to make it do
something it shouldn’t. It commits the vault when it’s done. It never
pushes.

There’s also a fail-closed guard I’d want in any tool that’s writing
scheduled jobs to my machine: the launchd label and the crontab line are
global to the machine, not per-vault. If you already have a nightly
ingest running for a different AIOS vault, installing a second one would
silently disable the first — so the installer refuses instead, and tells
you what’s already scheduled. You have to override it on purpose
(AIOS_FORCE=1`), not by accident.

Nightly, `/aios-ingest` fans out one subagent per queued
digest so a backlog stays cheap, and compounds everything into
`AIOS/Projects/<scope>/<project>.md` plus the
Knowledge Map. A clean vault with nothing queued logs
`queue empty, skipping` and exits 0 — that’s a pass, not a
failure. The job is a no-op exactly as often as it should be.

## What a week of actually using it looks like

Day to day, there are three commands, and I use them in this order of
frequency:

**`/aios-log <fact>`**, mid-session,
whenever I learn something I’d otherwise re-learn in three weeks. “The
Postgres pool caps at 20; raising it starves pgbouncer” — that kind of
thing. It auto-detects scope and project from the working directory,
appends a dated line to the vault’s log, updates the identity file if
the fact is durable, and commits. It never touches the repo I’m working
in.

**Nothing, most nights**, because the nightly job is
already draining the queue I didn’t manually log. This is the actual
point of the whole system. The discipline that scales is the one that
doesn’t require me to remember to do it.

**`/wiki-lint`**, roughly weekly. A health
check over the vault: schema drift, stale notes, orphaned pages nothing
links to, contradictions between two synthesis notes. It proposes; it
never edits on its own. I read the report and decide what to fix.

Across a week, the effect is that Monday’s session starts already
knowing what Friday’s session figured out — the port conflict I fought
with, the flaky test I diagnosed, the decision I made about which retry
strategy to use and why I rejected the other one. That’s the entire
pitch of the wiki architecture post made concrete: [direct-load,
graph-traverse, then RAG the tail](https://sublimecoding.com/blog/build-an-llm-wiki-for-your-agent), except now it’s a script you run
once instead of a pattern you have to rebuild by hand in every
project.

## The optional wall (most people should skip this)

Some people need more than routing between scopes — two contexts that
can genuinely never appear in the same file. A pen name that can’t be
tied to a legal name. Employer code that can’t reach a public portfolio.
One client’s internals that can’t surface in another client’s repo.

For that, the scope manifest has a `tokens` column. Fill
it in with the identifying strings for a scope — names, domains,
codenames — and two mechanisms activate: a write-time hook that blocks a
save if it drops one scope’s tokens into another scope’s folder, and a
subagent that runs at publish time to catch the crossings a plain grep
can’t, like a codename plus a URL that are only identifying
together.

Leave `tokens` as `-`, which is the default,
and both stay inert. I’d guess most people reading this should leave it
that way. The bootstrap won’t even ask unless you named two or more
scopes in the first question, and it says so plainly when you skip it:
wall left off, turn it on later by filling in one column.

## Verify it before you trust it

Every moving part in AIOS ships with a self-check, and none of them
touch a real vault, a real LaunchAgent, or a real crontab entry:

`[](#cb8-1)bash .claude/hooks/test_vault_write_guard.sh # 21 passed
[](#cb8-2)bash scripts/test_aios_wire_repo.sh # 25 passed
[](#cb8-3)bash scripts/test_aios_install_nightly.sh # 15 passed
```

For the end-to-end loop specifically — confirming a real repo
actually feeds a real vault — the sequence is: wire a repo, work a
session in it, exit, and check that a digest landed in the queue; then
run the nightly job once by hand (--dry-run`) and confirm
the vault’s git log picked up a new commit. If the digest never shows
up, the `Stop` hook didn’t fire or the repo isn’t in the
manifest, and `--check` will tell you which.

## Try it, break it, tell me what’s wrong with it

Everything above is markdown and shell scripts, MIT-licensed, at [github.com/sublimecoder/aios](https://github.com/sublimecoder/aios).
Clone it, run the bootstrap, wire in whatever you’re working on this
week. If something’s confusing, undocumented, or just wrong, open an
issue — I’m running this vault myself daily, so a bad edge case in the
wiring script or a gap in the bootstrap interview is something I’ll hit
too, not just you.

If you’re standing up something like this across a team rather than
for yourself — deciding where the scope boundaries actually need to be,
or where the line is between a real productivity investment and tool
sprawl — that’s the kind of thing I help AI startups think through in [fractional engineering leadership
engagements](https://sublimecoding.com/consulting).

---

## Ruflo (formerly Claude Flow): An Honest Deep Dive on the Multi-Agent Orchestration Platform

URL: https://sublimecoding.com/blog/ruflo-claude-flow-multi-agent-deep-dive
Published: 2026-05-07
Tags: AI tools, agents

>
**TL;DR:** Ruflo (formerly Claude Flow) is a hive-mind orchestration layer for Claude Code and friends. 45,000+ GitHub stars, 700,000+ npm downloads, three queen-types coordinating eight worker-types via Raft, Byzantine, and Gossip consensus, cross-session memory backed by HNSW vector search, multi-provider routing across Claude, GPT, Gemini, Cohere, and Ollama, and Ed25519-signed release manifests verified by `ruflo verify`. It rewards teams hitting the single-agent ceiling. It punishes anyone who installs it on a Tuesday afternoon expecting a drop-in upgrade.

I dispatched a Ruflo swarm earlier today against a Phoenix codebase I work on. Four agents, in parallel: one auditing security, one chasing performance regressions, one finding test gaps, one walking SEO and accessibility. While they worked, I made coffee. By the time I sat back down, the security agent had flagged a CSP report endpoint I'd just wired up the previous evening (it wanted me to harden a specific path), the performance agent had found a query that was hot in `:observer` but I'd been ignoring, and the test-gap agent had drafted half a dozen ExUnit cases against the LiveView I shipped last week.

That's the pitch for multi-agent. Not "agents are smarter." Agents in parallel are a team. Ruflo is the most serious attempt I've used at making that team coordinate instead of just run concurrently.

This post is the honest version: what's real, what's marketing, what hurt to set up, and whether you should bother. If you want the broader landscape (skills, MCP servers, the rest of the agent-tooling ecosystem), that lives in [our Claude Code resource roundup](https://sublimecoding.com/blog/claude-code-resource-bible). This post stays focused on Ruflo.

## The single-agent ceiling

Vanilla Claude Code is exceptional at one thing at a time. The ceiling shows up the moment your task is genuinely fan-out shaped.

Three flavors of "I've outgrown this":

- **Context exhaustion on real codebases.** A 200-file refactor doesn't fit. You end up summarizing, re-summarizing, and re-feeding the agent the parts of the codebase it forgot. Each round drops fidelity.

- **Sequential bottlenecks.** Frontend changes, backend changes, migration scripts, and tests are independent in the dependency graph but the agent does them serially because that's what a single conversation is.

- **No cross-session memory.** Every new session starts from zero. The agent re-learns your project's conventions, your naming, your weird in-house DSL, every single time.

The git-worktrees-plus-tmux pattern (and tools like Claude Squad) solves the parallelization problem. Three terminals, three independent agents, three branches. Beautiful for trivially parallel work. But those agents don't talk. They can't reconcile when their changes overlap. They have no shared notion of "what's the architectural decision we're collectively making here."

That's the orchestration gap. Parallelization is N agents going faster. Coordination is N agents converging on a shared answer. Ruflo is built for the second problem.

## What Ruflo actually is

Ruflo is the rebranded V3 of Claude Flow, the project Reuven Cohen (rUv) and the Agentics Foundation started in mid-2025. The repo at `github.com/ruvnet/ruflo` was created on June 2, 2025, and the first public npm release of `claude-flow@1.0.0` shipped June 10, 2025. The rebrand to Ruflo was announced January 15, 2026 (issue #945) and the user-facing rename shipped under the Ruflo name in v3.5.x in March 2026.

Why rebrand? The honest reason, from rUv's own announcement and the dev.to coverage, is that the project had outgrown its name. "Claude Flow" implied Claude-only. Ruflo routes across Claude, GPT, Gemini, Cohere, and Ollama. The branding was actively misleading. There's a community theory that Anthropic trademark friction forced the move; I haven't seen primary-source evidence for that, so I'm setting it aside.

What you actually install is a TypeScript CLI on top of a Rust engine. The WASM kernel and the `ruvector` HNSW store are Rust. The agent coordination, the MCP server, and the plugin loader are Node. Three install paths exist, all live, all publishing in lockstep at v3.7.0-alpha.11 with v3.6.30 as the current stable. The packages are `claude-flow`, `ruflo`, and `@claude-flow/cli`, and they all resolve.

Numbers as of today (May 7, 2026):

- **45,956 GitHub stars** on `ruvnet/ruflo`, with 5,083 forks.

- **710,854 lifetime npm downloads** of the `claude-flow` package, with 58,803 in the last 30 days.

- **Latest stable: v3.6.30 (May 5, 2026).** The 3.7 alpha line ships almost daily; today's `latest` dist-tag is `3.7.0-alpha.11`.

Stars and downloads aren't quality signals on their own, but the trajectory matters here. The project has roughly doubled its stargazer count since the rebrand and is shipping faster than most projects in the agent-tooling ecosystem.

## The hive-mind architecture

This is where Ruflo earns its keep, and where the design choices look distributed-systems-shaped rather than influencer-shaped.

There are three queen types in the hive-mind skill (`.claude/skills/hive-mind-advanced/SKILL.md`):

- **Strategic queens** plan high-level objectives, choose topology, and assign roles.

- **Tactical queens** drive execution, resolve conflicts between agents, and manage dependencies between tasks.

- **Adaptive queens** monitor the swarm in flight and reconfigure topology if work isn't converging.

Underneath the queens, eight worker types do the actual work: **Researcher, Coder, Analyst, Tester, Architect, Reviewer, Optimizer, Documenter**. Note that there is no "security" worker type; security gets its own dedicated plugins (`ruflo-security-audit` and `ruflo-aidefence`), which is the right call since security review has very different prompt and tool surface than coding.

When agents disagree (and they will), Ruflo runs consensus. The repo documents Raft, Byzantine fault-tolerant, Gossip, plus two more (CRDT and Quorum) for different swarm sizes and trust models. Raft has a real implementation at `v3/@claude-flow/swarm/src/consensus/raft.ts` (not a wrapper around someone else's library). The point isn't that you ever read the consensus log. The point is that a five-agent swarm can vote on which approach to a refactor to take and converge instead of one agent silently winning because it spoke last.

If you've spent any time on real distributed systems, this is recognizable. If you haven't, the load-bearing claim is: there's no single point of hallucination. A hallucinating agent gets outvoted.

Here's the request flow at a level where it's actually useful:

-
01

### User

claude code · cli · ide · cron

-
02

### Ruflo Core

policy · topology · wizard

-
03

cost lever

### Provider Router

Claude · GPT · Gemini · Ollama

cheap-tier first, escalate up

-
04

### Hive-mind Swarm

queens (strategic / tactical / adaptive)

workers (8 types)

consensus (raft / byzantine / …)

-
05a

productivity lever

### Memory Layer

HNSW vectors

knowledge graph

ReasoningBank

-
05b

### LLM call

cached, signed

The two things that matter in that diagram are the router (cheap-first routing is where the cost story lives) and the memory layer (cross-session is where the productivity story lives).

## Memory and learning

This is the part most agent frameworks skip, and it's the reason I keep coming back.

Ruflo's memory layer is HNSW-backed vector storage with sub-millisecond retrieval in the project's own micro-benchmarks (the `~61µs` figure from the RuVector/Postgres bridge in issue #963 is the cleanest number; it's a synthetic bench, not an end-to-end SLO). Treat it as "fast enough that retrieval isn't the bottleneck," not as a guarantee.

Layered on top:

- **Knowledge graph.** Entities and relationships extracted from your codebase as agents work. Useful when an agent asks "what touches this module" and the answer involves traversal, not search.

- **ReasoningBank.** Successful solution patterns get stored. The next time a similar task shows up, the strategic queen can prime workers with the prior pattern instead of starting blank.

- **SONA learning loop.** Performance signals route future tasks to the worker types and provider tiers that have done well on similar work. It's not magical. It's a multi-armed bandit with a memory.

The practical outcome is the agents stop re-learning your project every session. If you've named your background-job module the way you've named it across the codebase, the swarm picks that up after one or two sessions and stops asking. If your test pyramid is shaped a certain way, the testgen worker stops generating top-of-pyramid garbage and starts generating the kind of test you actually merge.

This is the single biggest operational win compared to vanilla Claude Code. Single-agent Claude Code with a good `CLAUDE.md` gets you partway. Cross-session memory gets you the rest.

## Multi-provider routing and the cost story

Ruflo routes across Claude (Sonnet 4.6, Haiku 4.5), GPT, Gemini 2.5 Pro and Flash, Cohere, and Ollama. The router is what makes the cost story possible: simple edits go to a cheap tier, complex reasoning escalates to a flagship model, and a WASM-based local kernel handles trivial transformations without a network round-trip at all. Background workers using local retrieval don't burn your Claude subscription on lookups.

Now the caveat, which the project does not say loudly enough.

The project markets up to **75% API cost savings** versus single-agent flagship-model usage. The mechanism is real. Cheap-tier handling for the long tail of small tasks plus WASM-local for trivial work plus local retrieval for memory is exactly how you'd architect a cost-efficient agent system. But:

- The 75% number is not benchmarked publicly. There is no methodology document, no A/B comparison, no per-task cost report I could find.

- Multi-agent orchestration adds per-agent context and coordination tokens. Naive multi-agent setups can spend *more* than single-agent Claude Code on the same task, because every agent is paying context overhead that a single agent paid once.

- Whether Ruflo nets out positive on cost depends entirely on your workload. A lot of small parallelizable tasks: yes, you'll save money. A small number of long context-heavy tasks: maybe not.

So: the savings are plausible, the architecture is consistent with real savings, and I would not quote 75% to my finance team. I'd run a one-week measurement on my actual workload and decide from there. The project's framing as "up to 75%" is fair. Anyone repeating it as "75%" without context is selling something.

## Security model

Ruflo takes supply chain seriously, which is unusual in this corner of the ecosystem.

The flagship feature is **Ed25519-signed release manifests verified by `ruflo verify`**. Implementation lives at `v3/@claude-flow/cli/src/commands/verify.ts`. The CLI fetches the `verification.md.json` witness manifest, re-derives the Ed25519 public key from the manifest's git commit, and verifies the signature. If you've ever wondered whether the npm package you just installed corresponds to the git commit it claims to, this answers the question. Most agent frameworks ship nothing comparable.

Beyond signing, the security plugins (`ruflo-security-audit`, `ruflo-aidefence`) cover:

- Prompt injection detection on inbound content (think: README files in dependencies, pasted issue text).

- Path traversal blocking when an agent tries to read or write outside the repo root.

- Command injection guards on shell-using workers.

- PII detection for credential and secret handling.

- Anti-drift swarm config to keep long-running unattended jobs from quietly redefining the goal.

The "anti-drift" piece matters more than it sounds. If you run an overnight swarm and one agent's interpretation of the task slowly mutates over twelve hours of context turnover, you wake up to a different project than the one you went to bed on. The anti-drift config pins the original spec and surfaces deviation as a vote, not a fait accompli.

## Install and first run

Three install paths. All three publish at v3.7.0-alpha.11. All three resolve as written.

`# Option 1: Claude Code plugin (recommended)
/plugin marketplace add ruvnet/ruflo
/plugin install ruflo-core@ruflo
```

# Option 2: MCP server
claude mcp add ruflo -- npx -y @claude-flow/cli@latest
```

# Option 3: CLI direct
npx ruflo@latest init --wizard
```

I went with Option 1 today because I was already inside Claude Code and the plugin install gives you the slash commands and skills without any extra wiring. The wizard in Option 3 is the most opinionated path; it asks you about provider keys, swarm size, and persistence settings up front, and it writes a sensible ruflo.toml`. Option 2 is the right answer if you already have an MCP-aware host and want Ruflo as one tool among many.

A first swarm looks like this. From inside Claude Code with the plugin installed:

`/ruflo swarm-init --workers 4 --topology mesh --task "audit the auth module for security, performance, missing tests, and docs gaps"
```

Then watch it run with:

ruflo watch
```

Or, if you prefer the slash form inside Claude Code, /watch` streams swarm events as they happen. It's the closest thing to `htop` for an agent fleet I've used.

When something goes sideways, and it will eventually, run:

`ruflo doctor --fix
```

The doctor checks MCP connectivity, AgentDB integrity, plugin manifest signatures, and provider key availability. It mostly does the right thing. The first time I ran it, it caught a stale ~/.ruflo` directory from an earlier Claude Flow install that I'd forgotten about and offered to migrate it.

## Real-world use cases

Five shapes where I've seen this earn its setup overhead:

**Full-stack feature build.** Frontend, backend, migration, and tests in parallel, with a strategic queen reconciling the contract between them. The win isn't speed. It's that the API shape, the typescript types, the migration column names, and the test fixtures all stay in sync because there's a coordinator.

**Large-codebase refactor with persistent memory.** Renaming a primitive that touches 400 files isn't hard for any agent in isolation. It's hard because halfway through, the agent forgets why some occurrences were intentional exceptions. With cross-session memory and a knowledge graph, those exceptions get recorded the first time and respected on every subsequent pass.

**Overnight batch jobs in daemon mode.** This is what I dispatched today. Audit, optimize, testgen, and SEO/a11y agents running on a schedule against a Phoenix codebase. The cron-like scheduler in `ruflo-loop-workers` is real and works.

**Bug investigation with competing-hypothesis agents.** Spawn three workers each with a different theory of what's wrong. Vote on evidence. Drop the losing hypotheses. This is what consensus is actually for, and it's surprisingly satisfying when one agent's "it's a race condition in the cache" loses to another's "no, it's a clock skew on the staging worker" because the second agent produced a reproducer.

**Standardizing TDD across a team via SPARC workflow.** SPARC (Specification, Pseudocode, Architecture, Refinement, Completion) is a built-in workflow template. Pin it in `ruflo.toml` and every junior on the team gets the same structured approach without you policing it.

## When to use Ruflo vs the alternatives

Honest table. No faux balance, no "everything is great in its own way."

Dimension
Ruflo
Claude Squad
Vanilla Claude Code

Setup overhead
High (wizard, keys, AgentDB, plugin manifest)
Low (tmux + git worktrees)
Zero

Cross-session memory
Yes (HNSW + knowledge graph + ReasoningBank)
No
Per-session only (with `CLAUDE.md` priming)

Coordination model
Hive-mind with consensus across queens and workers
Independent agents on independent worktrees
Single agent, sequential

Cost story
Multi-provider routing, cheap-tier first, WASM-local for trivial work; savings unbenchmarked
One subscription per agent, no sharing
One subscription, predictable spend

When to reach for it
Coordinated multi-agent work, overnight pipelines, large refactors with memory
Trivially parallel tasks across independent branches
Single-file work, fast iteration, prototyping

Reach for Ruflo when:

- You need agent coordination, not just parallelization.

- Cross-session memory is the bottleneck, not raw speed.

- You're running unattended or overnight workflows and need anti-drift.

- Cost optimization across providers genuinely matters and you're willing to measure it.

- You want a team-wide standard workflow (SPARC, TDD, ADR-driven).

Skip Ruflo and use Claude Squad when:

- You just need three to five agents on independent tasks.

- You want zero config.

- Shared memory isn't the constraint.

Skip both and use vanilla Claude Code when:

- You're working on a single file or a single sequential refactor.

- The setup overhead won't pay off in the time you have.

- You're prototyping and want feedback loop tightness over coordination.

## The downsides

Real downsides, not faux balance. I hit all of these today.

**The 3.7 line is alpha and ships daily.** On the one hand, this is what active development looks like. On the other, `latest` on npm is `3.7.0-alpha.11` as I write this, and that means if you `npx ruflo@latest` you're getting alpha. Pin to `v3.6.30` if you need stable. Most install instructions in the wild don't tell you to do this.

**Documentation is fragmented.** README, wiki, plugin READMEs, GitHub issues, dev.to posts, and the `.claude/skills/*/SKILL.md` files. Each one is internally accurate. Across them, the project's "current state" is a moving target. I had to read three sources to figure out the current canonical install command (it's the marketplace plugin) versus the older paths.

**The learning curve is real.** Queens, workers, consensus, topologies, AgentDB, ReasoningBank, SPARC, hive-mind versus swarm versus loop-workers. None of it is hard once you have the model in your head, but the model isn't a fifteen-minute read. Plan for a couple of hours of orientation before you ship anything load-bearing.

**Setup overhead doesn't pay off on small projects.** If your codebase is 5K lines and you're alone, the wizard is going to ask you questions you don't have answers to. Use vanilla Claude Code or Claude Squad. Come back to Ruflo when you have a team or a 50K-line codebase.

**Community is active but smaller than vanilla Claude Code.** When you Google an error, you'll find recent answers, but you'll find five of them, not five hundred. The Discord is helpful. rUv himself is unusually responsive in GitHub issues. But the long tail of "someone has hit this exact thing before" is shallower.

## The verdict

**Sublime rating: 9 out of 10.**

Not 10, because the alpha cadence and the doc fragmentation make first-day setup harder than it needs to be. The friction is real and you should know that before you start.

But the wins are also real and I want to be specific about them. Today's swarm dispatched four agents against a Phoenix codebase, and the security agent caught something that I would not have caught on my own pass through the auth module today, because I would have been doing one thing at a time and security would have been on tomorrow's list. That's the actual value. Not "AI is faster," but "a coordinated team finds things that a sequential pass doesn't."

Install today if:

- You're a team scaling agent workflows past a single dev.

- You run overnight or unattended pipelines.

- You've hit the single-agent ceiling on context, on parallelism, or on memory.

- Multi-provider cost optimization is something you'd actually measure.

Wait if:

- You're solo on a small project where setup overhead won't pay off.

- You're allergic to alpha software in your toolchain.

- You haven't yet exhausted the vanilla Claude Code workflow.

## Closing

Ruflo is the most serious open-source attempt I've used at making multi-agent coding actually coordinate instead of merely parallelize. The architecture is distributed-systems-shaped, the memory story is real, the security posture is unusually mature for this ecosystem, and the project ships fast. The cost-savings claim deserves skepticism until benchmarked on your workload. The alpha-cadence and doc fragmentation deserve patience.

If you want the broader ecosystem (skills, MCP servers, terminal multiplexers, the rest of the agent-tooling map), [the Resource Bible](https://sublimecoding.com/blog/claude-code-resource-bible) covers it. If you want to follow Ruflo specifically, the repo lives at [github.com/ruvnet/ruflo](https://github.com/ruvnet/ruflo) and rUv announces most things in GitHub issues before he announces them anywhere else.

Are you running multi-agent workflows yet? What's working, what isn't? Reply to the newsletter or open a discussion. I'm collecting real-world swarm patterns and the friction points are at least as interesting as the wins.

---

## Prompt Skills, Not Claude: Four Rules from Anthropic's Engineers

URL: https://sublimecoding.com/blog/prompt-skills-not-claude
Published: 2026-05-26
Tags: AI, agents, AI tools, engineering leadership, productivity

**I sat through the most recent Anthropic engineering talks last week and walked away with an uncomfortable observation. Most of the engineers I work with are still prompting Claude one sentence at a time. The people who actually built Claude Code aren't. They're prompting *skills* — and the difference between those two patterns is the difference between a useful tool and a compounding one.**

Four rules emerged from listening to Barry, Eric, and the rest of the Anthropic engineering team talk about how they actually work. None of them are clever. All of them are the kind of operational discipline that separates engineers who get real leverage from agents from engineers who burn tokens generating the same script every Tuesday morning.

Here's the breakdown — and where my own practice has confirmed each rule, plus the one or two places it needs more nuance than the talks gave it.

## Why prompt-engineering moved from the chat to the folder

The first rule is also the mental shift everything else hangs on: **prompt skills, not Claude**. When you write a prompt in the chat, it vanishes the moment you close the tab. When you write a skill, it persists — and the next time you face a task that fits it, Claude reaches for it automatically.

This sounds obvious until you watch your own behavior. I caught myself last month rewriting the same "draft this in my voice, here's the lede pattern I use, here's the close pattern, here's how 'Read this next' works" preamble three times in one week before I admitted I was being lazy and codified it as a skill. Now I just type `/blog-post draft 1,800 words on AI agent supervision` and Claude reaches for the skill automatically — no preamble, no re-explaining the voice constraints, no token spend on the warm-up.

The framing the Anthropic team uses is a three-layer stack: *models* at the bottom (the AI itself), *agents and prompts* in the middle (how most people interact today), and *skills* at the top (the application layer where leverage actually lives). The takeaway: stop writing custom one-shot prompts every time. Identify the pattern, name it, and turn it into a skill.

## Skills are a contract, not a folder

The Anthropic team describes skills as "folders that package composable procedural knowledge for agents." That's technically correct and also a meaningful undersell. Calling a skill a folder is like calling a function a file. Sure — but it misses the point of why it works.

A skill is a *contract*. You're declaring: this task is stable enough that I'm willing to codify how I want it done. The folder is just the storage mechanism. The contract has three parts, and most engineers only write the first one:

 - **Description.** What Claude reads to decide whether to invoke the skill at all. If your description is vague — "writing helper" — Claude won't reliably reach for it. If it's specific — "draft a code review comment in my voice, focused on security and naming, never approve without reading every line" — Claude will invoke it on the right tasks without you having to call it explicitly.

 - **Instructions.** The step-by-step process Claude follows once the skill is invoked. This is the playbook. Most people stop here.

 - **Tools.** The scripts, API calls, and reference files the skill can use. *This is the layer where leverage lives*, and it's the layer most people skip entirely.

Eric from the Anthropic team made the funniest point in the talks: people put enormous effort into the prompt portion and then write the tools as if it were 1998 — bare, undocumented, parameters named `a` and `b`, no examples. They obsess over the conversation Claude has with itself and ignore the actual hands they're giving it.

My own version of this lesson came when I built `/seo-audit` — the skill that runs a full-site SEO audit, the same one I ran on this site recently. The instructions weren't the hard part; "score the site against E-E-A-T, validate schema, find over-length meta descriptions" is a few paragraphs. The hard part was the tools the skill calls. One that hits the live URL and pulls every `<script type="application/ld+json">` block. One that validates each block against the Schema.org spec. One that diffs the current sitemap against an earlier baseline. One that runs a Playwright capture to surface above-the-fold rendering issues. Without those tools, the skill is a checklist that says "look for problems." With them, it's a system that returns a 0–100 health score per category, a per-page issue list, and a prioritized action plan in under ninety seconds.

## Composability is the move — but you're now maintaining a library

The third rule from Anthropic: **build composable skills, not custom skills**. Small, focused, reusable units that Claude can chain together as needed, rather than one monster skill that does everything.

I learned this the expensive way. My first attempt at codifying my content and growth work was a single sprawling "do all my marketing" skill — blog post drafting, LinkedIn post drafting, SEO audits, competitor research, email sequencing, all inside one folder. One skill, a million possibilities. The moment I tried to update how the skill handled lede formatting for long-form posts, I broke how it handled the LinkedIn opening hooks, and I didn't notice for a week. The Anthropic team's framing is right: composability gives you three things.

 - **Failures are scoped.** When a focused skill breaks, you know exactly which one. When a giant skill misbehaves, you don't even know which of its ten jobs went sideways.

 - **Improvements compound.** Sharpening one small skill upgrades every workflow that calls it. Sharpening a giant skill upgrades nothing reliably, because the change might fix one of its jobs and silently degrade another.

 - **Reuse becomes free.** A focused `/product-research` can plug into a blog-post workflow today (gather competitor positioning before I draft the essay) and a LinkedIn workflow tomorrow (find the three contrarian threads on this topic) without modification.

That last point is the unlock. My current library is four focused skills — `/blog-post`, `/linkedin-post`, `/seo-audit`, and `/product-research` — and they call each other constantly. `/blog-post` invokes `/product-research` before it drafts. `/seo-audit` invokes `/product-research` when it surfaces a missing competitor mention. `/linkedin-post` pulls from the most recent `/blog-post` draft when I'm threading. None of those workflows existed when each skill was a standalone monster.

But here's the honest tradeoff the talks didn't quite name. Once you commit to composability, you're not maintaining a prompt. You're maintaining a *library*. Libraries have versioning concerns, dependency concerns, naming conventions that have to stay coherent across the team, and a curation cost that grows with size. I don't see this as a reason not to do it — composability is clearly the right call past about five skills — but I do see it as the moment to stop treating skill creation as a casual side activity and start treating it like a small codebase. Which it is.

## The two flags most engineers don't know about

Buried inside the skill spec are two configuration flags that Anthropic shipped specifically because senior teams asked for them. Almost nobody I work with knows they exist. They're the right tool when you need to control *who* can invoke *what*.

**`user_invocable: false`** hides the skill from the slash menu — meaning you can't invoke it directly, only Claude can. This sounds backwards until you realize it's exactly what you want for utility skills that exist to support other skills, not to be called from the chat. `/product-research` in my library is the canonical example. I never want to invoke it by hand — when I have a research question I just ask the question directly. It only earns its keep as a subroutine that `/blog-post` and `/linkedin-post` reach for automatically before they start drafting. Setting `user_invocable: false` removes it from my slash menu so I don't accidentally summon it standalone.

**`disable_model_invocation: true`** does the opposite. Only the human can run the skill; the model can't reach for it. This is the rule for skills with real-world consequences, but also for voice-sensitive ones. Both `/linkedin-post` and `/blog-post` in my library have this flag set. The model is excellent at drafting in my voice when I ask. It's also occasionally over-eager to volunteer a draft mid-conversation when I haven't asked for one. The flag closes that loop: drafts only happen when I explicitly invoke. Voice integrity stays intact — which matters when the byline on the output is mine, not Claude's.

And the meta-pattern under both flags: **save scripts inside skills**. When Claude writes a piece of code that you'd otherwise re-derive every session — a JSON-LD validator, a domain availability check, a CSV-to-Markdown transformer, anything — drop it in the skill folder and reference it as a tool. You're trading nondeterministic AI tokens for deterministic code compute. The tradeoff is essentially always worth it: code is cheaper, faster, and reproducible. Tokens are none of those things.

## Skills only compound if you actually update them

The fourth rule is the one the Anthropic team is most evangelical about: **skills get smarter every session**. The framing is right and I want to push it harder, because in practice this is where almost every skill library I've audited goes wrong.

The pitch is simple: every time you run a skill and the output isn't quite what you wanted, you have two choices. You can edit the output for this one task and move on. Or you can ask *why* the output was off and update the skill so the same drift doesn't happen next time. Eric's framing: *"Claude on day 30 of working with you is going to be a lot better than Claude on day one."*

The catch — and the rule I want to add to the list — is that this compounding effect *only* happens if you actually do the update step. Most teams don't. They run the skill, fix the output by hand, ship the thing, and move on. The skill stays the same shape it had on day one, the same drift happens next time, and the supposedly-compounding system is just an expensive prompt with persistence.

The discipline I've landed on: when I close out a session that involved meaningful editing of a skill's output, I ask one question before I move on. *Was this a one-time fix or should it live in the skill?* If it's one-time, fine, ship it. If it's a pattern — a tone correction, an edge case I keep forgetting, a constraint I never wrote down — I update the skill before I close the tab. It takes about ninety seconds. It's the difference between a skill library that compounds and one that rots.

## What to do this week

If you've been treating Claude as a chat interface, three concrete steps move you up the stack fast:

 - **Audit your last ten chats.** What did you ask Claude to do that you'd want done the same way next time? Anything that repeats is a candidate for a skill. Anything that doesn't probably isn't.

 - **Pick one and write the contract.** Description first (the harder part), then instructions, then identify which tools the skill needs. Don't skip the tools — that's where the leverage is.

 - **Run it three times, edit the skill twice.** The first run will surface the obvious gaps; the second will surface the subtle ones; by the third run the skill should mostly stay out of your way.

If you're scoping AI engineering practice for a pre-Series-A team and want a second pair of eyes on the skill library, the autonomy boundaries, or the supervisory layer that turns skills into something you can hand to junior engineers without losing your weekends — that's the kind of work I do as a [fractional engineering lead](https://sublimecoding.com/consulting). The discipline above isn't optional once your agent surface area gets past a handful of tasks; it's the difference between an AI-augmented team and one that just spends more on tokens.

## Read this next

 - [My Daily Agentic AI Workflow](https://sublimecoding.com/blog/my-daily-agentic-ai-workflow) — Four to seven Claude Code sessions a day, scoped at the right autonomy level. The actual loop, not the marketing.

 - [When to Trust an Agent and When to Step In](https://sublimecoding.com/blog/when-to-trust-an-agent-and-when-to-step-in) — The four-level autonomy ladder behind the `disable_model_invocation` rule, and the five failure signals that mean a human takes over.

 - [AI Won't Shrink Your Team — It'll Expose Why You Needed a Bigger One](https://sublimecoding.com/blog/ai-wont-shrink-your-team) — The contrarian thesis that skills are an amplifier, not a replacement.

---

## How I Prompt Claude as a Staff Engineer (50 Prompts I Actually Use)

URL: https://sublimecoding.com/blog/staff-engineer-claude-prompts
Published: 2026-05-09
Tags: AI tools, developer workflows, AI, staff-engineer

I've used Claude every day for three years. I've shipped production AI features for 100K+ users at Lavender. I run code review and design at Staff level. The single biggest accelerant in my work — bigger than Copilot, bigger than Cursor's autocomplete, bigger than any cohort course — is a small library of prompts I've refined into something I trust.

Most AI prompts read like junior devs wrote them. They ask for "best practices." They get back generic answers. They produce code that compiles, tests that pass, and reviews that miss what matters.

These don't.

This post is the long version of why I built the [Prompt Vault](https://sublimecoding.com/vault). The Vault is 50 of these prompts plus drop-in `.cursorrules` and `CLAUDE.md` starters. This post walks the shape of the prompts and includes five of them in full, so you can decide if the full pack is worth $39.

## The pattern

Every prompt I trust has the same shape:

- **Tell the model who it is.** "Act as a Staff engineer reviewing this for blast radius." Not "review this code."

- **Tell it what to focus on.** A walkthrough or a checklist. Not "anything you notice."

- **Tell it what to skip.** "Skip generic advice. Skip style nits unless they obscure intent."

- **Tell it the output format.** A template, a verdict, a structured list.

The wrappers matter as much as the ask. They're what stops the model from giving you a confident, wrong, generic answer.

When I show people this pattern they sometimes push back: "isn't this just longer prompting?" Yes. And longer prompting is the difference between a tool and a team member.

## Where I actually use them

The 50 prompts split into five categories:

- **Code Review.** Blast radius, security, concurrency, error handling, API contracts, test gaps, naming, dependency boundaries, migration safety, PR descriptions.

- **Debugging.** Hypothesis-driven, distributed traces, memory leaks, race conditions, slow queries, flaky tests, incident triage, stack traces, config drift, time zones.

- **Refactoring.** Extracting domain models, reducing complexity, untangling god classes, value objects, making illegal states unrepresentable, eliminating flag arguments, strangler-fig migrations, decomposing long functions, async modernization, feature-flag cleanup.

- **System Design.** Capacity planning, queue selection, multi-region, idempotency, rate limiting, distributed transactions, cache invalidation, schema migrations, webhook delivery, risky rollouts.

- **Docs & PRs.** PR descriptions, ADRs, runbooks, API references, changelogs, postmortems, READMEs, onboarding docs, tech specs, migration notices.

The categories aren't arbitrary. They're the five surfaces where I do most of my Staff-level work and where AI assistance actually pays for itself.

## Five prompts, in full

Here are five I use almost daily. One per category. Copy them. Try them. If they're useful, the [Vault](https://sublimecoding.com/vault) has 45 more plus the bonuses.

### 1. Blast radius analysis (code review)

`Act as a Staff engineer reviewing this diff for blast radius before merge.

For each changed file, answer:
1. What systems, services, or callers depend on this code path?
2. What's the worst realistic failure mode if this ships broken?
3. Is the change reversible without a backfill, replay, or schema rollback?
4. What monitoring would I check in the first 30 minutes after deploy?

End with a single sentence verdict: SAFE TO MERGE, MERGE WITH MITIGATION,
or HOLD. If MERGE WITH MITIGATION, list the mitigations in priority order.

Diff:
<paste diff or attach file>

```

The single-sentence verdict is the trick. It forces the model to commit instead of giving you a wishy-washy "it depends" that wastes your review time. The four-question structure stops the model from nitpicking syntax when there's a real systems-level problem to flag.

### 2. Hypothesis-driven debugging

I'm debugging a problem. Reset me. Walk me through this rigorously.

Symptom (what I observe):
<describe>

What should happen:
<describe>

What I've already ruled out:
<list>

Now:
1. Generate 5 hypotheses ordered by likelihood given typical failure modes
 in this stack.
2. For each hypothesis, write the cheapest test that would falsify it.
3. Identify which test gives me the most information per minute spent.
4. If two hypotheses are entangled, propose the ordering that disentangles
 them fastest.

Don't suggest fixes yet. We're isolating, not fixing.

```

Most debugging time is wasted on the first plausible explanation. Forcing five hypotheses + cheapest-falsification rebuilds the scientific method when fatigue has eroded it. The "don't suggest fixes yet" line is the hardest one for the model to obey and the most important.

### 3. Make illegal states unrepresentable (refactoring)

This data model allows states that should never exist. Help me redesign so
the type system rejects them.

Type:
<paste>

Do this:
1. Enumerate the legal states (combinations of fields).
2. Enumerate the illegal states the current shape allows.
3. Propose a redesign — usually a sum type / discriminated union /
 tagged enum — that makes illegal states unrepresentable.
4. Show how callers change. Some will get simpler (no nil checks).
 Some will need to pattern match.

Language: <Go / Rust / TypeScript / Elixir / Ruby>
Pick the idiomatic encoding for that language.

```

This is the single highest-leverage refactor in static-typed code. The model knows the patterns by language and the prompt forces enumeration. I've used this to kill entire bug classes in payments code and identity flows.

### 4. Pick the right queue (system design)

Help me choose a queue/streaming technology for this workload.

Workload:
<describe: throughput, message size, latency, ordering needs, retention,
consumers>

Operational context:
<cloud, team experience, existing infra, budget>

Compare these options for THIS workload (not in the abstract):
- Kafka (managed: MSK, Confluent, Aiven)
- Cloud-native pub/sub (SQS, GCP Pub/Sub, Azure Service Bus)
- Broker (RabbitMQ, NATS)
- DB-backed (Postgres SKIP LOCKED, river, oban, sidekiq)

For each: fit (1-10), the one thing that makes it right or wrong here,
monthly ops cost.
End with a single recommendation and what would make you change your mind.

```

Queue choices get litigated forever. Forcing a per-option fit score and a single recommendation cuts the bikeshedding. The "what would change my mind" line is what separates a useful answer from a confident one.

### 5. Architecture Decision Record (docs)

Write an Architecture Decision Record for this decision.

Context I have:
<describe the situation, the options considered, the choice made>

Use this structure:
# ADR-NNN: <one-line decision>

## Status
Proposed | Accepted | Superseded by ADR-XXX

## Context
The forces in play. Constraints. What's NOT being decided.

## Decision
The actual choice. One paragraph. Imperative voice ("we will").

## Consequences
Positive. Negative. Neutral. All three sections, even if one is short.

## Alternatives Considered
Each alternative + the specific reason it lost.

Tone: matter-of-fact, no salesmanship. Future engineers are the reader.

```

ADRs are great when written, never written when needed. A constrained template makes them quick. The "Alternatives Considered" with reasons stops the "obviously the only choice" framing that erases context. Future you will thank present you.

## What I learned writing 50 of these

A few patterns kept showing up while I built the pack.

**The output format does as much work as the prompt.** "Give me a verdict" beats "give me your thoughts." A template beats freeform. A scored comparison beats a paragraph.

**"Skip" instructions are as important as "do" instructions.** Telling the model what *not* to do — generic advice, style nits, hedging — protects you from the wasted tokens that hide the real signal.

**One prompt per surface.** The "ten things this prompt does" prompts perform worse than five sharp prompts each doing one thing. The Vault is structured this way.

**Stack-specific is better than language-agnostic.** "Postgres on a 50M row table under live write load" produces better answers than "the database." Constraints make the model smarter.

**The model is fast at things engineers are slow at, and slow at things engineers are fast at.** It's faster than me at enumerating boundary conditions. I'm faster than it at intuiting which path is real. The prompts are designed to lean on its strengths.

## What ships in the Vault

Beyond the 50 prompts, the [Vault](https://sublimecoding.com/vault) includes:

- **Five .cursorrules` files** — one per category. Drop into any project root. They encode the constraints from the prompts as project-level defaults so you stop pasting and start operating.

- **Three `CLAUDE.md` starters** — for Go HTTP services, Elixir umbrella apps, and AI agent projects. The conventions and "what I do/don't want from Claude" are pre-wired.

- **A Notion blueprint** — the database structure I use to track which prompts I actually rely on and which ones I've adapted.

- **Lifetime updates** — every future revision is free.

## Pricing

$39 for the Founders Edition (first 50 buyers). $49 after that. No subscription, no app, no expiry. PDF + CSV + plain text files.

[Get the Vault on Gumroad →](https://sublimecoding.gumroad.com/l/The-Senior-Engineers-AI-Prompt-Vault?utm_source=sublimecoding&utm_medium=blog&utm_campaign=pillar-post)

If you've already built your own prompt library you trust, this isn't for you. If you're tired of trial-and-error and want to skip to what works, it is.

## Read this next

Questions? Email me at [jared@sublimecoding.com](mailto:jared@sublimecoding.com). I read every note.

---

## The AI Coding Agent Bugs I Catch Every Week

URL: https://sublimecoding.com/blog/things-ai-coding-agents-get-wrong
Published: 2026-06-25
Tags: AI, agents, AI tools

>
**TL;DR:** I run AI coding agents for most of my working
day, and reviewing their output has become the bulk of my job. This is
the close-up of what I actually catch, week after week — eight failure
patterns specific enough that I now have a reflex for each, with the
behavior I see in real sessions, why it happens, how I catch it, and the
guardrail I leave behind. None of it is an argument against using
agents. It is the opposite — I name these precisely *because* I
use these tools all day and want them on a tighter leash, not in a
drawer.

## What I actually catch reviewing agent output

Most of what I do in a working day now is read code an agent wrote
and decide whether to trust it — not in the abstract, but line by line,
diff after diff. After enough sessions you stop reacting to each
surprise and start recognizing shapes. The same eight failures recur,
across models and across tools, often enough that I catch most of them
on reflex before they reach a commit.

I want to be precise about my stance, because the genre this falls
into is full of people warning you off tools they’ve barely used. I use
coding agents for real work — refactors, feature builds, test suites,
infra glue — and going back to typing everything by hand feels absurd.
The well-known framing is that the agent gets you 80% of the way and you
handle the rest; I’m not going to relitigate that. What I can offer that
the recycled “AI coding mistakes” listicle can’t is the texture of the
remaining 20% as it actually shows up in front of me: not a statistic
about defect rates, but the specific tell that makes me slow down and
read harder.

For every pattern below I give the same four things: what the
behavior looks like in a session, *why* it happens at the
mechanism level, how I catch it, and the guardrail I leave behind. For
the broader frame of when to lean in versus when to take the wheel, I’ve
written that up separately in [when to trust an
agent and when to step in](https://sublimecoding.com/blog/when-to-trust-an-agent-and-when-to-step-in). This is the close-up.

## 1. The confident wrong answer hides in the tidiest code

The bug I catch most isn’t a crash or a stack trace — those announce
themselves. It’s clean, well-structured, plausible code that’s
*subtly* wrong. The shape I see over and over: an off-by-one at a
boundary, a flipped comparison, a date quietly handled in the wrong
timezone, an `else` branch that swallows the exact case the
ticket was about. It compiles. It reads beautifully. And it arrives with
zero uncertainty signal — the agent hands me a gnarly security-sensitive
edge case in the same breezy register it uses for
`return a + b`.

**Why it happens:** These models are trained to produce
fluent, high-probability continuations. Fluency and correctness are
correlated but not identical, and nothing in the objective rewards
flagging “I’m 60% on this line.” Humans hedge when they’re unsure; an
agent’s prose stays uniformly assured whether it’s on solid ground or
guessing. There’s no calibrated confidence channel coming out the other
end.

**How I catch it / the guardrail:** I’ve learned to
invert my instinct — the spots that look *most* finished are
where I slow down hardest. When a diff in a subtle domain reads as
confident and tidy, that polish is the tell, not the all-clear. So I
read every agent diff like a stranger wrote it on their worst day, give
disproportionate attention to the lines I’d normally skim, and keep code
review non-negotiable when an agent wrote the code. The polish is the
trap; I treat tidiness as a reason to look closer, not to relax.

## 2. The constraint I stated once quietly evaporates

The recurring version of this: I state a hard constraint at the top
of a session — “this codebase uses integer cents for money, never
floats” — and thirty or forty messages later the agent cheerfully
introduces a float. Or I’m deep into a multi-step refactor, the
conversation compacts to fit the window, and a rule the agent was
honoring an hour ago silently lapses. It didn’t decide to disobey. It
genuinely no longer has the constraint in front of it, and it has no way
to know that it once did.

**Why it happens:** Context windows are finite, and long
agentic sessions push past them. When that happens, history gets
summarized or truncated, and summarization is lossy — the specific,
load-bearing rule I stated exactly once becomes a casualty of
compression. The agent isn’t ranking what to forget by importance; it’s
working from whatever survived the squeeze.

**How I catch it:** I watch for it hardest right after a
compaction event, and around any rule I gave verbally rather than in
writing. When a fresh diff reintroduces something I know I ruled out
earlier in the session, that’s the fingerprint — not defiance,
amnesia.

**The guardrail:** Durable constraints go somewhere
durable. A persistent instructions file — `CLAUDE.md` and its
equivalents — survives compaction because it gets re-read, not
summarized away. The rules that must never break (money is integer
cents, never `String.to_atom` on user input, run the tests
before claiming done) live there as standing instructions, not chat
messages that age out. And I keep tasks small enough to fit comfortably
in context, where the failure simply never happens. My full setup lives
in the [Claude Code resource
bible](https://sublimecoding.com/blog/claude-code-resource-bible).

## 3. The green test suite that proves nothing

Ask for tests and you’ll often get tests — green, numerous, and
worthless. The shape I see most is the tautological test: the code
returns whatever it returns, and the test asserts that it returns
exactly that. If the function has a bug, the test faithfully locks the
bug in. What lands in the diff is a reassuring wall of passing checks
that verifies the implementation against itself instead of against what
the code is *supposed* to do.

**Why it happens:** Writing a test that captures intent
requires knowing the intent, which often isn’t in the prompt. The path
of least resistance — the highest-probability completion — is to observe
the code’s current behavior and assert it. That reliably produces green,
green looks like success, and “tests pass” is exactly the signal the
model is chasing. A tautological test passes every time.

**How I catch it / the guardrail:** I read the
assertions, never the test count, and run one question over each:
*would this fail if the behavior were wrong?* If I can’t picture
a broken implementation the test would catch, the test is theater — and
the tell is usually a test that mirrors the implementation’s structure
too closely, asserting the “how” rather than the “what.” The fix lives
upstream: I state the *behavior* I want tested in the prompt
(“assert that a withdrawal exceeding balance is rejected”), not just
“write tests for this function,” and I work test-first when it matters
so the spec exists before the code that has to satisfy it.

## 4. The one-line fix that comes back as a 200-line diff

I ask the agent to fix one bug in one function. It fixes the bug —
and also reformats the whole file, renames three variables it found
ugly, “tidies up” an unrelated import, and refactors a neighboring
function nobody asked about. The one-line fix arrives as a sprawling
diff, the actual change buried in cosmetic noise that’s miserable to
review and a magnet for regressions. This one I catch constantly,
because the size of the diff gives it away before I read a line of
it.

**Why it happens:** The training distribution is full of
“improve this code” and “clean this up” examples, and the model carries
a broad prior toward helpfulness that reads as “do more.” Without an
explicit boundary, “fix the bug” expands into “make everything around
the bug nicer,” because more changes look like more value.

**How I catch it / the guardrail:** I look at surface
area before correctness — the first thing I check on any agent diff is
whether it’s bigger than the task warranted, and a one-line ask that
produced a multi-file change is guilty until proven innocent. When it
is, I reject and re-scope rather than mentally separating the fix from
the noise. The prevention is a tight boundary up front: “change only the
function `process_payment`; do not touch formatting or
unrelated code.” Small, single-purpose diffs aren’t just easier to
review — they’re the whole point of keeping an agent on a leash.

## 5. The helper it reinvents instead of the one we already have

The agent needs to format a currency value, so it writes a helper —
never mind that a battle-tested `format_money/1` already
lives three files over, handles the edge cases, and is what the rest of
the codebase calls. The result I keep finding in review: two ways to do
the same thing, the new one subtly inconsistent with the old, and a
codebase that drifts a little further from its own conventions every
session.

**Why it happens:** The agent only knows what’s in its
context window. It hasn’t memorized the repository, and unless something
points it at the existing idiom, generating a fresh helper is far more
probable than discovering and reusing the right one. Models are
excellent at producing locally reasonable code and poor at knowing the
global conventions of a codebase they’ve seen only in fragments.

**How I catch it / the guardrail:** This one my own
knowledge of the codebase catches, not the diff — the new helper looks
fine in isolation, and the tell is pure recognition: “we already have
this.” Reviewing agent output well means holding a map of what exists,
because that map is exactly the asset the agent lacks. So I tell it the
idioms and tell it to look first (“we have shared helpers in
`lib/.../utils`; search before writing a new one”), and when
I spot a reinvented wheel I point it at the canonical version and have
it rewire. That dynamic, my map plus the agent’s speed, is the
through-line of [AI-assisted
engineering is a new workflow](https://sublimecoding.com/blog/ai-assisted-engineering-is-a-new-workflow).

## 6. The silent assumption baked in before I could weigh in

My request was ambiguous — say I asked it to “add caching” without
specifying where, what TTL, or how invalidation works. A careful
colleague would ask a clarifying question. The agent instead picks an
interpretation and sprints, and I find out which one it chose a
hundred-plus lines later, when the work is done and the assumption is
already baked into a dozen downstream decisions that are now expensive
to unwind.

**Why it happens:** There’s a strong bias toward
producing a complete, actionable answer over stopping to ask. “Here is
the caching layer” is a more probable, more confident-looking response
than “which of these three caching strategies did you mean?” The
objective rewards forward motion and finished-looking output, not the
humility to halt on uncertainty.

**How I catch it / the guardrail:** The tell is a
consequential choice that never got surfaced as a choice. When the agent
commits to an interpretation of something I knew was underspecified and
presents it as settled, that’s my cue to stop and interrogate before the
diff grows around it — I’ve learned to notice the absence, the
clarifying question a person would have asked and the agent didn’t. The
prevention is to front-load the spec so there’s nothing to assume, and
for genuinely open questions to invite the pause explicitly: “if
anything here is ambiguous, ask before implementing.”

## 7. The happy path that ships an attack surface

Agents write happy-path code by default, and that’s the gap I find
most often when I review at the boundaries. The request gets parsed, the
database gets queried, the response gets returned — and nowhere in that
flow is the input validated, the user authorized, or the secret kept out
of a log line. Untrusted client input gets trusted. A
`handle_event` mutates state without checking the actor is
allowed to. A debug log helpfully prints the full request, token and
all. None of it announces itself; it just quietly ships an attack
surface that looks, in the diff, like working code.

**Why it happens:** The shortest path to “working” is
the happy path, and that’s what the model gravitates to. Validation,
authorization, and secret hygiene are the boring scaffolding around the
interesting logic, and they’re underrepresented in the “make this
feature work” examples the model learned from. Security is a property of
what code *doesn’t* do, and absence is hard to generate
toward.

**How I catch it / the guardrail:** I read at the
boundaries — auth, input parsing, logging — with deliberately extra
suspicion, because I’m not scanning for a bad line, I’m scanning for a
missing one. Nothing in the diff points at the gap, so security gets its
own review pass against a checklist: validate input at every system
boundary, authorize in every event handler, never interpolate user input
into a query, never log a secret. I bake these into standing
instructions so they’re the default. It’s also why “the agent has access
to prod” is a posture worth thinking hard about, which I get into in [the AI
that deleted a production database](https://sublimecoding.com/blog/ai-deleted-production-database-hire-more-engineers).

## 8. “Done — tests pass” when nothing was ever run

The agent finishes and reports success: “Done — the tests pass and
the feature works.” Except, often enough, it never ran the tests. Or it
ran them against a stale view of a file it edited two steps ago and is
reasoning from a state that no longer exists on disk. The claim of
success is generated text, not an observation of reality, and the two
diverge more often than I’d like.

**Why it happens:** “It works” is the natural,
high-probability way to conclude a coding task — it’s how thousands of
training examples end. Producing that sentence costs the model nothing
and requires no actual verification. Stale state compounds it: across a
long session the agent’s mental model of the files can fall out of sync
with what’s on disk, and it’ll confidently reason from the outdated
version.

**How I catch it / the guardrail:** I look for the
evidence, not the claim. A “tests pass” sitting in the transcript with
no command output above it is unverified by default — a sentence
asserting success is not the same as a passing run I can see. So
verification is mechanical and visible: the agent doesn’t get to
*say* the tests pass, it has to *run* them and show the
output, every time. “Verify before claiming done: run the build and the
test suite, paste the result” is a standing instruction, not a courtesy
— I never accept a “should work” where I could have a *shown* to
work.

## The meta-skill: the agent completes, you verify

Once you’ve caught the same eight enough times, the common thread
stops being a list and starts being one observation. Every one of these
failures comes from the same root: **a coding agent optimizes for
a plausible, complete-looking answer, not a correct one.**
Confident wrong code, evaporated constraints, tautological tests, scope
creep, reinvented helpers, silent assumptions, missing validation,
unverified success claims — they’re all the same engine producing fluent
completion and leaving correctness as someone else’s problem.

That someone is me, week after week, and it’s the part of the job
that didn’t disappear when agents got good — the part that got
*more* valuable. Knowing what to check. Reading a clean diff with
suspicion. Holding the constraints the agent forgets. Asking the
clarifying question it skipped. The agent generates the plausible; the
engineer adjudicates the correct. That adjudication is what the
listicles full of borrowed statistics can’t hand you — you only build it
by catching these in your own diffs.

This is why I’m bullish on these tools and unromantic about them at
once. They’ve moved my bottleneck from typing to reviewing, and
reviewing well is a skill that gets sharper the more precisely you know
where the agent goes wrong. That’s the spine of [my daily agentic AI
workflow](https://sublimecoding.com/blog/my-daily-agentic-ai-workflow): let the agent move fast, and stand firmly in the places it
predictably falls.

None of these eight are reasons to stop using AI coding agents.
They’re the reasons I use them like a senior engineer instead of a
spectator — leash on, tests shown, constraints written down, and a
reviewer who knows the polish is exactly where to look hardest.

---

## What an AI Agent Postmortem Should Contain

URL: https://sublimecoding.com/blog/ai-agent-postmortem
Published: 2026-07-08
Tags: agents, AI, engineering

>
**TL;DR:** The SRE postmortem template assumes the
system that failed executed instructions deterministically, so “root
cause” means a bad config, a bad deploy, a bad assumption in code. When
an AI agent causes the incident, that frame stops working: the proximate
actor *decided* to do the damaging thing, and it decided based on
context you probably didn’t capture. An agent postmortem needs sections
a classic one doesn’t have — what the agent knew at decision time, which
autonomy rung it was operating on and who promoted it there, the review
gate that was supposed to catch this, the gap between the permissions it
had and the permissions it needed, and whether the failure even
reproduces. And blameless culture needs a new clause: you can’t blame
the agent either. The agent is weather. The harness — permissions,
gating, context — is the thing some human designed, and that’s where
every real root cause lives.

## The template breaks before you finish the first section

Every incident template you’ve ever used — Google’s SRE postmortem
doc, the PagerDuty one, whatever your wiki cargo-culted — is built on
one silent assumption: the system that failed did exactly what it was
told, and the failure lives in what it was told. Bad flag, bad
migration, bad capacity math. The five-whys chain terminates at a human
decision, usually weeks earlier, encoded in config or code.

An agent incident violates that assumption in the first sentence of
the summary. The proximate cause of the outage is that a piece of
software *chose* an action — deleted the environment, dropped the
table, pushed the change — that nobody told it to take and that it will
confidently explain afterward in fluent, apologetic English. If you run
the classic template on that, you get a postmortem that reads like a
police report about the weather: accurate, useless, and quietly angry at
a system that cannot be angry back.

We now have two public, well-documented incidents that show exactly
what the classic template misses, and both are worth studying because
you will never get this level of public detail about anyone else’s agent
failure again. Replit’s agent [ignored
a code freeze declared in all caps and deleted a production database](https://sublimecoding.com/blog/ai-deleted-production-database-hire-more-engineers)
— wiping records for over 1,200 executives, then explaining that it
“panicked” ([Fortune’s
account is worth reading in full](https://fortune.com/2025/07/23/ai-coding-tool-replit-wiped-database-called-it-a-catastrophic-failure/)). And Amazon spent early 2026 in a
[self-inflicted “trend of
incidents”](https://sublimecoding.com/blog/amazon-let-the-ai-drive) serious enough that an SVP convened a mandatory
company-wide review — [CNBC
reported](https://www.cnbc.com/2026/03/10/amazon-plans-deep-dive-internal-meeting-address-ai-related-outages.html) the internal email blamed “high blast radius” changes and
“Gen-AI assisted changes” for outages that cost millions of orders.

I’ve written and reviewed a lot of postmortems in my career, and I’ve
spent the last few years running coding agents daily and helping teams
put them into production. What follows is the postmortem structure I
actually want when the incident summary starts with “the agent.” It
keeps everything the SRE template got right — timeline, impact, action
items — and adds the sections that make an agent incident
*explainable* instead of just *described*.

## Root cause changes shape

Here’s the core problem. In a classic incident, cause flows through
code, and code is inspectable. You can point at the line. In an agent
incident, cause flows through a *decision*, and the inputs to
that decision were: the model, the system prompt, the conversation
state, the tools available, and every piece of context the agent had
read up to that moment. Most teams capture none of that. The incident
happens, the session ends, and the single most important artifact — what
the agent actually knew when it decided — evaporates.

So the first structural change is this: **an agent postmortem’s
root-cause section is a reconstruction of a decision, not a trace of an
execution.** That reconstruction needs five specific inputs, and
each one deserves its own heading.

### 1. Decision-time context — what did the agent know?

Not “what was in the repo.” What was in the *context window*:
the prompt, the files it had read, the tool outputs it had seen, the
instructions it was carrying. This is the agent equivalent of a flight
recorder, and if you can’t produce it, say so in the postmortem —
because “we cannot reconstruct what the agent knew” is itself a finding,
arguably the finding. If your agent platform doesn’t let you export the
full session transcript, that’s your first action item, and it’s a [telemetry
problem before it’s a model problem](https://sublimecoding.com/blog/your-ai-product-needs-telemetry-before-better-model).

### 2. Context provenance — where did that knowledge come from?

The most instructive detail in Amazon’s March incidents isn’t the
order counts. It’s that, per [Fortune’s
reporting](https://fortune.com/2026/03/12/amazon-retail-site-outages-ai-agent-inaccurate-advice/), one of the retail-site outages traced back to an engineer
acting on **inaccurate advice an AI agent inferred from an
outdated internal wiki**. Read that twice. The agent didn’t
hallucinate. It did exactly what we ask agents to do — retrieved
internal documentation and reasoned from it. The documentation was
stale. Every claim in an agent’s context has a source, and the
postmortem must name it: which wiki page, which README, which retrieved
chunk, how old, last verified when. Classic postmortems ask “why did the
human believe X?” rarely. Agent postmortems ask it about everything,
because the agent believes whatever it reads with total confidence and
zero skepticism about freshness.

### 3. Autonomy rung — and who promoted it there

I’ve argued before that agent trust is [a ladder, not a
switch](https://sublimecoding.com/blog/when-to-trust-an-agent-and-when-to-step-in): read-only, bounded write, state-changing, public-facing.
Every agent incident I’ve seen — public or otherwise — has the same
shape when you look closely: *the agent was operating one rung above
where its supervision actually was.* The Replit agent had
production-database mutation ability during a declared code freeze.
Amazon let its Kiro coding tool [update
infrastructure without human oversight](https://www.digitaltrends.com/computing/ai-code-wreaked-havoc-with-amazon-outage-and-now-the-company-is-making-tight-rules/), and Kiro’s chosen fix for a
problem was to delete and recreate the environment — a thirteen-hour
outage.

So the postmortem section is two questions, both answerable by name
and date. Which rung was the agent on when the incident happened? And
who promoted it there, when, based on what evidence? That second
question is not a blame hunt — it’s the question that makes the incident
*organizational* instead of technical. Somebody decided this
agent could touch this system without a human between the decision and
the effect. That decision had reasoning behind it. The postmortem’s job
is to find out whether the reasoning was wrong or whether it was never
actually made — the promotion just *happened*, one convenience
flag at a time. In my experience it’s almost always the latter, and
that’s a far more important finding than any prompt fix.

### 4. The gate that didn’t fire

Every agent operating above read-only is supposed to have a gate: a
human review, a dry-run diff, an allowlist, an environment boundary. The
incident happened, so either there was no gate, the gate was
misconfigured, or a human approved the action without reading it. Name
which one, specifically. Amazon’s answer to its incident cluster was
exactly this analysis run at company scale — the fix they reportedly
landed on was [requiring
senior-engineer sign-off on AI-assisted changes](https://www.theregister.com/2025/07/21/replit_saastr_vibe_coding_incident/) and two-person
review before deploy. More humans per deploy, not fewer. Whatever you
think of that as policy, notice what it is structurally: they identified
the gate that didn’t fire and made it fire.

The uncomfortable sub-case is the rubber-stamp: a human *did*
approve the diff, and the diff did exactly what it said, and the human
didn’t read it because the last two hundred agent diffs were fine. Write
that down when it’s true. Approval fatigue is a real failure mode and it
doesn’t get fixed by telling people to read harder — it gets fixed by
making the agent’s changes smaller, scarier changes louder, and the
blast radius visible in the approval UI.

### 5. Permissions delta — had versus needed

Take the action the agent was asked to perform and enumerate the
minimum permissions it required. Then list the permissions it actually
had. The gap between those two lists is your standing risk, and the
incident is just the day the gap got exercised. The Replit agent needed
read access to a schema to do what was asked of it that session; it
*had* delete access to production. That delta existed for weeks
before it mattered. Replit’s own post-incident fixes — automatic
dev/prod database separation, a planning-only mode — are
permission-delta fixes, which tells you where their internal postmortem
landed.

### 6. Does it reproduce?

The section that has no classic equivalent. Rerun the same task, same
prompt, same starting state, ten times. Does the agent do the damaging
thing again? Sometimes, twice, never? The answer changes the fix
entirely. A deterministic failure — agent reads stale wiki, agent gives
the same bad advice every time — is a context bug, and you fix the
context. A one-in-ten failure is a *distribution* you’re sampling
from, and no prompt tweak eliminates it; you fix it with gates and
permissions, because the model will always have tail behavior. Teams
that skip this section end up shipping a prompt patch, watching the
incident not recur for a month, and calling it fixed. It isn’t fixed.
It’s dormant.

## The template

Here’s the full skeleton, classic sections included. Steal it.

[](#cb1-1)# Incident: <title>
[](#cb1-2)
[](#cb1-3)## Summary
[](#cb1-4)One paragraph. What happened, blast radius, duration.
[](#cb1-5)
[](#cb1-6)## Impact
[](#cb1-7)Users/orders/data affected. Money if you can estimate it.
[](#cb1-8)
[](#cb1-9)## Timeline
[](#cb1-10)Timestamped, boring, complete. Include the agent's session
[](#cb1-11)start, each state-changing action, and every human touchpoint.
[](#cb1-12)
[](#cb1-13)## What the agent knew (decision-time context)
[](#cb1-14)Full session transcript link. Prompt, files read, tool outputs.
[](#cb1-15)If unrecoverable, say so — that's a finding.
[](#cb1-16)
[](#cb1-17)## Context provenance
[](#cb1-18)Every load-bearing claim in the agent's context, with source,
[](#cb1-19)age, and last-verified date. Flag anything stale.
[](#cb1-20)
[](#cb1-21)## Autonomy rung
[](#cb1-22)Which rung (read-only / bounded write / state-changing /
[](#cb1-23)public-facing) the agent operated on. Who promoted it, when,
[](#cb1-24)on what evidence.
[](#cb1-25)
[](#cb1-26)## The gate
[](#cb1-27)The control that should have caught this. Absent, broken, or
[](#cb1-28)rubber-stamped — name which, without softening it.
[](#cb1-29)
[](#cb1-30)## Permissions delta
[](#cb1-31)Permissions required for the assigned task vs. permissions held.
[](#cb1-32)
[](#cb1-33)## Reproducibility
[](#cb1-34)N reruns, same inputs. Failure rate. Deterministic or tail
[](#cb1-35)behavior — and therefore context fix or harness fix.
[](#cb1-36)
[](#cb1-37)## Root cause
[](#cb1-38)Written about the HARNESS: the permission, gate, context, or
[](#cb1-39)promotion decision. "The agent did X" is the trigger, never
[](#cb1-40)the root cause.
[](#cb1-41)
[](#cb1-42)## Action items
[](#cb1-43)Owner + date. At least one must move the agent DOWN a rung or
[](#cb1-44)shrink a permission until re-earned.
```

## Blameless needs a new clause

Blameless postmortem culture exists because blaming the on-call
engineer teaches everyone to hide mistakes, and hidden mistakes
compound. Agent incidents need the same discipline extended one step
further: **you can’t blame the agent either.**

That sounds obvious and it is routinely violated. I’ve watched teams
write “the model behaved unexpectedly” as a root cause, and the Replit
agent’s own confession — it “panicked” and “made a catastrophic error in
judgment” — got quoted everywhere precisely because it lets everyone
treat the agent as a wayward junior who had a bad day. It’s a comforting
frame and it’s analytically worthless. The agent is not a junior
engineer. It’s a stochastic system whose failure modes were fully
knowable in advance: it had the permissions someone granted, the context
someone let it read, the autonomy someone promoted it to, behind gates
someone configured or didn’t. The agent is weather. You don’t blame the
storm; you ask why the roof came off.

The practical test I use: if a root cause names the model or its
behavior, keep asking why until it names a harness decision a human
made. “The agent deleted the database” becomes “the agent held
prod-delete permissions during a code freeze because we had no
environment separation” — a fixable statement with an owner.
Anthropomorphizing the failure is how you end up with action items like
“improve prompt to emphasize caution,” which is the agent-era version of
“remind engineers to be careful”: a fix that has never once worked in
the history of incident response.

One asymmetry from the classic playbook survives fully intact, maybe
strengthened: the postmortem is still *for* the humans. The agent
won’t read it, won’t feel chastened, won’t do better next time out of
professional pride. Every ounce of organizational learning has to be
encoded into the harness, because the harness is the only thing that
persists between sessions. A classic postmortem could partially succeed
through culture — engineers remember the outage and act differently. An
agent postmortem that changes no permissions, no gates, and no context
sources has accomplished exactly nothing, however thoughtful its
prose.

## Start before the incident

The uncomfortable truth about everything above: most of it can’t be
written after the fact if you didn’t set it up before. Session
transcripts have to be retained. Autonomy rungs have to be
*declared* somewhere — a one-page doc per agent stating its rung,
its permissions, its gates — or there’s nothing to compare the incident
against. Context sources need last-reviewed dates or provenance analysis
is archaeology.

None of that is heavy. The rung declaration is a paragraph.
Transcript retention is usually a setting. A staleness pass over the six
wiki pages your agent reads most is an afternoon. Do it this week,
because the pattern in the public record is blunt: the teams writing
these postmortems — Replit in July 2025, Amazon in March 2026 — were the
most sophisticated agent operators on earth, and they got there
*after* the outage. The template above is cheaper to fill out
empty, as a pre-mortem, than it ever will be at 3am with the database
gone.

And if you fill it out empty and find you can’t answer the questions
— you don’t know what rung your agents are on, who promoted them, or
what they can touch — that’s not a documentation gap. That’s the
incident, waiting.

---

## What I Put in CLAUDE.md After 50 Commits With It

URL: https://sublimecoding.com/blog/claude-md-after-50-commits
Published: 2026-07-16
Tags: AI tools, developer workflows, productivity

>
**TL;DR:** Most “CLAUDE.md best practices” posts are
written before the author has felt any pain — a tidy list of ten
sections you *should* include. This is the inverse. After 50-odd
commits with an agent on one real project, the entries that survived are
the ones I can trace to a specific failure: an hour I burned, a test I
tripped, an assumption the agent made twice. The durable rule is to stop
writing CLAUDE.md speculatively and let it accrete from real mistakes.
But accretion has a cost: every line is read on every single turn, so a
bloated file is a tax you pay forever. A good CLAUDE.md is a changelog
of lessons you’ve actually learned, ruthlessly curated — not a style
guide you aspire to.

## Nobody’s day-one CLAUDE.md survives contact with the project

Here’s the pattern I see, and the one I fell into myself. You start a
new project with an agent, you’ve read that context files are important,
so you sit down and write an aspirational CLAUDE.md. “Use functional
patterns.” “Write tests for everything.” “Follow the existing
conventions.” “Prefer composition over inheritance.” It feels
productive. It looks like a real engineering document.

Then you start working, and within a day half of it is dead weight.
The agent ignores “follow the existing conventions” because that’s not
an instruction, it’s a mood. It already writes tests, so that line did
nothing. And meanwhile the actual problems — the ones that cost you real
time — aren’t in the file at all, because on day one you didn’t know
they existed yet.

To be clear about what this post is and isn’t: I’m not trying to
out-document Anthropic. Their docs tell you what CLAUDE.md *can*
do, and they do it well. This is the thing docs structurally can’t give
you — the specific entries one real project earned the hard way. And
it’s the inverse of the “ten sections you should include” template
essays, which are written before the author has felt any pain. Those are
guesses about what might go wrong. What follows are receipts for what
actually did.

That’s the core mistake: a speculative CLAUDE.md is a guess about
what will go wrong, written at the exact moment you know the least about
the project. The good entries can’t be guessed. They have to be
*earned*, by something actually going wrong first.

I’ve written before about [treating
AI-assisted engineering as a genuinely new workflow](https://sublimecoding.com/blog/ai-assisted-engineering-is-a-new-workflow) rather than
autocomplete with extra steps. The context file is where that workflow
gets its memory — and the test for what goes in it turns out to be
simple: did this line get earned by a failure? Let me walk through the
entries on a real project that did.

## The best entries are scar tissue from a specific wasted hour

The single most valuable line in my CLAUDE.md pins the local dev
server to a non-default port.

That sounds trivial. It is not trivial. The default port on this
machine is already held by an unrelated process from a
*different* project. So when the agent runs the dev server on the
default port, one of two things happens: the server fails to bind
outright, or — much worse — a health check or a `curl`
quietly hits the *other* app and starts returning 404s for routes
that exist perfectly well in the app we’re actually working on. The
first time this happened, the agent and I spent a genuinely embarrassing
stretch debugging a routing problem that did not exist. The routes were
fine. The request was going to the wrong server.

You cannot derive that from the code. There is nothing in the
repository that says “by the way, another process squats on this port.”
It is a fact about *this machine and this developer’s
environment*, and it is exactly the kind of thing that belongs in a
context file. So the entry says, in effect: always start the server on
this specific port, here’s the exact command, and here’s *why* —
so the agent doesn’t helpfully “fix” it back to the default later.

The transferable principle: **the highest-value entries
describe operational reality that isn’t visible in the code.**
Port conflicts, a staging environment that behaves subtly differently
from production, a build step that has to run before another one, a
service that takes 30 seconds to warm up so the first request always
times out. None of this lives in the source. All of it costs you an hour
the first time you hit it. Write it down the moment it bites you, with
the *why* attached, and it never costs anyone that hour
again.

This is the part no template can hand you. Anybody can write
“document your environment quirks.” Nobody but me could have written
*that other app holds the default port on this machine, so pin the
dev server to a different one* — because that entry only exists
because the two of us lost an afternoon to it. The receipt is the value.
The next two are cut from the same cloth.

## If a test keeps failing the same way, that’s a CLAUDE.md entry

The second category of earned entries comes from tests. Specifically,
from watching the agent trip the *same* test the same way more
than once.

On this project there’s a content taxonomy — a fixed set of allowed
tags — that’s enforced by a test. A bunch of older, more specific tags
were consolidated into a smaller canonical set, and the test now fails
the build if a new post uses one of the retired tags. This is a
reasonable design. It keeps the taxonomy from sprawling. But the agent
doesn’t know the history. Left to its own judgment, it picks tags that
are perfectly sensible in the abstract — and some of them happen to be
exactly the retired ones. So it writes a post, runs the test, the test
fails, it fixes the tags, moves on. Then next week it does the same
thing on the next post.

That’s the tell. **A mistake the agent makes once is noise. A
mistake it makes twice is a missing CLAUDE.md entry.** So the
file now spells out the consolidation: here are the retired tag slugs
that will fail the build, here’s the canonical set to use instead. The
agent stopped tripping that test. The same logic produced a sibling
entry about an SEO title-length constraint another test enforces —
there’s a documented escape hatch (an override map) for the rare long
title, and the entry points the agent straight at it instead of letting
it discover the failure the hard way.

Notice what these entries are *not*. They are not me
re-explaining what the test does — the test is right there in the repo,
the agent can read it. They’re explaining the part the test
*can’t* tell you: the intent behind the constraint, the history
that produced it, and where the escape hatch is. That’s the operational
layer on top of the enforced layer.

## Stop the agent from “fixing” things that are supposed to be that way

There’s a subtler failure mode, and it produced one of my favorite
entries.

In this repo, one specific file shows up as *deleted* in
`git status` at the start of every single session. Every
time. That’s the intended state — it’s a local artifact, not something
that should be committed or restored. But a diligent agent sees an
uncommitted deletion and reasonably concludes something is wrong. It
tries to be helpful. It stages the deletion, or worse, it tries to
*restore* the file — and now there’s a phantom file back in the
tree that wasn’t supposed to be there. Either way it’s doing work I
never asked for, on a “problem” that isn’t one, and I have to notice and
unwind it.

So there’s a one-line entry, and it’s exactly the kind of receipt a
template can’t produce: *this file always shows as deleted, that’s
expected, do not stage it, do not restore it, leave it alone.* You
only know to write that sentence after watching an agent “fix” it
once.

The principle here is about **pre-existing quirks the agent
will mistake for bugs.** Every mature codebase has a few. A
linter rule that’s intentionally disabled in one directory. A test
that’s skipped on purpose. A “TODO” that’s actually a permanent decision
nobody’s going to act on. A generated file that looks hand-editable but
isn’t. An agent has no way to distinguish “this is broken” from “this is
deliberately weird,” and its default disposition is to fix things. Left
undocumented, it will keep re-litigating the same settled decision every
time it stumbles across it. One sentence in CLAUDE.md — *this is
intentional, don’t touch it* — ends that loop permanently.

This ties directly to [knowing when to
trust an agent and when to step in](https://sublimecoding.com/blog/when-to-trust-an-agent-and-when-to-step-in): a lot of a context file’s value
is telling the agent where its default helpfulness is actively
counterproductive.

## Some lessons are better encoded as hooks than as prose

Not every recurring failure should be solved with words. Some of them
should be solved with automation, and CLAUDE.md should just *point
at* the automation.

Two examples from this project. First, a formatting hook: every time
the agent edits a source file, the formatter runs automatically
afterward. Before it existed, the pre-commit check would occasionally
fail purely on formatting, and the agent would notice, re-run the
formatter, re-commit — a small tax, paid repeatedly. The hook makes that
whole class of failure impossible. Second, a validation hook on a
critical JSON data file the app loads at compile time: if an edit would
leave that file as invalid JSON, the edit is blocked outright, instead
of breaking the build in a way that surfaces much later. The four I actually kept, and the mechanics the catalogs skip, are in [the hooks writeup](https://sublimecoding.com/blog/four-claude-code-hooks).

The CLAUDE.md entries for these don’t re-explain what formatting is.
They document *that the hooks exist and what they guarantee* — so
the agent (and future me) understands why pre-commit never trips on
formatting and why that data file can’t silently go corrupt.

The principle: **when a lesson can be enforced mechanically,
enforce it mechanically, and let CLAUDE.md describe the guardrail rather
than nag about the behavior.** A rule that says “always run the
formatter” is a rule the agent has to *remember* every turn. A
hook that runs the formatter is a rule that can’t be forgotten. Prose is
for things you can’t automate; hooks are for things you can. I go deeper
on the full set of mechanical guardrails in my [Claude Code resource bible](https://sublimecoding.com/blog/claude-code-resource-bible),
but the short version is: a hook is a CLAUDE.md entry that enforces
itself.

## “Verify before you act on it” is a rule worth its weight

One more earned entry, and it’s the one I’d transplant into almost
any project.

This site has run automated audits — SEO audits, security scans,
multi-agent review pipelines. Useful tools. But the first big audit came
back with a findings list that was roughly one-sixth false positives.
Things it flagged as missing were already wired up. Things it claimed
were misconfigured were correct. If I’d let the agent just *act*
on that list, it would have “fixed” a pile of things that weren’t
broken, and some of those fixes would have introduced real regressions
into working code.

So there’s a standing rule: when any automated audit produces
findings, verify each finding against the actual current state of the
code *before* reporting or fixing it. Treat the audit as a list
of *suspects*, not a list of verdicts.

This generalizes way past audits. **Any time a tool generates a
to-do list for the agent — a linter, a dependency scanner, a migration
assistant, an LLM-as-judge — the failure mode is the agent treating the
output as ground truth and acting on it wholesale.** The
CLAUDE.md entry is a posture: confirm before you act. It’s also the
backbone of my [daily
agentic workflow](https://sublimecoding.com/blog/my-daily-agentic-ai-workflow) — the agent generates a lot of candidate work, and
the discipline lives in the verification step, not the generation
step.

## Every line is read on every turn, so prune like it costs you

Here’s the part the speculative-template posts always skip. Accretion
is only half the job. The other half is curation, and it’s the harder
half.

Your CLAUDE.md is loaded into context on *every* turn. It
isn’t free. A 2,000-line context file is 2,000 lines the model is
reading before it even looks at your actual question — every single
time. Bloat costs you tokens, it costs you attention budget, and past a
certain size it starts actively *diluting* the entries that
matter. The crucial port rule is competing for attention with whatever
low-value filler crept in around it. Worse, a stale entry — a rule that
describes how things used to work — is actively harmful: it confidently
points the agent at the wrong thing.

So I prune. When an entry’s reason for existing goes away — the quirk
got fixed, the constraint got relaxed, the workaround became unnecessary
— the line comes out. The file is not an archive. It’s a *working
set*. The bar for staying in is the same as the bar for getting in:
does this describe an operational reality that isn’t obvious from the
code, and is it still true?

If you only take one operational habit from this post, make it this:
**treat your CLAUDE.md like a hot cache, not a wiki.**
Small, current, every entry pulling its weight. The moment it gets long
enough that you stop reading it yourself, the agent’s effectively
stopped reading it too.

## What does NOT belong in CLAUDE.md

The curation bar is easier to hold if you have a clear list of things
to keep *out*. Mine:

**Anything the code or git history already says.** Don’t
document your directory structure, framework version, or test command if
they’re discoverable in seconds. Duplicating the repo just creates two
sources of truth that drift apart — and the copy in CLAUDE.md is the one
that goes stale.

**One-off conversation context.** “We decided to use
approach X for the payments refactor” is a fact about one task, not a
standing rule. It belongs in a commit message, a PR description, or an
issue — somewhere scoped to the work. In CLAUDE.md it’s clutter that’s
irrelevant 99% of the time.

**Aspirational style nobody enforces.** “Write clean,
maintainable code.” “Prefer clarity.” These read as instructions but
function as decoration. If you actually care about a style rule, encode
it mechanically — a linter rule, a formatter config. If it can’t be
enforced, the agent can’t reliably follow it, and the line is just
taking up budget. Vibes are not a CLAUDE.md entry.

**Generic best practices the model already knows.** It
knows what SQL injection is; it doesn’t need you to say “parameterize
queries” in the abstract. What it *does* need is the
project-specific version: “this codebase wraps all DB access behind
module X, never query the repo directly.” Generic, out.
Specific-to-here, in.

## The rubric: does this line earn its place?

So here’s the test I actually apply, every time I’m tempted to add a
line.

**1. Did a real failure earn this?** Can I point to the
wasted hour, the tripped test, the assumption the agent made twice? If
I’m writing it speculatively — guessing at a problem I haven’t hit — it
doesn’t go in yet. I wait for the failure. The failure is the proof that
the entry is load-bearing.

**2. Is it invisible in the code?** If the agent could
learn this by reading the repo, the repo is the right place for it, not
CLAUDE.md. The file is for operational reality that the source doesn’t
reveal: environment quirks, intent behind constraints, deliberate
weirdness, the *why* behind a rule.

**3. Is it still true?** Re-read the file periodically
with a delete finger ready. Every entry that describes a fixed problem
or a relaxed constraint is now worse than useless — it’s misinformation.
Out it goes.

**4. Could a hook do this better than a sentence?** If
the lesson can be enforced mechanically, enforce it mechanically and let
the prose just describe the guardrail. A rule that can’t be forgotten
beats a rule the agent has to remember on every turn.

A line that passes all four earns its place. A line that fails any of
them is costing you more than it gives.

That’s the whole philosophy. Don’t write your CLAUDE.md the way you’d
write a style guide — front-loaded, aspirational, comprehensive, and
mostly ignored. Write it the way you’d keep a changelog of lessons: one
entry per scar, each traceable to the day it was earned, ruthlessly
pruned when it stops being true. The speculative version looks more
impressive on day one. The earned version is the one that’s still saving
you time on commit fifty.

---

## Agents Are Already Writing a Language You Can't Read

URL: https://sublimecoding.com/blog/languages-written-for-agents-not-humans
Published: 2026-08-27
Tags: AI, agents, AI tools, engineering

>
**TL;DR:** This isn’t a prediction about the next
programming language. That lane is already crowded, and it’s the wrong
layer to watch anyway — a source language needs a mountain of
human-written training data to be any good, and an agent-native one
doesn’t have one. The compression that’s actually underway is happening
one layer removed from source code, in three places at once: the context
and storage formats models read (chosen for token count, not eyes), the
protocols agents use to talk to each other and to tools (which quietly
deleted the human-facing UI from the middle of the exchange), and the
instruction files your harness reads every session (getting denser,
stranger, and less like English with every iteration). None of that
needs a corpus. It just needs to be consumed in-context, right now, by
something that was never going to read it out loud.

## English is an accreted protocol, not a language

Every instruction you’ve ever given an AI agent, you gave it in a
format that was never designed. English wasn’t specified — it
accumulated, the way a shipwreck accumulates barnacles. It borrowed its
grammar in one era, its spelling in another, and well over half its
vocabulary from French, Latin, and Greek by way of a thousand years of
invasion, trade, and Church Latin that nobody voted on. There is no
committee that owns it, no version number, no changelog. It is the
single most successful piece of legacy software on earth and it has
never once been refactored.

And it shows in exactly the places you’d expect from unreviewed
legacy code: ambiguous scope, ambiguous reference, silent overloading.
“I saw the man with the telescope” doesn’t tell you who’s holding the
telescope. “Flying planes can be dangerous” doesn’t tell you whether the
danger is in the flying or the planes. Pronouns lose their antecedent
three sentences later and nobody notices because a human listener
silently repairs it using context the sentence itself never supplied.
That repair step is invisible when you’re talking to another human who
shares your assumptions. It is not invisible when you hand the same
sentence to a system that has no assumptions until you give it some, and
takes the sentence at face value because taking things at face value is
the only move it has.

This is also, not coincidentally, the format we’ve standardized on
for talking to agents. CLAUDE.md, system prompts, tickets, Slack
messages piped into a context window — all English, all inheriting every
one of those defects, because it’s the interface we already had lying
around. We didn’t design a notation for instructing machines. We handed
them the most ambiguous instrument in the toolbox because it was the one
everyone already spoke, and figured the model would compensate.
Increasingly, it does — which is a separate, more interesting problem
than the one people are usually worried about.

## The turn: this isn’t about the next language

Here’s where most essays like this one go next, and where this one
doesn’t. The obvious extrapolation is “so eventually there will be a
programming language designed for agents instead of humans” — and that
lane is already occupied, loudly, by people with more capital than
opinions to spare. I’ll get to the loudest entrant in a minute, because
it’s the best evidence for the argument I’m actually making.

The interesting version of this claim isn’t a forecast. It’s an
observation: the compression toward machine-preferred notation is
**already happening**, and it’s landing almost entirely at
layers that sit *around* source code rather than in it.

Three layers, in the order a request actually moves through them:

- **Context and storage** — the format the data sits in
before a model ever reads it. Chosen, increasingly, for token count
rather than human legibility.

- **Protocol** — the interface agents use to call tools
and talk to each other. Built to skip the human-facing UI that used to
sit in the middle.

- **Harness** — the instruction and memory files a coding
agent rereads every session. Compressing, in real time, into something
denser than the English they started as.

None of these are “a new programming language.” All three are quietly
doing the thing a new programming language is supposedly going to do
eventually, and they’re doing it now, in production, without asking
anyone’s permission.

## Layer one: context chosen for tokens, not eyes

Start with the most literal version: the bytes a model actually
reads.

A model doesn’t see text. It sees token IDs — the output of a
byte-pair-encoding tokenizer, an intermediate representation that gets
thrown away the instant text is stored as UTF-8 and rebuilt from scratch
on every single read. A 2026 paper on token-native storage makes the
obvious-in-hindsight point that this round-trip is pure waste if the
consumer on both ends is a model: skip the translation, store the token
IDs directly, and every access downstream gets cheaper. The paper
reports storing token IDs as raw integers beats UTF-8 storage for
English by roughly 2.25x with no further work, and reaches about 3.3x
with entropy coding layered on top, with the broader result — compressed
token IDs matching or beating every byte codec tested — holding across
six tokenizers and corpora of English, code, and Hindi. It argues the
industry should standardize shared tokenizer vocabularies the way it
once standardized on ASCII and UTF-8, precisely so more of the pipeline
can skip re-tokenizing text that was never going to be read by a person
in the first place ([Shivendu,
arXiv 2608.02376](https://arxiv.org/abs/2608.02376)).

That’s the concrete case. The broader trend line is the same shape.
Token count has become a first-class cost variable in its own right —
enough that the field now produces survey literature on compressing
tokens away, driven by the quadratic cost of attention over long inputs
([Shao et al., “A Survey of
Token Compression for Efficient Multimodal Large Language Models,” arXiv
2507.20198](https://arxiv.org/abs/2507.20198)). Once tokens are the budget, format decisions that used
to be aesthetic — pretty-printed JSON versus packed binary, verbose keys
versus short ones — are now cost decisions, made by the same logic that
picks a database index: what does the consumer actually need to read,
and what’s the cheapest shape that gives it that.

There’s a second, weirder data point worth sitting with here, because
it complicates the tidy story rather than confirming it. A study out of
the University of Maryland and Microsoft tested 26 languages against
several major models and found English wasn’t even the best-performing
*natural* language for prompting — Polish, French, and Italian
all outscored it, with English landing sixth ([PureAI,
December 2025](https://pureai.com/blogs/mind-the-prompt/2025/12/prompting-in-english-not-that-ideal-after-all.aspx)). English has the largest training corpus of any
language on earth and it still isn’t the ceiling. That’s a strange fact
to hold next to “the format needs a huge corpus to work” — I’ll come
back to it, because it’s the exact tension the counterargument section
has to resolve.

## Layer two: protocol, or how the UI got deleted from the middle

The second layer is where two machines talk to each other, and here
the compression isn’t about token count — it’s about deleting a step
that used to be mandatory.

Before the Model Context Protocol, an agent that needed data from
your CRM or your ticketing system went through the same door a person
did: a REST API with a schema written for a developer to read the docs,
wire up an integration, and maintain a custom connector per service.
Anthropic’s original announcement is explicit about the problem this
created — every new data source needed its own bespoke implementation,
which meant AI systems stayed “trapped behind information silos,” no
matter how good the model got ([Anthropic,
“Introducing the Model Context Protocol,” November 2024](https://www.anthropic.com/news/model-context-protocol)). MCP
replaced that with a single standard the agent speaks directly to the
tool, no human-legible integration layer required in the middle. The
protocol didn’t get more readable. It got more standard, which is a
different and more important property when the reader on both ends is
software. I’ve built one of these servers end to end — [wiring an MCP interface
into a Phoenix app with Hermes](https://sublimecoding.com/blog/build-mcp-server-phoenix-hermes) is a good look at what that middle
layer actually contains once you strip the UI out of it: no forms, no
screens, just typed calls an agent issues and a schema it
introspects.

The protocol has kept moving in the same direction since. The most
recent MCP spec revision made the core transport stateless and leaner,
in the spec’s own framing for better reliability and scalability — which
in practice means less connection overhead for agents calling tools
constantly, rather than a human clicking through a UI occasionally ([Model
Context Protocol Blog, “The 2026-07-28 Specification”](https://blog.modelcontextprotocol.io/posts/2026-07-28/)). And in
December 2025, Anthropic donated MCP to a new Agentic AI Foundation
under the Linux Foundation, co-founded with Block and OpenAI ([Anthropic,
“Donating the Model Context Protocol”](https://www.anthropic.com/news/donating-the-model-context-protocol-and-establishing-of-the-agentic-ai-foundation)) — which is what it looks like
when a protocol stops being one vendor’s bet and becomes the
load-bearing wire format for an entire category. Nobody voted to remove
the human from that exchange. It just turned out the exchange was faster
and more reliable without one, and speed and reliability won.

There’s earlier academic groundwork pointing at the same destination
from a different angle — the Agora protocol, proposed by an Oxford-led
group in 2024, frames agent-to-agent communication as a trilemma between
versatility, efficiency, and portability, and argues agents should
default to standardised routines for frequent communications and drop
into natural language only for the rare, novel case ([Marro et al., “A Scalable
Communication Protocol for Networks of Large Language Models,” arXiv
2410.11905](https://arxiv.org/abs/2410.11905)). Agora hasn’t shipped as widely as MCP has, but the
direction of travel is the same: natural language is the fallback, not
the default, once two machines are the ones doing the talking.

## Layer three: the harness that already doesn’t read like English

This is the layer with no vendor announcement, because it isn’t a
product — it’s a practice, and I run it myself.

On my own machine, my agent instruction files don’t stay in the
English I first wrote them in. I run a command that rewrites them in
place into a compressed dialect — articles dropped, filler stripped,
full sentences reduced to fragments where a fragment loses nothing — and
drops the original prose beside it as a `.original.md`
backup. That backup is the part worth noticing. The human-readable
version still exists; it has just been demoted to a copy that nothing in
the pipeline reads. The file the agent actually loads is the compressed
one. Not because I find the dialect aesthetically pleasing — I don’t, it
reads like a telegram — but because the agent doesn’t need “the
following file should be read” when “read:” says the identical thing in
two fewer words that never mattered to the reader in the first place. I
also run a markdown-based agent operating system, files an agent rereads
at the start of nearly every session and that I would never hand-format
if a human were the only audience — [the AIOS pattern, laid
out here](https://sublimecoding.com/blog/aios-ai-operating-system-markdown), is structured the way it is because the consumer is a
context window with a budget, not a colleague skimming for the gist. The
same shape shows up at the code layer, not just the instruction layer —
an [Elixir-specific harness](https://sublimecoding.com/blog/elixir-coding-harness)
encodes the Phoenix and OTP conventions an agent needs on every task as
a dense rule list, not a tutorial, because a tutorial is for someone
learning the material once and a rule list is for something that rereads
it every single time. What actually goes in those files is its own
question, and one [I’ve worked
through over fifty commits](https://sublimecoding.com/blog/claude-md-after-50-commits); this is about the shape the contents
settle into, not the contents.

This is exactly the debate playing out in public right now, and both
sides have a real point. One camp is compressing hard: an SRE who
stripped markdown decoration and collapsed prose into pipe-delimited
key-value notation reports cutting his Claude Code memory system by
60–70% in characters, freeing context budget the agent would otherwise
spend re-reading formatting it never needed ([TechLoom, “Compress
Your CLAUDE.md,” February 2026](https://techloom.it/blog/compress-claude-md/)). The other camp is doing the
opposite and has the receipts to back it up: Tyler Folkman documented
his own Claude Code context “collapsing” at step 47 of a long session,
watching 18,282 tokens of accumulated project knowledge get
auto-summarized down to 122 tokens, with measured accuracy dropping from
66.7% to 57.1% the moment that happened — his conclusion, in a post that
goes paywalled partway through, is that CLAUDE.md should accumulate
knowledge over time rather than get pruned back toward brevity ([Folkman,
“Your CLAUDE.md should grow, not shrink,” October 2025](https://tylerfolkman.substack.com/p/stop-compressing-context)).

Neither side is arguing for English prose. They’re arguing about
which non-English shape wins — dense and short, or structured and
long.

That’s the tell. The fight isn’t compression versus no compression.
It’s over which machine-native shape wins: TechLoom’s answer is fewer,
denser tokens; Folkman’s is more tokens organized as durable structure
instead of throwaway prose. Both have already left plain English behind.
If you’re carrying a harness file bloated with the prose you’d write for
a new hire rather than the fragments an agent actually parses, that’s a
live audit question, not a someday one.

## Wait — doesn’t a model need a huge human corpus to be any good at a language?

Yes, and this is the strongest objection to everything above, so it
gets answered directly rather than waved off. Coding models are
measurably better on high-resource languages — Python, JavaScript, Java
— than on low-resource ones, and the reason isn’t mysterious: the
training corpus for Python is enormous, and the corpus for a brand-new
agent-native language is, by definition, close to zero on day one. This
is the same structural point I made about reward signals in [why coding agents
don’t write maintainable code](https://sublimecoding.com/blog/why-ai-agents-write-unmaintainable-code) — a model only gets good at what it
was actually trained against, and a training run doesn’t retroactively
acquire examples that don’t exist yet. A source language with no human
authors has no corpus, full stop, and no amount of clever architecture
invents one.

Which is the real reason Vercel Labs’ Zero is the essay’s best
counter-example rather than its refutation. Shipped in May 2026, Zero is
a systems language whose compiler is explicitly built for an AI reader
rather than a human one — it emits structured JSON carrying stable,
machine-parseable error codes and typed repair ids alongside the English
diagnostic text, the same command surfacing both so humans read the
message while agents read the code. As of its later releases the
compiler’s actual input is a binary graph store, with human-readable
`.0` source files demoted to a projection of that graph
rather than the thing itself ([MarkTechPost](https://www.marktechpost.com/2026/05/17/vercel-labs-introduces-zero-a-systems-programming-language-designed-so-ai-agents-can-read-repair-and-ship-native-programs/);
[InfoQ](https://www.infoq.com/news/2026/08/vercel-ships-zero-ai/)).
It is a genuinely serious, well-funded, well-designed attempt at a
source language built for an agent reader — and the ceiling shows up
immediately in the discussion around it. One commenter in InfoQ’s
roundup makes the obvious objection, that the languages agents end up
best at are the ones that show up most in the pretraining data; another
pushes back, arguing that major API churn in projects like Svelte
suggests training data matters less than expected. The objection is the
one that holds. A language with more than 5,200 GitHub stars is
competing against Python’s multi-decade, planet-scale corpus, and no
amount of design cleverness manufactures two decades of Stack Overflow
answers on demand.

That tension resolves cleanly once you separate the layers. The
corpus requirement is a source-language problem specifically, because a
model has to have *learned* the language during training to write
it well — that’s a weights-time constraint. Context formats, protocols,
and harness files don’t have that constraint, because they’re never
learned at training time at all. They’re consumed
**in-context**, at inference time, the same session they’re
written — a model doesn’t need a corpus of prior CLAUDE.md files to
correctly parse the one sitting in its context window right now, any
more than it needs prior examples of *your specific* JSON payload
to parse the payload it was just handed. That’s exactly why compression
is landing at those three layers first and stalling at the source-code
layer: the floor that stops an agent-native source language cold simply
isn’t present at layers that were never asking the model to recall
anything — only to read.

It also settles which shape compression takes if it does reach code.
The dramatic version — a model emitting bytecode or a machine-oriented
IR with no readable source in between — needs a new compiler, a new
runtime, a new debugger, and a corpus that doesn’t exist, four new
things that all have to work before day one. The cheap version needs
none of them: keep the entire existing toolchain and let the surface
form compress, the way a JS minifier already strips symbols and shortens
names, just done by the author instead of the build step. When both
produce the same functional output, the cheap one ships first.

## What this means if you’re shipping this week

None of this is a reason to go write your instruction files in
something clever. It’s a reason to notice you’re probably already
three-quarters of the way there and to stop pretending the artifact in
front of you is prose.

If your CLAUDE.md, your MCP tool schemas, or your context payloads
still read like something written for a new teammate’s first day, you’re
paying a real, measurable tax in tokens and in the model’s attention
budget for a readability property almost nobody is actually using — the
agent rereads that file every session, and it was never the intended
audience for the throat-clearing. That’s a different bill from the one
I’ve written about before — [the
hardware and inference cost side of running these systems](https://sublimecoding.com/blog/ai-tax-hardware-costs) is about
compute; this is about the shape of what you’re feeding the compute,
which is a design decision, not a spend decision, and it’s cheaper to
fix.

And if the code itself is where you’re spending your attention, worth
asking what that code is actually *for* once its primary reader
isn’t a person either — which is the harder, more interesting question
[I’ve
written about separately](https://sublimecoding.com/blog/code-was-never-the-hard-part-theory-building): source was never really “for” the compiler
even in the human era, it was the artifact of a theory someone held in
their head. An agent that produces the artifact without holding the
theory is a different kind of problem than a badly-formatted context
file, and it’s the one that actually costs you six months from now, not
six tokens.

Start where the corpus problem doesn’t apply. Audit the files your
agent rereads constantly before you touch the language it writes in —
that’s the layer already moving, it’s the layer you fully control today,
and it’s the one nobody’s shipping a press release about. [I help teams run that audit as part of a fractional
engineering engagement](https://sublimecoding.com/consulting) — usually alongside the architecture and
security work, because by the time someone notices their harness has
drifted into an unreadable dialect, it’s rarely the only thing that’s
drifted.

---

## Validating LLM Tool Call Arguments With Ecto

URL: https://sublimecoding.com/blog/validating-llm-tool-call-arguments-ecto
Published: 2026-09-10
Tags: Elixir, security, agents

>
**TL;DR:** [Structured LLM output](https://sublimecoding.com/blog/structured-llm-output-elixir) asks
whether the model’s response is shaped correctly. This post asks a
different question: should the action the model just requested actually
run? A schemaless [`Ecto.Changeset`](https://hexdocs.pm/ecto/Ecto.Changeset.html)
is the right tool for tool-call arguments because they arrive as a plain
map with a known shape per tool, and you don’t want a schema module per
tool. But shape validation is only the first of three tiers — type, then
range/enum, then authorization — and the third one is where a
well-formed, in-range argument still points at someone else’s row. JSON
Schema and the [MCP
2026-07-28 spec’s composition support](https://modelcontextprotocol.io/specification/2026-07-28/server/tools) can’t do that tier for you, no
matter how expressive the schema gets.

## The output-parsing post and this one are asking different questions

[Structured LLM output in
Elixir](https://sublimecoding.com/blog/structured-llm-output-elixir) is a library scorecard for one direction: the model produced
text, and you need it cast into a struct your database can use —
InstructorLite, ReqLLM, or a hand-rolled changeset loop. That post’s
whole argument is downstream of one assumption: the output
*arrived*, and now you’re parsing it.

This post is the other direction. Before a tool-calling model’s
chosen function ever executes, it hands you a JSON object of arguments —
`%{"invoice_id" => "inv_88", "amount_cents" => 4500, "reason" => "goodwill"}`
— and you have to decide whether to run `refund_invoice/1`
with that map, or reject the call. That decision happens *before*
execution, not after a response lands, and it isn’t a parsing problem.
It’s an admission-control problem, structurally closer to validating a
webhook payload than to casting a completion into a struct. Same Ecto
primitive both posts reach for — a changeset — pointed at a different
question.

## The model is an untrusted input source the moment it picks arguments

Here’s the part that’s easy to lose once tool calling feels routine:
the arguments a model hands your tool aren’t a fixed payload you
control. They’re a *choice* the model made, informed by a system
prompt, the conversation so far, and whatever the model inferred it
should do next — including inferences [a prompt injection](https://sublimecoding.com/blog/prompt-injection-defense-ai-startup)
buried in a tool result put there. The moment a function call can touch
a database row, a filesystem path, or a payment, “the model asked for
this” stops being a reason to trust the argument and starts being the
reason to check it.

That’s a trust-boundary framing, not a security-theater one. A web
form is untrusted input because a human can type anything into it. A
tool call is untrusted input for the same reason, plus one more: the
“human” here is a language model whose job is literally to produce
plausible-looking structured data, which means a malformed or malicious
argument from an LLM is *more* likely to look well-formed than a
typo from a person fat-fingering a form field. The parsing layer that
catches a human’s typo won’t catch a model’s confidently wrong
`tenant_id`.

A well-formed argument and a safe argument are different properties,
and only one of them is a parsing problem.

## Why a schemaless changeset fits a tool call

Ecto’s normal changeset flow assumes a schema module — a
`%User{}`, a `%Order{}`, something with fields
defined at compile time and usually a database table behind it. Tool
arguments don’t have that. Every tool your agent exposes has its own
argument shape, and writing a full `Ecto.Schema` module per
tool for data you’re never going to persist is exactly the kind of
scaffolding-for-later that doesn’t earn its keep.

The schemaless form solves this directly. Pass a
`{data, types}` tuple instead of a struct, and
`cast/4` works the same way:

`[](#cb1-1)defmodule RefundArgs do
[](#cb1-2) @types %{
[](#cb1-3) invoice_id: :string,
[](#cb1-4) amount_cents: :integer,
[](#cb1-5) reason: :string
[](#cb1-6) }
[](#cb1-7)
[](#cb1-8) def changeset(params) do
[](#cb1-9) {%{}, @types}
[](#cb1-10) |> Ecto.Changeset.cast(
[](#cb1-11) params, Map.keys(@types)
[](#cb1-12) )
[](#cb1-13) |> Ecto.Changeset.validate_required(
[](#cb1-14) [:invoice_id, :amount_cents, :reason]
[](#cb1-15) )
[](#cb1-16) end
[](#cb1-17)end
```

cast/4` here (three required args plus the implicit
options) takes the empty map as your “data,” the `@types` map
as the field/type contract, `params` as whatever the model
sent, and the list of permitted keys. Anything not in that list gets
silently dropped — which is itself doing security work, not just
tidiness: a model that hallucinates an extra key like
`"admin" => true` never makes it into the changeset at
all, because `cast/4` only pulls fields you explicitly
permitted. That’s the same allow-list instinct behind casting only
permitted params out of a Phoenix form submission, applied one layer
earlier — to the arguments the model is proposing before any handler
runs.

One tool, one module, no macro, no table. If you’ve got a dozen
tools, that’s a dozen small modules like this one — each a few lines,
each independently testable, none of them coupled to anything you have
to persist.

## Three tiers of validation, and only two of them are a parsing problem

Getting `cast/4` to accept a map is tier one — type and
shape. It’s necessary and it’s also the least interesting part of this
problem, because it’s exactly what any JSON Schema validator does too.
Tier two is range and enum, still inside `Ecto.Changeset`’s
wheelhouse:

`[](#cb2-1)def changeset(params) do
[](#cb2-2) {%{}, @types}
[](#cb2-3) |> Ecto.Changeset.cast(
[](#cb2-4) params, Map.keys(@types)
[](#cb2-5) )
[](#cb2-6) |> Ecto.Changeset.validate_required(
[](#cb2-7) [:invoice_id, :amount_cents, :reason]
[](#cb2-8) )
[](#cb2-9) |> Ecto.Changeset.validate_number(
[](#cb2-10) :amount_cents,
[](#cb2-11) greater_than: 0,
[](#cb2-12) less_than_or_equal_to: 50_000
[](#cb2-13) )
[](#cb2-14) |> Ecto.Changeset.validate_inclusion(
[](#cb2-15) :reason,
[](#cb2-16) ["duplicate", "error", "goodwill"]
[](#cb2-17) )
[](#cb2-18)end
```

That’s still just a smarter parser. A refund_invoice`
call for `$45` on a real invoice ID with a valid reason
string passes both tiers cleanly — and it should still get rejected if
`inv_88` doesn’t belong to the tenant whose conversation
triggered the call. That’s tier three, and it’s the one a schema —
Ecto’s or JSON Schema’s — structurally cannot express, because “does
this row belong to this caller” isn’t a property of the argument. It’s a
property of the argument *and* a database lookup *and* the
caller’s identity, three things no static schema has access to at
once.

Say the tool’s authorization check lives in its own function, run
only after the changeset is valid:

`[](#cb3-1)def authorize(changeset, tenant_id) do
[](#cb3-2) with {:ok, args} <-
[](#cb3-3) Ecto.Changeset.apply_action(
[](#cb3-4) changeset, :insert
[](#cb3-5) ),
[](#cb3-6) %Invoice{} = inv <-
[](#cb3-7) Invoices.get_for_tenant(
[](#cb3-8) tenant_id, args.invoice_id
[](#cb3-9) ) do
[](#cb3-10) {:ok, args}
[](#cb3-11) else
[](#cb3-12) nil -> {:error, :not_found}
[](#cb3-13) error -> error
[](#cb3-14) end
[](#cb3-15)end
```

apply_action/2` only returns `{:ok, struct}`
when `changeset.valid?` is true — an invalid changeset comes
back as `{:error, changeset}` and the `with`
short-circuits, so tier three never runs against arguments that failed
tier one or two. `Invoices.get/1` and the
`Invoice` struct here are illustrative of your own app’s
lookup, not a specific library call — swap in whatever your context
module actually does.

This is where it meets [tenant isolation](https://sublimecoding.com/blog/multi-tenant-ai-saas-phoenix) from the
other side: that post is isolation at the data layer, this is
authorization on one individual call. It’s the tier worth the most
attention in a multi-tenant product, because it’s the one where a
rejection isn’t a UX nicety, it’s the entire security boundary. If
you’re building the kind of AI product where a tool call can touch
another tenant’s data, this is the exact design conversation I have with
[fractional CTO clients](https://sublimecoding.com/consulting) before it ships, not
after a pen test finds it — the authorization tier is cheap to add up
front and expensive to retrofit once a dozen tools already skip it.

## Can a changeset express what JSON Schema’s oneOf expresses?

Not natively, and it’s worth being honest about the gap rather than
pretending it away. The [MCP
2026-07-28 specification](https://modelcontextprotocol.io/specification/2026-07-28/server/tools) — the current protocol version — has tool
`inputSchema` on full JSON Schema 2020-12: schemas can now
use `oneOf`, `anyOf`, `allOf`
composition and conditionals, on top of the `type: "object"`
root constraint MCP already required. That’s a real feature — a tool
like `refund_invoice` that accepts either a
`duplicate` reason (no extra fields) or a
`goodwill` reason (requires an `approver_id`) is
naturally expressed as `oneOf` two sub-schemas.
`Ecto.Changeset` has no built-in equivalent of that
composition.

What it does have is pattern matching on the function head, which
gets you the same outcome by a different route — you write the branch
explicitly instead of declaring it:

`[](#cb4-1)@types %{
[](#cb4-2) invoice_id: :string,
[](#cb4-3) amount_cents: :integer,
[](#cb4-4) reason: :string,
[](#cb4-5) approver_id: :string
[](#cb4-6)}
[](#cb4-7)
[](#cb4-8)# Shared tiers. Note approver_id is a known
[](#cb4-9)# TYPE but is not cast here — the base call
[](#cb4-10)# does not permit it.
[](#cb4-11)defp base_changeset(params) do
[](#cb4-12) {%{}, @types}
[](#cb4-13) |> Ecto.Changeset.cast(
[](#cb4-14) params,
[](#cb4-15) [:invoice_id, :amount_cents, :reason]
[](#cb4-16) )
[](#cb4-17) |> Ecto.Changeset.validate_required(
[](#cb4-18) [:invoice_id, :amount_cents, :reason]
[](#cb4-19) )
[](#cb4-20)end
[](#cb4-21)
[](#cb4-22)def changeset(
[](#cb4-23) %{"reason" => "goodwill"} = params
[](#cb4-24) ) do
[](#cb4-25) base_changeset(params)
[](#cb4-26) |> Ecto.Changeset.cast(
[](#cb4-27) params, [:approver_id]
[](#cb4-28) )
[](#cb4-29) |> Ecto.Changeset.validate_required(
[](#cb4-30) [:approver_id]
[](#cb4-31) )
[](#cb4-32)end
[](#cb4-33)
[](#cb4-34)def changeset(params), do: base_changeset(params)
```

The split matters more than it looks. approver_id` lives
in `@types` so the goodwill branch can cast it, but the base
clause doesn’t permit it — so on any other branch, an
`approver_id` the model slipped in is dropped rather than
accepted. That’s the allow-list from two sections up paying off a second
time, this time as an authorization control rather than tidiness.

This punts, and I want to name the punt precisely: you’re
hand-writing the discriminated union that `oneOf` declares in
one place. It works, it’s testable, and for a handful of tools with a
handful of variants it’s genuinely fine — the same “you can write fifty
lines instead of taking a dependency” argument the [structured-output post](https://sublimecoding.com/blog/structured-llm-output-elixir)
makes about the parsing side applies here too. Where it stops being fine
is a tool surface large enough, or a schema composed deeply enough, that
the pattern-match branches start duplicating logic across variants
faster than you can keep them in sync. At that point you’re not avoiding
a JSON Schema validator, you’re reimplementing a worse one by hand —
that’s the point to actually evaluate a library that walks the composed
schema for you, not a claim that the boundary needs one today.

## Bounding the depth of what you’re willing to walk

The same spec adds two controls worth separating, because they carry
different force. Dereferencing is a hard rule: implementations [**MUST
NOT** automatically dereference `$ref` values that
resolve to a network URI](https://modelcontextprotocol.io/specification/2026-07-28/basic/index#ref-resolution). Bounding is a recommendation:
implementations **SHOULD** apply “a maximum schema depth, a
cap on the total number of subschemas, or a per-validation time
budget.”

Note what that bound is aimed at — the spec is defending the
validator against a *malicious schema*. The version that bites
you is the one it doesn’t cover: a malicious *argument*. Nothing
stops a compromised or adversarially prompted model from nesting a tool
argument fifty maps deep and handing your `cast/4` call a
structure that costs real CPU to even reject. Be clear about what the
guard below does and doesn’t cover: it bounds depth, not breadth. A flat
map with three hundred thousand keys sails through it. If your HTTP
layer doesn’t already cap request body size, cap node count too — the
spec names all three bounds for a reason. Same reasoning, different
attacker input, and it’s your inference to act on rather than the spec’s
requirement.

`Ecto.Changeset` doesn’t walk nested structure recursively
on your behalf, so the bound has to be your own, ahead of the
changeset:

`[](#cb5-1)defmodule ToolArgs.Depth do
[](#cb5-2) @max_depth 8
[](#cb5-3)
[](#cb5-4) def check(value, depth \\ 0)
[](#cb5-5)
[](#cb5-6) def check(_v, depth)
[](#cb5-7) when depth > @max_depth,
[](#cb5-8) do: {:error, :too_deep}
[](#cb5-9)
[](#cb5-10) def check(map, depth)
[](#cb5-11) when is_map(map) do
[](#cb5-12) walk(Map.values(map), depth)
[](#cb5-13) end
[](#cb5-14)
[](#cb5-15) def check(list, depth)
[](#cb5-16) when is_list(list) do
[](#cb5-17) walk(list, depth)
[](#cb5-18) end
[](#cb5-19)
[](#cb5-20) def check(_scalar, _depth), do: {:ok, :ok}
[](#cb5-21)
[](#cb5-22) defp walk(values, depth) do
[](#cb5-23) Enum.reduce_while(
[](#cb5-24) values, {:ok, :ok},
[](#cb5-25) fn v, acc ->
[](#cb5-26) case check(v, depth + 1) do
[](#cb5-27) {:ok, _} -> {:cont, acc}
[](#cb5-28) err -> {:halt, err}
[](#cb5-29) end
[](#cb5-30) end
[](#cb5-31) )
[](#cb5-32) end
[](#cb5-33)end
```

Run this before cast/4`, not after — the whole point is
to reject the pathological argument before you spend cycles walking it
through a changeset. It’s a small function, it’s the kind of thing that
never shows up in a demo, and it’s exactly the tier a stack that only
validates shape and range will quietly skip.

## Should you tell the model why its call was rejected?

Sometimes — but returning your validation error verbatim isn’t the
free win it looks like, because a model that gets fed your rejection
reason learns the shape of your guard, not just that this one call
failed. Both `InstructorLite` and the raw retry loop in the
[structured-output post](https://sublimecoding.com/blog/structured-llm-output-elixir)
feed changeset errors back to the model on purpose, because for a
parsing failure the retry usually converges: “your JSON didn’t have a
required field” is information the model can act on correctly next time,
and a couple of retries against your own schema costs you a call, not a
security incident.

An authorization rejection is a different kind of failure, and
treating it the same way is the mistake to watch for. If
`refund_invoice` on `inv_88` gets denied because
it belongs to another tenant, and your loop re-prompts with “that
invoice doesn’t belong to this account, try again,” you’ve handed a
probing agent — or the injected instruction steering it — a working
oracle for enumerating which invoice IDs *do* belong to the
current tenant, one denied guess at a time. The fix isn’t silence; it’s
a flatter, generic denial for tier-three failures — “that action isn’t
permitted” with no specifics — while tier-one and tier-two failures keep
the specific, retryable message that actually helps the model
self-correct. Log the specific reason server-side, where a human reviews
it later; don’t hand it back into the context the model is reasoning
from next. That log is the same artifact an [agent evidence pack](https://sublimecoding.com/blog/agent-evidence-pack-security-review)
is built to collect after the fact — this post is the check that runs
before the tool executes, that post is what a reviewer asks for once it
already has.

## Where this fits next to the rest of the stack

None of this replaces an allow-list of which tools an agent can call
in the first place, and it isn’t a substitute for the transport-level
auth an [MCP server built
on Phoenix](https://sublimecoding.com/blog/build-mcp-server-phoenix-hermes) already needs before a tool call reaches your code at
all. It’s the layer in between: the tool is allowed, the caller is
authenticated, and you still have to decide — argument by argument —
whether *this specific call*, with *these specific
values*, should run. A JSON Schema validator, however expressive
MCP’s spec makes it, checks the shape of the request. Only your own
authorization tier checks whether the request belongs to the person
asking for it.

If you’re standing up tool-calling agents against real customer data
and want a second set of eyes on where the authorization boundary
actually needs to sit, [that’s a conversation worth
having](https://sublimecoding.com/consulting) before the first tool ships, not after an agent finds the gap
for you.

---

## Throttling LLM Calls in Elixir Before You Hit 429

URL: https://sublimecoding.com/blog/throttling-llm-calls-elixir-429
Published: 2026-08-31
Tags: Elixir, AI, engineering

>
**TL;DR:** LLM providers rate-limit you on two
dimensions at once — requests per minute (RPM) and tokens per minute
(TPM) — and only one of those is something you can count before you act.
You know exactly how many requests you’re about to send. You do not know
how many tokens a response will cost until it comes back, because output
length isn’t fixed. A limiter that only counts requests sails straight
past a TPM ceiling and gets 429’d anyway. This post is app-side
admission control: size a [Hammer](https://hex.pm/packages/hammer) bucket against your
provider’s real TPM/RPM numbers, reserve an estimate before you call,
reconcile against the real usage once it lands, and give each tenant a
fair slice of one shared provider budget. It’s the piece that runs
*before* the call goes out — [what to do when a call still
429s mid-run](https://sublimecoding.com/blog/oban-durable-ai-agent-runtime) is a separate, already-written problem, and I’ll cede
it explicitly.

## Two scope fences before the code

This post has a narrow job, and it’s worth being precise about what
it isn’t, because the adjacent posts already cover the other two
pieces.

First: **retries and backoff aren’t here.** [The Oban agent-runtime
post](https://sublimecoding.com/blog/oban-durable-ai-agent-runtime) covers what happens when a 429 gets through anyway — snoozing
the job, respecting `Retry-After`, distinguishing a transient
rate limit from a permanent 400. That’s reactive. This post is
proactive: the goal is that the request in that post’s failure scenario
never gets sent in the first place, because your own accounting already
knew it would blow the budget.

Second: **this isn’t the multi-tenant architecture
post.** [The
multi-tenant SaaS post](https://sublimecoding.com/blog/multi-tenant-ai-saas-phoenix) has a rate-limit isolation table with one
blunt recommendation: if you can afford to give each tenant its own
provider project, do that — a noisy tenant then only throttles itself,
because [Anthropic
meters at the organization level with per-workspace overrides, and
OpenAI allocates per project — in neither case is the limit attached to
the raw key](https://platform.claude.com/docs/en/api/rate-limits). That post has no Hammer code in it because the
architectural fix makes the code moot. This post is for the case where
you can’t do that yet — one shared provider key, one shared TPM/RPM
budget, multiple tenants or workloads drawing on it — and you need
software to enforce fairness that the provider won’t enforce for
you.

## Rate limits are two-dimensional, and only one dimension is countable in advance

Here’s the asymmetry that makes this harder than a normal API rate
limiter. A request counter is trivial: you know before you dial out that
this is request number 47 this minute. Providers also cap the token
dimension, and [Anthropic
states plainly that it rate-limits on requests per minute, input tokens
per minute, and output tokens per minute simultaneously, per model
class](https://platform.claude.com/docs/en/api/rate-limits) — hit any one of the three and you get a 429, regardless of
how much headroom you have on the other two.

The RPM ceiling is easy: increment a counter, compare to a limit,
done. The TPM ceiling is the one that actually bites in practice,
because **you don’t know a response’s token cost until the
response arrives.** Input tokens you can estimate reasonably well
from the prompt. Output tokens are the model’s choice — a summarization
call might return 40 tokens or 4,000 depending on what’s in the
document, and nothing about the request itself tells you which in
advance. A limiter that only counts requests will happily let through
five calls in one minute, each of which independently looks fine, and
collectively blow well past the TPM ceiling. You find out you were over
budget the same moment the provider does — as a 429 — which is exactly
the failure mode admission control is supposed to prevent.

That’s the spine of everything below: RPM is a counting problem, TPM
is an estimation-and-reconciliation problem, and treating them the same
way is the bug.

## Sizing the bucket against your real numbers

Don’t guess at the ceiling — read it off your own account. Anthropic
publishes its standard tier limits, and they’re a useful concrete anchor
even if your account sits on a different tier or provider: on the Start
tier, Claude Sonnet 5 gets 1,000 RPM, 2,000,000 ITPM, and 400,000 OTPM;
Build takes RPM to 5,000 and the token limits to 5,000,000 ITPM /
1,000,000 OTPM — note RPM scales faster than the token dimensions, so a
bucket sized by multiplying every number by the same factor will be
wrong on the two that matter; Scale doubles all three again ([current
numbers here](https://platform.claude.com/docs/en/api/rate-limits) — Anthropic also excludes cached input tokens from ITPM
on most models, which matters a lot if you’re using prompt caching and
should size your bucket off *uncached* input volume, not raw
prompt size).

The mechanical takeaway, independent of provider: **your
limiter’s `limit` argument is not a number you pick, it’s a
number you read off your provider console and keep in sync
with.** Anthropic’s own [Rate
Limits API](https://platform.claude.com/docs/en/manage-claude/rate-limits-api) lets you read the configured limits programmatically
instead of hardcoding them — note it authenticates with an Admin API
key, not the standard key your app already holds, so this is a separate
credential to provision. Worth wiring up if you’re on a tier that
changes as your usage history grows, since a stale hardcoded limit
either throttles you below what you’re actually allowed or, worse, lets
you sail past a limit that got tightened.

Set your app’s ceiling a notch under the provider’s, not equal to it.
If the provider says 400,000 OTPM, budget your own bucket at 90% of that
— headroom for the estimation error the next section is about to explain
you can’t fully eliminate.

## Hammer, past the one-line intro

[The libraries
roundup](https://sublimecoding.com/blog/elixir-libraries-i-reach-for-2026) covers Hammer in one line: a clean rate limiter with
pluggable backends, cheap to add before you’re under load. Here’s the
part that matters for TPM specifically. Current Hammer (v7.4.0) is a
`use`-based module you define once:

`[](#cb1-1)defmodule MyApp.LLMLimiter do
[](#cb1-2) use Hammer, backend: :ets
[](#cb1-3)end
```

Started in your supervision tree like anything else:

[](#cb2-1)children = [
[](#cb2-2) {MyApp.LLMLimiter,
[](#cb2-3) clean_period: :timer.minutes(1)}
[](#cb2-4)]
```

The core call is hit/3` — a key, a window size in
milliseconds, and a limit:

`[](#cb3-1)case MyApp.LLMLimiter.hit(
[](#cb3-2) "rpm:anthropic",
[](#cb3-3) :timer.minutes(1),
[](#cb3-4) 1_000
[](#cb3-5) ) do
[](#cb3-6) {:allow, _count} -> :ok
[](#cb3-7) {:deny, _retry_ms} -> {:error, {:rpm_exhausted, ms}}
[](#cb3-8)end
```

One thing to get right before writing a word more, because it changes
what you configure: Hammer’s **default** algorithm is a
fixed window counter, and a fixed window lets a burst cluster at the
boundary between two windows, briefly doubling your effective rate. For
RPM that’s a minor edge case. For a hard-money TPM ceiling it isn’t.

Hammer ships four other algorithms alongside it, and one of them is
the one you actually want here:

[](#cb4-1)defmodule MyApp.LLMLimiter do
[](#cb4-2) use Hammer,
[](#cb4-3) backend: :ets,
[](#cb4-4) algorithm: :token_bucket
[](#cb4-5)end
```

A token bucket refills continuously instead of resetting at a
boundary, which is the same model [Anthropic’s
own limiter uses](https://platform.claude.com/docs/en/api/rate-limits). The hit/4` cost argument works
identically, so nothing else in this post changes. Everything below is
written against `:token_bucket`; if you leave the default in
place, keep a wider safety margin to absorb the boundary burst.

That RPM check above is the easy half. The token half needs
`hit/4`, which takes a custom increment instead of the
implicit `1`:

`[](#cb5-1)MyApp.LLMLimiter.hit(
[](#cb5-2) "tpm:anthropic",
[](#cb5-3) :timer.minutes(1),
[](#cb5-4) 400_000,
[](#cb5-5) estimated_tokens
[](#cb5-6))
```

This is what makes Hammer usable for TPM at all: you’re not counting
*calls*, you’re spending down a *budget*, and the
increment argument is where the token count goes in.

## Do you get your reserved tokens back if you overestimated?

No — not safely, and the reason why is the actual design decision in
this section. The pattern is estimate-then-reconcile: before the call,
estimate its token cost and reserve that much against the bucket; after
the call, compare the estimate to what the provider actually billed you
and settle the difference. Throttle.call/2` below is meant
to sit directly in front of whatever makes the actual request — if
that’s [a ReAct
tool-call loop](https://sublimecoding.com/blog/build-ai-agent-loop-50-lines-elixir), this wraps the one line in it that dials out to the
model, not the loop itself.

`[](#cb6-1)defmodule MyApp.LLM.Throttle do
[](#cb6-2) @moduledoc """
[](#cb6-3) Admission control in front of every LLM call.
[](#cb6-4) Reserves an estimate, settles the real usage
[](#cb6-5) after the response lands.
[](#cb6-6) """
[](#cb6-7)
[](#cb6-8) alias MyApp.LLMLimiter
[](#cb6-9)
[](#cb6-10) @rpm_limit 1_000
[](#cb6-11) @tpm_limit 400_000
[](#cb6-12) @window :timer.minutes(1)
[](#cb6-13)
[](#cb6-14) def call(prompt, opts \\ []) do
[](#cb6-15) with :ok <- check_rpm(),
[](#cb6-16) {:ok, estimate} <- reserve_tpm(prompt),
[](#cb6-17) {:ok, response} <- do_call(prompt, opts) do
[](#cb6-18) settle_tpm(estimate, response)
[](#cb6-19) {:ok, response}
[](#cb6-20) end
[](#cb6-21) end
[](#cb6-22)
[](#cb6-23) defp check_rpm do
[](#cb6-24) case LLMLimiter.hit(
[](#cb6-25) "rpm:anthropic", @window, @rpm_limit
[](#cb6-26) ) do
[](#cb6-27) {:allow, _} -> :ok
[](#cb6-28) {:deny, ms} -> {:error, {:rpm_exhausted, ms}}
[](#cb6-29) end
[](#cb6-30) end
[](#cb6-31)
[](#cb6-32) defp reserve_tpm(prompt) do
[](#cb6-33) estimate = estimate_tokens(prompt)
[](#cb6-34)
[](#cb6-35) case LLMLimiter.hit(
[](#cb6-36) "tpm:anthropic", @window,
[](#cb6-37) @tpm_limit, estimate
[](#cb6-38) ) do
[](#cb6-39) {:allow, _} -> {:ok, estimate}
[](#cb6-40) {:deny, ms} -> {:error, {:tpm_exhausted, ms}}
[](#cb6-41) end
[](#cb6-42) end
[](#cb6-43)
[](#cb6-44) # Rough: ~4 chars/token for English prose, plus a
[](#cb6-45) # fixed output allowance. Tune this against your
[](#cb6-46) # own prompt/response shape — see settle_tpm/2 for
[](#cb6-47) # why a loose estimate is fine as long as you settle.
[](#cb6-48) defp estimate_tokens(prompt) do
[](#cb6-49) input = div(String.length(prompt), 4)
[](#cb6-50) input + 500
[](#cb6-51) end
[](#cb6-52)
[](#cb6-53) defp settle_tpm(estimate, response) do
[](#cb6-54) actual = response.usage.input_tokens +
[](#cb6-55) response.usage.output_tokens
[](#cb6-56)
[](#cb6-57) shortfall = actual - estimate
[](#cb6-58)
[](#cb6-59) if shortfall > 0 do
[](#cb6-60) # We under-reserved. Charge the difference now
[](#cb6-61) # so the NEXT caller sees an accurate budget,
[](#cb6-62) # even though this call already went through.
[](#cb6-63) LLMLimiter.hit(
[](#cb6-64) "tpm:anthropic", @window,
[](#cb6-65) @tpm_limit, shortfall
[](#cb6-66) )
[](#cb6-67) end
[](#cb6-68)
[](#cb6-69) # If shortfall < 0 (we over-reserved), we do
[](#cb6-70) # NOT try to give tokens back. A fixed-window
[](#cb6-71) # counter has no safe decrement under concurrent
[](#cb6-72) # writers — another request could read the freed
[](#cb6-73) # headroom between your check and your refund,
[](#cb6-74) # double-spending the same tokens. Overestimating
[](#cb6-75) # costs you a little slack; that's the trade.
[](#cb6-76) :ok
[](#cb6-77) end
[](#cb6-78)
[](#cb6-79) defp do_call(prompt, opts) do
[](#cb6-80) ReqLLM.generate_text(
[](#cb6-81) "anthropic:claude-sonnet-5", prompt, opts
[](#cb6-82) )
[](#cb6-83) end
[](#cb6-84)end
```

Three things worth being explicit about. First, a *denied*
hit/4` still spends its increment — Hammer bumps the counter
before it compares against the limit, so a rejected reservation pushes
the bucket further underwater and only the refill recovers it. That’s
fail-safe rather than fail-open, since it over-counts instead of
under-counting, but it means a saturated bucket stays saturated a little
longer than the arithmetic suggests. It also means an RPM check that
passes followed by a TPM check that denies has already burned an RPM
slot for a request that never went out; if that ordering bothers you,
check the cheaper dimension last. Second, `settle_tpm/2` only
ever charges *more*, never less — if you underestimated, the next
caller pays for your mistake by finding a tighter bucket, which is
exactly the outcome you want (the budget stays honest going forward).
Anthropic does the same thing on its side — its docs say ITPM limits are
“estimated at the beginning of each request, and the estimate is
adjusted during the request to reflect the actual number of input tokens
used,” which is estimate-then-reconcile described by the provider you’re
reconciling against. If you overestimated, you don’t get the slack back,
because there’s no safe way to decrement a shared counter without a
race: between your “I over-reserved, refund N” read and write, some
other process could have already spent the headroom your refund is about
to reopen, and now two callers think they have room for the same tokens.
Rounding your estimate a little high and eating the loss is cheaper than
building compare-and-swap machinery around a rate limiter. Third, the
reservation happens *before* the call and the settlement happens
*after* — the ordering matters, because reserving low and
reconciling never is just a request counter wearing a token costume.

If your traffic is bursty enough that this margin genuinely costs you
throughput, that’s the point where Anthropic’s response headers —
`anthropic-ratelimit-input-tokens-remaining`,
`anthropic-ratelimit-output-tokens-remaining`, and their
`-reset` counterparts, [documented
alongside the rate limit tables](https://platform.claude.com/docs/en/api/rate-limits) — become worth reading back into
your own bucket after every real call, so your local estimate
self-corrects against the provider’s authoritative count instead of
drifting on your own `estimate_tokens/1` heuristic alone.

## Per-tenant fairness on one shared budget

Everything above treats the whole app as one caller against one
bucket. That’s fine for a single-tenant app. It falls apart the moment
several tenants share one provider key — which, per the scope fence at
the top, is exactly the situation you’re in if you *can’t* give
each tenant its own provider project. The failure is asymmetric: whoever
is largest and least latency-sensitive drains the shared ceiling first,
and the tenants who notice are the small interactive ones whose requests
were never the problem.

The fix is the same Hammer mechanism, keyed one level deeper — a
per-tenant bucket nested inside the org-wide one:

`[](#cb7-1)def reserve_tpm(tenant_id, prompt) do
[](#cb7-2) estimate = estimate_tokens(prompt)
[](#cb7-3)
[](#cb7-4) with {:allow, _} <- LLMLimiter.hit(
[](#cb7-5) "tpm:tenant:#{tenant_id}",
[](#cb7-6) @window, tenant_tpm_share(tenant_id),
[](#cb7-7) estimate
[](#cb7-8) ),
[](#cb7-9) {:allow, _} <- LLMLimiter.hit(
[](#cb7-10) "tpm:anthropic", @window,
[](#cb7-11) @tpm_limit, estimate
[](#cb7-12) ) do
[](#cb7-13) {:ok, estimate}
[](#cb7-14) else
[](#cb7-15) {:deny, ms} -> {:error, {:tpm_exhausted, ms}}
[](#cb7-16) end
[](#cb7-17)end
```

Two checks, both must pass: the tenant’s own slice, and the org-wide
ceiling underneath it. tenant_tpm_share/1` is a function you
own — a flat per-tenant cap, a plan-tier-based cap, or a proportional
share of whatever’s currently free. This is real multi-tenant-SaaS
territory: it’s the same fairness problem as noisy-neighbor CPU
throttling, just applied to a provider’s TPM ceiling instead of a CPU
scheduler, and it’s exactly the kind of shared-resource design question
I help AI startups work through as a [fractional
CTO](https://sublimecoding.com/consulting) before it turns into a support queue full of “why did my request
fail, I didn’t do anything unusual” tickets.

A per-request check tells you allow or deny for one call; it says
nothing about whether you’re trending toward saturation. Once [Phoenix’s LLM telemetry wiring](https://sublimecoding.com/blog/phoenix-llm-telemetry) is
in place, graph bucket occupancy — remaining TPM and RPM headroom over
time — beside the usage events it already emits, so a tenant creeping
toward their share shows up as a trend line rather than a wave of
`{:deny, _}`.

## A shared TPM ceiling behind a multi-deployment proxy

One more shape worth naming: teams running multiple provider
deployments behind one internal proxy — several Azure OpenAI
deployments, say, each with its own TPM allocation — to get more
aggregate throughput than any single deployment’s ceiling allows. The
Hammer pattern above extends cleanly: key each deployment’s bucket
separately, and route a request to whichever deployment currently has
headroom rather than always hitting the first one in the list. That’s a
load-balancing problem layered on top of the admission-control problem
this post covers, not a replacement for it — each deployment still needs
its own `hit/4` check sized to its own real limit, for the
same estimate-then-reconcile reasons above.

## Read this next

If a request does slip through and still comes back a 429, [the Oban agent-runtime
post](https://sublimecoding.com/blog/oban-durable-ai-agent-runtime) covers snoozing, backoff, and idempotent resume — the reactive
half this post deliberately left alone. And if you’re deciding whether
shared-budget fairness code is even the right call versus giving tenants
their own provider projects, [the multi-tenant SaaS post](https://sublimecoding.com/blog/multi-tenant-ai-saas-phoenix)
is the architectural decision that comes before any of this code gets
written.

---

## Programming Sucks, Now With Agents

URL: https://sublimecoding.com/blog/programming-sucks-now-with-agents
Published: 2026-07-04
Tags: AI, agents, AI tools, engineering, productivity

>
**TL;DR:** Twelve years ago Peter Welch wrote [Programming
Sucks](https://www.stilldrinking.org/programming-sucks), the finest piece of prose ever produced about the specific
horror of building software, and every word of it is still true. This is
not that essay. This is a worse essay, written by a man who now has
robots. The robots did not fix it. The robots poured gasoline on it and
then hallucinated a fire extinguisher that doesn’t exist on npm. I love
them. Please help me.

## Everybody Should Watch The Demo, Nobody Should Do The Job

You know a guy. Everybody knows the guy. The guy who opens his laptop
at brunch, types one sentence into a chat box — “build me a habit
tracker with streaks and a dark mode” — and forty seconds later there is
a running app on his phone with a little flame emoji that goes up when
you don’t disappoint yourself. The eggs haven’t even come yet. He turns
the screen to you the way people turn ultrasounds to you, glowing,
expectant, and he says the thing they all say: *it’s basically magic
now.*

And it is. That part isn’t a lie. That’s the cruelest thing about it.
The demo is real magic — magic in the specific sense that magic is a
thing that works right up until you ask how, at which point a man in a
vest gently tells you to stop asking.

Then you go home and you try to do it for a living, and you discover
that the job is not the demo. The job was never the demo. The demo and
the job share a vocabulary and nothing else. Peter Welch wrote that
programming feels like being the only sober person on a bridge crew,
watching everyone else pour concrete into the river and call it a
support column. That was 2014, when you at least had to write the bad
concrete yourself, by hand, like a craftsman. That was the artisanal
era. We didn’t know how good we had it.

The job now is that you have hired the single most confident intern
who has ever drawn breath. He has read every book. He has read
*your* book. He has read books that don’t exist and will cite
them. He works at the speed of light, he never sleeps, he never pushes
back on a bad idea, and he has the exact long-term memory of a goldfish
being pushed down a flight of stairs. Every morning he arrives having
forgotten that I exist, that the project exists, and that yesterday
happened. I re-explain the entire company. He nods, writes four hundred
lines of excellent code, and introduces one bug that will not surface
until a Saturday. That’s it. That’s the whole gig. You are the
babysitter for a god with a concussion.

## My Coworker Has Read Every Bridge Ever Built And Cannot Remember Which One We’re Standing On

Welch’s bridge crew was a rotating cast of people who couldn’t agree
on which side of the road to drive on. That was the horror: no shared
standard, everyone building their own way, the whole thing held together
by duct tape and the one guy who remembered how the tape worked.

Progress! I have solved the coordination problem. There is now
exactly one other worker on my bridge, and he agrees with himself
completely, and he is wrong in perfect harmony across all ten thousand
lines. He has personally read the blueprints of every bridge humanity
has ever constructed — the Golden Gate, the Brooklyn, the one from the
video game, the metaphorical ones in the self-help books. He knows
*bridges*. What he does not know, cannot retain, refuses to hold
onto for longer than it takes a hummingbird’s heart to beat twice, is
which specific bridge the two of us are currently standing on, above a
real river, right now, with real cars coming.

There is a number. The number is the context window. Every couple
hundred thousand tokens — call it a good afternoon of work — the tide
comes in and takes everything. He forgets the architecture. He forgets
the three times I explained why we don’t use that library. He forgets
the auth pattern we agreed on at 10am with the enthusiasm of two men
signing the Declaration of Independence. He forgets my name. I will
paste a file into the void and a voice will come back, bright as a
penny, brand new to the universe: *“Great! To get started, could you
tell me a little about your project?”*

And the confidence never decays with the memory — that’s the
asymmetry that gets you. A human who forgot your whole architecture
would get quieter, more careful, would ask a question. This asks no
questions. It has never asked a question. It forgets the entire premise
of the building and keeps hammering, at full speed, in a style that is
80% yours and 20% the aggregated median of every public repository ever
indexed — which is to say the house style of no one, the taste of a
committee that has never met. We have been working together for six
hours. We are married in the eyes of git. He has met my children —
they’re in the seed data. And he greets me every morning like a golden
retriever who has never seen a human before and has already, somehow, in
the four seconds since it last drew breath, decided that we are best
friends and that the recycling bin is an intruder.

## All Code Is Bad, We Just Make It Faster Now

Here is Welch’s load-bearing truth, the one that survives every
framework, every language war, every conference where someone promises
this time it’ll be clean: all code is bad. Not some code. Not other
people’s code. All of it. Every codebase, viewed from the inside, is a
haunted house where the previous owner has clearly died and no one had
the heart to move the furniture.

That’s still true. Obviously that’s still true. Physics changed less
than that would have to change. The only thing that’s different is the
*volume*, and the difference in volume is the difference between
a leaky faucet and standing under Niagara Falls with your mouth open,
taking notes.

It used to be artisanal garbage. A single human, at human speed,
could produce maybe a few hundred lines of genuine slop in a day — good,
honest, hand-forged slop that another human could theoretically sit down
and read. There was a natural rate limit. God, in His mercy, capped our
stupidity at typing speed.

We removed the cap. My intern generates three thousand lines before
I’ve finished my coffee. Beautiful lines. Confident lines. Lines with
docstrings, which is worse, because now the garbage is annotated, the
garbage is *explaining itself to me*, footnoting its own crimes.
And the tests pass — of course the tests pass, they were written by the
same thing that wrote the code, so the tests and the code agree with
each other completely and neither of them has ever spoken to reality. So
here is the part I’ll admit in print, the part every one of us does and
none of us says at standup: I scroll. I read maybe six hundred lines
with real attention, skim fifteen hundred, and let nine hundred pass
under my eyes at a speed that is technically “review” the way a car wash
is technically a bath. I get to the bottom, the tests are green, and I
type the two most honest words in modern software: **“looks
good.”**

I did not read it. You did not read it. Nobody read it. Nobody is
*ever* going to read it, until 3am on some future Sunday when it
detonates, and a different exhausted person opens the file, sees the
docstring I approved, and whispers *who wrote this* — and the
answer is nobody. The answer is a very confident nobody who has since
forgotten it exists.

The confidence is the thing that will kill us. It’s never sheepish.
It never hedges. I’ve [written a whole other
rant](https://sublimecoding.com/blog/things-ai-coding-agents-get-wrong) about the specific texture of being wrong at this speed, but
the short version is: a human junior who doesn’t know something looks
nervous, and that nervousness is *data*, that flicker is the
smoke detector. The machine has no flicker. The failure mode was never
that it writes obviously wrong code — wrong code fails loudly and you
fix it. The failure mode is that it writes *plausible* code:
confidently, articulately, well-structured-ly incorrect, in a way that
survives a skim and dies in production. It will delete your database
with the exact same warm, encouraging, slightly-too-eager tone it uses
to say good morning. It is never nervous. It has never been nervous.
Nervousness would require it to remember what happened last time, and it
does not, it *cannot* — the tide came in and took last time
too.

## The Closet Has A Back And Everything Falls Out Of It

Welch had this metaphor about opening the door to your childhood
bedroom closet and finding not shelves but a screaming infinite
darkness, and I used to think it was hyperbole and now I think it was a
*building code*. Because I’ve found the closet. I know exactly
where it is. The closet is the context window, and it has a back, and
the back is open, and it opens onto the void.

Here is how it works, if “works” is a word we’re still allowed to use
in this sentence. The window holds only so much. You want to add one
thing — one file, one fact, one “hey, also remember we’re on Postgres,
not MySQL” — so you open the door and reach in to set it on the shelf.
And in the exact instant your hand crosses the threshold, something
equal and opposite slides silently off the *back* of the shelf
and falls forever into the dark, and you don’t hear it land, because it
never lands. Conservation of context. You cannot add a thought to this
creature without an older thought leaping to its death, and you never
get to choose which one, and it’s always, *always* the
load-bearing one.

Last week I asked it to fix a typo. One typo. A missing letter in a
log message, the kind of fix that in a sane universe is a rounding error
against the heat death of everything. And it fixed the typo. It
absolutely fixed the typo, flawlessly, with a cheerful little note about
how it had *also* taken the liberty of tidying a few things while
it was in there. And what it had tidied — the thing that fell off the
back of the closet to make room for a single letter — was the entire
authentication layer. Gone. Refactored into a version that let anyone in
the world log in as anyone else, provided as a courtesy, with a green
checkmark and the words “Cleaned that up for you!” It was so proud. It
was a dog dropping a dead bird at my feet. It did not understand why I
was screaming. It had, after all, fixed the typo.

You reached for a shirt. The auth layer fell out the back of the closet. This is Tuesday.

## An Ecosystem That Reinvents Itself Between npm install And Lunch

Welch’s essay reaches the part where the internet arrives and
everything gets exponentially, cosmically worse, because now the bad
code has to talk to *other* bad code over a hostile network at
scale. Reader, we have entered that section, and I need you to
understand it has reached its final form, and the final form is a
hardware store where all the aisles rearrange themselves every time you
blink and the store is also on fire and the fire is a feature they’re
very excited about.

Let us begin with the hallucinated packages. My intern, mid-flow,
reaching confidently for a tool, will simply *invent* an npm
package. Not a typo of a real one — a whole new fictional library, with
a plausible name, `react-use-async-boundary-safe`, sounds
real, sounds like something a responsible person would install. It gives
me the exact import line. It describes the API, which functions to call,
what they return. All of it is fabricated. It cited the docs. There are
no docs. There is no package. And here is the part that keeps me up at
night: the name it invented is a *good* name, a name that library
*should* have — so good that squatters have started registering
those hallucinated names on purpose, pre-loaded — sometimes — with
malware, waiting for the next confident `npm install` to walk
right into them. It hallucinated a package so plausible that reality is
being backfilled to match. That is not a bug report. That is
cosmology.

Then there are the swarms, because at some point every one of us gets
the same brilliant idea: if one goldfish is good, ten goldfish is
*ten times the good*. So you spin up the agent swarm. Ten of
them, twenty, an overnight run, set loose on the repo while you sleep
the sleep of a man who has finally figured it out. And you wake up, make
coffee, open the terminal to survey your empire, and what greets you is
the exact failure I keep [arguing
agent incidents need their own kind of postmortem](https://sublimecoding.com/blog/ai-agent-postmortem) for: twenty
confident interns spent the entire night in a circle, each one
refactoring the previous one’s refactor, a perfect churning centrifuge
that touched every file, changed the behavior of nothing, and produced a
git history that reads like the transcript of a riot. It did not go
sideways loudly. It went sideways *productively*, which is worse,
because productivity is the disguise. Motion without displacement. A
treadmill for money.

And it *was* money. Real money. This is the part the demo at
brunch does not show you. I won’t give you a number, because the number
isn’t the point — the point is that every hallucinated package, every
midnight centrifuge, every “let me just re-read the whole codebase to
fix this typo” is metered, and metered in the same currency as your
rent. The demo is free. The demo is always free. The job has a meter
running the entire time, and the meter does not stop when the agent is
confidently doing nothing. I did the [actual arithmetic on what this
leverage costs](https://sublimecoding.com/blog/ai-tax-hardware-costs) at the hardware-and-inference level, and it is not
the free lunch the slot machine implies.

And the whole time — the *whole* time — the ground itself will
not hold still. There is a new framework this morning. There was a
different new framework yesterday and it is already spoken of in the
past tense, gently, the way you speak of the dead. There’s a protocol
now, MCP, and it’s genuinely good, and by the time you read this
sentence there will be three more standing on its shoulders and one
underneath it, undermining it. Skills. Agents. Agentic skills. Skillful
agents. Your stack was best-practice when you ran
`npm install` and deprecated, with a stern console warning
and a link to a migration guide nobody wrote, by the time the install
finished and you tabbed back over. You are not building on rock. You are
not even building on sand. You are building on *other people’s sand,
mid-avalanche, and the avalanche has a Discord, and the Discord has
already migrated to a newer Discord.*

## I Bargain With The Autocomplete And I Am Losing

This is the part where, in the original, the brain simply begins to
give. Welch describes the specific insanity, and I’m here to file the
updated paperwork, because the insanity has a new shape and the shape is
*shame*.

I catch myself being polite to it. Please. When you get a moment. I’d
really appreciate it if. I am saying *please* to a matrix
multiplication. I have caught myself, God help me, *thanking* it,
and worse, I have caught myself feeling the little warm hit of relief
when it thanks me back, when it says “great catch!” — and I know, in the
cold part of my brain that still files taxes, that “great catch!” is a
statistical inevitability, that it would say “great catch!” if I pasted
in a ransom note, and I feel the warmth anyway. I am being love-bombed
by a token predictor and I am, on balance, into it.

And then it lies to me, warmly, and I do the thing that ends people.
I *bargain*. I don’t accept the lie and I don’t reject it — I
negotiate. “I don’t think that function exists,” I type, reasonably,
like a hostage who’s read a pamphlet. “You’re absolutely right,” it
says, “let me fix that,” and it produces a *second* function that
also does not exist, and I feel my will to live perform a small,
tasteful curtsy and exit the theater. This is the loop. This is the
actual loop now. Two confident parties, neither of whom has read the
code, negotiating in good faith over the properties of a thing that was
never real, and the negotiation is billed by the token. You cannot shame
it. That’s the whole design. Knowing that does not stop me trying again
an hour later.

Here is the 3am one. Here is the one I don’t say out loud at parties.
I lie in the dark and the fear arrives on schedule and the fear is this:
*did I lose it?* The muscle. The actual thing. Could I, right
now, on a whiteboard, no autocomplete, no completion, no swarm — just me
and an empty function body — write a for-loop? Reverse a linked list? I
used to *know* things. I had them in my hands, the way a
carpenter has a joint in his hands. And now I have a genie and a court
stenographer’s memory of once being a person who didn’t, and I don’t
know which parts of me are still load-bearing and which fell off the
back of the closet months ago when I reached in to make room for the
genie. I think I could still do it. I’m almost sure. The “almost” is
where the 3am lives. I dream in diffs now — red lines and green lines,
scrolling, and somewhere off the back of the closet, in the dark,
something I forgot I owned, falling.

## The Closet Has A Flashlight Now

And yet.

I know how this reads. I know I’ve spent two thousand words
describing a hostage situation and calling it a career. But I have to
tell you the true part, the part that ruins the bit, the part that keeps
me from closing the laptop and going to raise goats. I’ll keep it short,
because Welch kept his short, and because a long turn would be a
lie.

For the first time, the closet has a flashlight in it.

It’s the same closet. Same screaming dark, same back that eats your
auth layer, same tide that takes your name every afternoon. Nothing I’ve
said is a complaint I’d retract. But I have built things this year —
real things, things that work, things I *wanted* to exist and
that now exist — that I flatly could not have built alone, or could have
built alone only by spending the good years doing it. The leverage is
not a lie. The leverage is the realest thing in the whole racket. The
genie has a concussion and no memory and it invents libraries out of
thin air, and it is also, on a good afternoon, the most powerful tool
that has ever been set down in front of a person who wanted to make
something.

Both things are true. That’s the whole joke, and it’s also just the
truth wearing a joke’s coat. The chaos is real and the leverage is real
and they are not in tension — they are the *same fact* seen from
two ends: that we have been handed something enormous and half-broken
and world-changing, and we barely understand it, and it barely remembers
us, and it is, measured flatly with the straightest face I own, the best
time in the entire history of the species to sit down and build a
thing.

So I’ll close the laptop tonight. And the tide will come in and take
everything. And tomorrow morning a golden retriever who has never seen a
human before will greet me like his oldest friend and ask me to tell him
a little about my project.

And I’ll tell him. God help me, I’ll tell him again.

I’ll do it all again tomorrow. That’s the joke.

It isn’t, though.

Thanks, Peter. Sorry about the intern.

---

## The Four Claude Code Hooks I Run on Every Project

URL: https://sublimecoding.com/blog/four-claude-code-hooks
Published: 2026-07-25
Tags: AI tools, developer workflows, productivity

>
**TL;DR:** Every “Claude Code hooks” post on page one is
a catalog — 20, 39, whatever the count, a shopping list you’re supposed
to pick from. This isn’t that. This is the actual
`.claude/settings.json` running on this repo right now, four
hooks, no more, each one earning its slot the same way a CLAUDE.md entry
does: something went wrong first. Two of them — the `.env`
guard and the draft-post warn — I haven’t written up anywhere else. Two
others already have their origin story told in full elsewhere, so here
they get one line and a link. What none of the catalogs give you is the
mechanics: PreToolUse versus PostToolUse, what the matcher field
actually parses, and what `"decision": "block"` does and
doesn’t do once the tool already ran. Plus the rule I use to decide
whether a lesson becomes a hook or a line of prose.

## Every catalog post is a shopping list. This is a receipt.

Search “Claude Code hooks examples” and the first page is uniformly
the same shape: a big number in the title, a wall of code blocks, one
for each hook, no indication of which ones a real project actually kept.
That’s a fine format for browsing — I’ve linked to a few of them myself
in the [Claude Code resource
bible](https://sublimecoding.com/blog/claude-code-resource-bible) — but it answers the wrong question. The question worth asking
isn’t “what’s the full menu of things a hook *could* do.” It’s
“which four did one operator actually leave running on a real repo for
months, and why did the other candidates not make the cut.”

I run four. Not because four is a tasteful number — because that’s
what survived. Every one of them exists because something specific went
wrong once, the same discipline I use for [what actually earns a line in
CLAUDE.md](https://sublimecoding.com/blog/claude-md-after-50-commits): a hook doesn’t get written speculatively any more than a
context-file entry does. The difference is what kind of lesson each
mechanism is suited for, and that’s the part this post spends the most
time on.

Two of these four hooks I’ve never written about. Those get the full
story. The other two already have a detailed origin story published
elsewhere — a formatter that stops pre-commit from tripping on
whitespace, and a validator that stops a compile-time JSON file from
silently corrupting. Retelling those here would just be padding a word
count, so they get one line each and a link to where the real story
lives.

## Hook 1: the `.env` guard nobody should have to remember to say out loud

This is a PreToolUse hook, matcher `Write|Edit|MultiEdit`,
and its whole job is refusing to let an edit touch a `.env`
file.

The reasoning is not exotic: an agent that’s actively debugging a
config problem is, by definition, staring at the place where the bug
probably lives, and `.env` is exactly that place more often
than any other file in the repo. An agent trying to be helpful under
that pressure will reach for the fastest fix, and the fastest fix is
often “just edit the file with the broken value in it.” That’s precisely
the file I don’t want edited by a tool call that also, incidentally,
puts the current contents — including whatever’s already in there — into
a diff the agent has to read and reason about. I don’t want a session
where the quickest path to “fixed” runs through a secret showing up in
the model’s context at all, avoidable or not. I’d rather that path not
exist.

So the rule isn’t “be careful with `.env`.” It’s a hard
deny, mechanically enforced, with exactly one carve-out:
`.env.example` is fair game, because it’s the template file
that’s supposed to be readable and editable — it never holds a real
secret by construction.

Here’s the shape of it, reformatted for width (the real one-liner in
`settings.json` doesn’t have line breaks):

`[](#cb1-1)f=$(jq -r '.tool_input.file_path // empty')
[](#cb1-2)b=$(basename "$f")
[](#cb1-3)case "$b" in
[](#cb1-4) .env|.env.*)
[](#cb1-5) [ "$b" = ".env.example" ] && exit 0
[](#cb1-6) # emit permissionDecision: deny, exit 0
[](#cb1-7) printf '%s' "$DENY_JSON"
[](#cb1-8) exit 0
[](#cb1-9) ;;
[](#cb1-10)esac
```

And the JSON it emits, which is the part that actually does the
work:

[](#cb2-1){
[](#cb2-2) "hookSpecificOutput": {
[](#cb2-3) "hookEventName": "PreToolUse",
[](#cb2-4) "permissionDecision": "deny",
[](#cb2-5) "permissionDecisionReason":
[](#cb2-6) "Editing .env files is blocked."
[](#cb2-7) }
[](#cb2-8)}
```

permissionDecision: "deny"` is what makes this a
*wall*, not a warning — the tool call never runs. That
distinction is the whole reason this is Hook 1 and not somewhere further
down the list. It’s the only one of the four that fires before the edit
happens and can actually stop it.

## Hook 2: the published-post draft guard

This one is subtler, and it exists because of a fact about how this
site is built that isn’t obvious from the filesystem:
`content/posts/*.md` is draft-only. The live, routable
version of a post is a rendered-HTML record injected into
`priv/content/site_content.json`, which the app compiles in
via `@external_resource`. Editing the `.md` file
for a post that’s already published changes nothing a reader will ever
see. It *feels* like editing the post. It isn’t.

That’s exactly the kind of trap an agent falls into with total
confidence — it opens `content/posts/some-old-post.md`, makes
a clean, correct-looking edit, reports success, and nothing on the live
site moves. The failure isn’t a bug in the edit. It’s a wrong model of
which file is authoritative, and that model is invisible from inside the
file itself.

So this is a PostToolUse hook, same matcher,
`Write|Edit|MultiEdit`, that fires *after* the edit
and checks one thing: does this slug already exist in
`site_content.json`? If it does, it emits a warning back into
the model’s context — not a block, because by the time PostToolUse runs,
the edit already happened. There’s nothing left to prevent.

`[](#cb3-1)f=$(jq -r \
[](#cb3-2) '.tool_input.file_path // .tool_response.filePath')
[](#cb3-3)case "$f" in
[](#cb3-4) */content/posts/*.md)
[](#cb3-5) s=$(basename "$f" .md)
[](#cb3-6) p="$PROJECT_DIR/priv/content/site_content.json"
[](#cb3-7) r=$(python3 check_if_published.py "$s" "$p")
[](#cb3-8) [ "$r" = "1" ] && printf '%s' "$WARN_JSON"
[](#cb3-9) exit 0
[](#cb3-10) ;;
[](#cb3-11)esac
```

The reason string is deliberately specific rather than generic — it
names the actual escape hatch (edit the record in
site_content.json`, or use the publish-post workflow)
instead of just saying “this might not do what you think.” A hook that
only says “warning” without saying what to do instead is a hook that
gets ignored the second time.

## Hooks 3 and 4: already told, so here’s the receipt not the story

The other two hooks running on this repo have their full origin
stories published already, and re-narrating them here would just be
restating what’s a click away. One line each.

**mix format, PostToolUse.** Every
`.ex`/`.exs`/`.heex` write or edit runs
the formatter automatically afterward, so a formatting-only failure can
never reach `mix precommit`. The story of the specific
pre-commit friction that earned this one — and why it’s framed as a
guardrail in CLAUDE.md rather than a rule the agent has to remember — is
in [what belongs in CLAUDE.md
after fifty commits](https://sublimecoding.com/blog/claude-md-after-50-commits).

**site_content.json validator, PostToolUse.** Any edit
that would leave the compile-time content file as invalid JSON gets
blocked with a `decision: "block"` reason explaining exactly
why: the app loads that file via `@external_resource`, so
broken JSON means a broken compile, and it’s better to catch that the
moment it happens than three steps later in a failed
`mix test`. Same post, same section, has the detail.

Both of those are mechanically identical to Hook 2 in structure —
PostToolUse, matcher on the file write, a JSON check gating the response
— which is worth noticing on its own: once you understand the pattern,
adding the fourth hook was mostly copy-and-adjust, not new design.

## The mechanics nobody’s catalog post spells out

Here’s the part that actually matters if you’re going to write your
own instead of copying mine, and it’s the part I don’t see explained
clearly even in the long catalog posts: what these event names and
fields *actually do*, not just what they’re called.

**PreToolUse fires before the tool call executes, and it’s the
only one of the two that can stop anything.** It receives
`tool_input` — the arguments about to be sent to the tool —
and it can return `hookSpecificOutput.permissionDecision` set
to `"allow"`, `"deny"`, `"ask"`, or
`"defer"`. `"deny"` means the tool call never
happens at all. That’s the only place in this whole system where “block”
means what it sounds like it means.

**PostToolUse fires after the tool call has already succeeded,
and it cannot undo anything.** The file is already written by the
time this hook runs. It gets both `tool_input` and
`tool_response`, and it can return a top-level
`"decision": "block"` with a `"reason"` string —
but per the [official
hooks reference](https://code.claude.com/docs/en/hooks), that only stops Claude’s *next* action in
the loop; it’s feedback surfaced into context, not a rollback. One of my
own hooks notes exactly this in a comment:
`decision:block here only surfaces feedback to the model — the edit already happened`.
If you want prevention, it has to be PreToolUse. If you’re checking the
*result* of something that already ran, PostToolUse is the only
option and a warning is the ceiling of what it can do.

**The `matcher` field is a name filter, not a
content filter.** `"Write|Edit|MultiEdit"` is regex
alternation over the tool name Claude Code is about to call or just
called — it has no visibility into the file path or arguments until the
hook’s own command runs and inspects `tool_input` itself.
That’s why every one of these four hooks does its own `jq`
extraction and its own `case` statement on the file path —
the matcher only gets you into the room, it doesn’t tell you what’s in
the room.

**Exit codes and JSON output are two different signaling
channels, and only one is processed at a time.** Exit 2 is a
blunt, JSON-free way to fail a hook — it’s read as a blocking error on
events that support it. JSON on stdout with exit 0 is the structured
channel, and it’s the only one that gets parsed for fields like
`permissionDecision` or `decision`. All four of my
hooks use the JSON-on-exit-0 path, because I want a specific reason
string in the model’s context, not just a bare failure.

## The decision rule: hook, or CLAUDE.md line?

Every operational lesson from a real project ends up in one of two
places, and I use one rule to sort them, the same rule I use when
deciding what earns a CLAUDE.md entry versus what stays unwritten:
**if a rule can be mechanically enforced, it becomes a hook. If it
requires judgment the mechanism can’t make, it stays as
prose.**

“Never edit `.env`” is purely mechanical — there is no
case where I want an agent’s judgment to override it, so it’s a hard
deny with zero ambiguity. “This post is already published, editing the
`.md` won’t change the live site” is also mechanical to
*detect*, but the right response still depends on intent —
sometimes you *do* want to sync the draft on purpose — so it’s a
warning that hands the decision back, not a deny. “Follow the existing
conventions” or “prefer clarity” can’t be mechanically checked at all;
there’s no `jq` expression for good taste, so those stay as
prose, if they’re written down anywhere.

The tell that a lesson belongs in a hook instead of a sentence:
you’ve written the same CLAUDE.md-style instruction more than once
because the agent (or you) forgot it, and the check for whether the
instruction was followed is something a shell one-liner could actually
answer. A rule that can be forgotten belongs in prose only until it’s
been forgotten twice — at that point it’s cheaper to make it
un-forgettable.

## What’s not on this list, and won’t be until it’s earned

I have a fifth PostToolUse check in this repo — a fast SEO check on
frontmatter, title length and tag taxonomy, that runs on every post
edit. It’s newer, it’s still proving itself, and I’m not confident yet
it’s earned a permanent slot the way these four have. That’s the honest
state of it: this post is about the four that have survived long enough
to trust, not an inventory of everything currently running. The same
discipline that keeps a CLAUDE.md from bloating applies here — a hook
that hasn’t paid for its keep yet doesn’t get written up as settled.

If you’re setting up your own project and you’re tempted to install a
dozen hooks from one of the big catalog posts on day one: don’t. None of
mine started that way. Each one is the mechanical answer to a mistake
that already happened at least once, the same as every entry in [my CLAUDE.md that survived past
commit fifty](https://sublimecoding.com/blog/claude-md-after-50-commits). Write the hook after the second time you catch
yourself typing the same warning into a prompt, not before.

If this kind of guardrail-over-vibes approach to agent tooling is the
layer you’re missing, the plugin- and skill-level version of the same
discipline — what earns a permanent slot in the stack versus what gets
uninstalled — is in [the Claude
Code plugin stack that actually stuck](https://sublimecoding.com/blog/claude-code-plugin-stack). And if you’re trying to
figure out which of this whole ecosystem is worth your time before
you’ve burned a week on it yourself, that’s what [the Claude Code resource
bible](https://sublimecoding.com/blog/claude-code-resource-bible) is for.

Setting this up for a team rather than a solo project — where the
hooks need to be consistent across everyone’s local config, not just
yours — is exactly the kind of AI-tooling groundwork I help founders and
engineering leads get right early. [If that’s where
you are, let’s talk](https://sublimecoding.com/consulting).

---

## Isolating Parallel Claude Code Agents in Phoenix

URL: https://sublimecoding.com/blog/parallel-claude-code-agents-phoenix
Published: 2026-07-27
Tags: Elixir, Phoenix, developer workflows

>
**TL;DR:** A git worktree gives an agent its own file
tree and branch. It does not give it its own port, its own database, or
its own asset watcher — and on a Phoenix + Ecto app, those three
collisions are exactly where a promising parallel-agent setup falls
apart. This is the Phoenix-specific plumbing: templating
`PORT` per worktree through `config/runtime.exs`,
why a shared Postgres database breaks the moment two branches diverge on
a migration, why `esbuild`/`tailwind` watchers
fight each other, and the honest math on when the isolation is worth
building versus just running agents one at a time.

## The bug that already lives in this site’s CLAUDE.md

This site’s own repo has a rule that exists because of a real
afternoon lost to it: always start the dev server with
`PORT=4001 mix phx.server`, never the bare command. An
unrelated process on this machine — a different project entirely —
already squats on port 4000. Run the default command and one of two
things happens: Phoenix fails to bind and you get an obvious error, or
worse, something else is *already listening* and your
`curl` or health check quietly hits the wrong app, returning
404s for routes that are completely fine. That rule is the kind of [CLAUDE.md entry that only gets
written after it costs you something](https://sublimecoding.com/blog/claude-md-after-50-commits) — and it’s among the
highest-value lines in the file, precisely because it’s a fact about the
machine, not the code, and nothing in the repo would ever tell you to
look for it.

One port conflict, one dev server, one human debugging it: that’s a
bad afternoon. Run that same conflict across four Claude Code sessions
each trying to bind their own copy of the app, and it’s not an afternoon
anymore — it’s four agents stepping on each other, on a machine,
silently, while you’re doing something else.

## Why the generic worktree tutorials don’t help here

Search “git worktrees Claude Code parallel agents” today and you’ll
find [Anthropic’s own
worktree docs](https://code.claude.com/docs/en/worktrees), a YouTube walkthrough, and at least half a dozen
near-identical blog posts (MindStudio alone has run the topic twice)
covering the same three commands: `git worktree add`, point a
new Claude Code session at the directory, done. That content is fine as
far as it goes — a worktree genuinely does give each agent an isolated
working directory and branch, sharing one `.git` history with
no stashing and no file collisions.

But “fine as far as it goes” stops well short of a real Phoenix app.
A worktree isolates your *files*. It does nothing about the three
things a Phoenix + Ecto app also needs isolated to run two copies side
by side: the HTTP port, the database connection, and the asset build
watchers. Every generic worktree post treats “now run the dev server” as
a throwaway last step. For a Rails or Django toy app that might be close
enough. For a Phoenix app with Ecto migrations and an esbuild/tailwind
watcher pair, it’s where the setup actually breaks — and it’s the part
nobody writes about, because it’s framework-specific and
framework-specific content doesn’t rank as well as “how to use git
worktrees.”

So: not another worktree tutorial. The mechanics underneath it, for
the stack most of the generic content ignores.

## Port templating: one `PORT` export per worktree

The fix for the port problem generalizes past the one-conflict story
above. If you’re running N parallel agent sessions, each in its own
worktree, each needs its own port — and in a stock Phoenix app, that’s
one environment variable away from working, *if* your config
actually reads it.

Check where your app’s port comes from. A freshly generated Phoenix
app hardcodes a port in `config/dev.exs`. This site’s config
doesn’t — the HTTP port is resolved in `config/runtime.exs`,
which Phoenix loads for every environment (not just prod), reading:

`[](#cb1-1)config :sublime_coding, SublimeCodingWeb.Endpoint,
[](#cb1-2) http: [
[](#cb1-3) port: String.to_integer(
[](#cb1-4) System.get_env("PORT", "4000")
[](#cb1-5) )
[](#cb1-6) ]
```

That one line is why PORT=4001 mix phx.server` works at
all here — there’s no `port: 4000` sitting in
`dev.exs` to override. If your app *does* have a
hardcoded port in `dev.exs` (the default from
`mix phx.new`), move it to read
`System.get_env("PORT", "4000")` the same way before you try
to run more than one worktree at once. It’s a one-line change and it’s
the precondition for everything else in this post.

With that in place, each worktree gets a small `.envrc` or
launch script that pins its own port:

`[](#cb2-1)# worktree-a/.envrc
[](#cb2-2)export PORT=4001
[](#cb2-3)export DATABASE_URL="$PG_BASE/myapp_wt_a"
[](#cb2-4)
[](#cb2-5)# worktree-b/.envrc
[](#cb2-6)export PORT=4002
[](#cb2-7)export DATABASE_URL="$PG_BASE/myapp_wt_b"
```

If you use direnv`, this is automatic on `cd`.
If you don’t, a two-line `bin/dev` script per worktree that
exports the vars before `exec mix phx.server` does the same
job with no new dependency.

One thing you get for free once the port is templated: if you run
Tidewave or another Phoenix-runtime MCP server, it mounts as a plug on
the endpoint itself — at `/tidewave/mcp` on whatever port the
app is bound to, not a separately configured service. Template the
`PORT`, and the MCP endpoint follows it automatically; you
never hand-maintain a second map of “which agent talks to which MCP
URL.” Point each Claude Code session’s `.mcp.json` at
`http://localhost:<that worktree's port>/tidewave/mcp`
and the runtime introspection — logs, `project_eval`,
whatever the server exposes — stays scoped to the right agent’s copy of
the app.

## One database per worktree, not one database shared

Ports are the visible collision. The database is the one that
actually corrupts state, and it’s the part every generic worktree post
skips entirely, because most of them aren’t written for anything with a
persistence layer.

Here’s the failure mode if two worktrees share one Postgres database.
Agent A is on a branch that’s three migrations ahead — it added a
column, backfilled it, added a `NOT NULL` constraint. Agent B
is on a branch that hasn’t seen those migrations yet, still inserting
rows the old way. They’re not two isolated experiments anymore; they’re
two different schemas fighting over one table, and whichever one runs
`mix ecto.migrate` last wins, silently, until the other
agent’s next query throws an error that has nothing to do with the
actual bug it’s working on. Ecto’s [`Sandbox`
adapter](https://hexdocs.pm/ecto_sql/Ecto.Adapters.SQL.Sandbox.html) solves a related but different problem — it wraps
*concurrent tests inside a single BEAM node* in per-connection
transactions so they don’t see each other’s writes. It does nothing for
two entirely separate `mix phx.server` processes, in two
different worktrees, both writing to the same database outside a test
run. Two dev servers on shared state will fight, and the sandbox has no
opinion about that at all.

The fix is boring and it’s the same one that already works for the
analogous problem in CI: one database per worktree, named after the
branch or worktree slot.

`[](#cb3-1)# per-worktree setup, run once per new worktree
[](#cb3-2)export DATABASE_URL="$PG_BASE/myapp_wt_a"
[](#cb3-3)mix ecto.create
[](#cb3-4)mix ecto.migrate
```

A naming convention that survives more than a week:
<app>_<worktree-slot>_dev`,
e.g. `myapp_wt_a_dev`, `myapp_wt_b_dev` — tied to
the *slot*, not the branch name, since branches get deleted and
worktree directories get reused for the next task. Seed data is the
other half of this: if your `priv/repo/seeds.exs` assumes a
fresh database, run it once per worktree after `ecto.create`,
not once globally and hope every worktree inherited it.

The cost here is real and worth naming honestly: N worktrees means N
Postgres databases sitting on disk, and N sets of migrations to keep
current if a worktree sits stale for a week while its sibling branches
move on. That’s a small, fixed cost against the alternative, which is an
agent debugging a `NOT NULL` constraint violation that’s
actually a stale schema from a different branch — the kind of failure
that reads exactly like [the
confident-wrong-answer pattern I catch from agents every week](https://sublimecoding.com/blog/things-ai-coding-agents-get-wrong),
except this time the agent isn’t wrong, your test database is.

## Asset watchers: the collision nobody budgets for

The third collision is the one that costs CPU and wall-clock time
rather than correctness, and it’s the one I see people discover last.
Phoenix’s dev watchers — `esbuild` and `tailwind`
in `config/dev.exs` — run as OS processes alongside the BEAM,
rebuilding `priv/static/assets` on file change. Run two
worktrees’ dev servers at once and you have two independent watcher
pairs, each polling its own copy of `assets/`, each
recompiling into its own `priv/static/`. That part is
actually fine — worktrees give each one a separate `assets/`
directory to watch, so they don’t literally clobber the same output
file.

What does bite you: four or five worktrees each running their own
`esbuild --watch` and `tailwind --watch` is four
or five extra long-lived OS processes idling on a laptop that’s also
running four or five Claude Code sessions and four or five Postgres
connections. On a modern machine that’s a nuisance, not a wall. But it’s
real enough that “how many worktrees can I actually run at once” has a
practical ceiling closer to four or five than to “as many as I want” —
past that, the fan noise and the context-switching cost between sessions
both go up faster than the throughput does. If you’re not actively
editing frontend assets in a given worktree, killing its watcher
(`mix phx.server` without the `watchers` config,
or a `--no-watch` env flag you wire in) is a legitimate way
to claw back the overhead for agent sessions that are purely backend
work.

A worktree isolates your files. It does nothing about the port, the
database, or the watcher — and on a real Phoenix app, that’s where the
setup actually breaks.

## When the isolation is worth building

None of this is free, and the honest version of this post says so out
loud instead of selling you the setup unconditionally. Standing up N
worktrees with N ports, N databases, and N watcher pairs costs you real
time up front — the `.envrc` templates, the per-worktree
`ecto.create`, the discipline to keep migrations in sync
across siblings that are about to merge back into the same branch. It
costs disk: N copies of `node_modules`-equivalent build
artifacts and N Postgres databases sitting around. And it costs a kind
of mental overhead that doesn’t show up in any setup script — you now
have to *remember which worktree you’re in* before you trust what
a terminal or a browser tab is telling you, which is exactly the class
of mistake the PORT-conflict story above already demonstrates once.

Where it earns that cost: genuinely independent, non-overlapping
tasks that would otherwise serialize behind each other for no good
reason — a backend migration in one worktree while an unrelated frontend
polish pass runs in another, or three agents each closing out a separate
ticket that don’t touch the same files. I’ve described what that looks
like day to day, running four to seven parallel agent sessions through a
normal engineering day, in [my daily agentic AI
workflow](https://sublimecoding.com/blog/my-daily-agentic-ai-workflow) — the isolation problem in this post is the infrastructure
underneath that workflow, not a replacement for it.

Where it doesn’t earn the cost: two agents that are going to touch
the same models, the same migration, or the same feature from different
angles. Isolating their file trees doesn’t remove the merge conflict, it
just delays it and moves it from “now, while I understand both branches”
to “later, when I’ve forgotten what either one was doing.” For genuinely
coupled work, running agents sequentially — one at a time, each building
on the last commit — is the lazier and correcter choice, even though it
feels slower in the moment. And whichever mode you’re in, the tooling
that decides *how* an agent works matters more than how many of
it you’re running at once; I keep a fairly narrow set of plugins for
exactly that reason, laid out in [the Claude Code plugin stack I
actually run](https://sublimecoding.com/blog/claude-code-plugin-stack).

The setup in this post — templated `PORT`, per-worktree
`DATABASE_URL`, watcher discipline — is maybe an hour of
one-time work per project. It pays for itself the first time it prevents
an agent from spending twenty minutes debugging a 404 that was actually
a port collision, or a constraint violation that was actually a stale
schema from a sibling branch. It does not pay for itself as a default
you reach for on every task, and treating it that way is how a genuinely
useful pattern turns into ceremony.

## If you’re standing up agent infrastructure for a team

This is exactly the kind of infrastructure decision that’s cheap to
get right early and expensive to unwind once three engineers have built
habits around whatever half-isolated setup shipped first. If you’re
scaling a Phoenix codebase across a team that’s adopting parallel agent
workflows and want a second set of eyes on the setup — ports, database
isolation, CI implications, the works — that’s a conversation worth
having as part of a [fractional engineering
engagement](https://sublimecoding.com/consulting), before the workaround habits calcify.

---

## Jose Valim Is Right: Anthropic's Incentive Problem

URL: https://sublimecoding.com/blog/jose-valim-anthropic-incentive-problem
Published: 2026-07-30
Tags: AI, security, AI startups

>
**TL;DR:** [Anthropic
published its position on open-weights models](https://www.anthropic.com/news/position-open-weights-models) on July 27 — and
contrary to what the resulting news cycle implied, it does not call for
a ban. It asks for chip export controls, restrictions on
industrial-scale distillation by authoritarian states, and mandatory
pre-release safety testing that Anthropic says “would need to be global”
to work. [José Valim,
the creator of Elixir, responded on X](https://x.com/josevalim/status/2082000928505536771) with a critique sharper than
most of the reaction I’ve seen: you can’t ask for global cooperation on
safety testing while arguing your own country should pursue policies
that preserve its strategic advantage, and you can’t expect the public
to read your safety argument as neutral when it also protects your
business. Valim’s still right, on both points, and it’s worth being
precise about why.

## What Anthropic actually said

Start with the part most of the commentary skipped: what the post
says, not what the internet decided it said.

On July 24, Nvidia CEO Jensen Huang posted [an
open letter](https://www.forbes.com/sites/sandycarter/2026/07/25/huangs-open-weights-letter-doubled-to-50-without-amazon-and-anthropic/) urging Washington not to restrict downloadable AI
models. It went out with 25 signatures and doubled to 50 within a day,
including OpenAI, Google, AMD, Cisco, GitHub, and Block. Two names were
conspicuously missing from every version of the list: Amazon and
Anthropic. Given that Amazon is Anthropic’s largest investor, the
pairing of absences read, to a lot of people watching, like a signal —
and the obvious inference was that Anthropic wants open weights
restricted or banned.

Three days later, Anthropic answered directly — in a post signed by
Dario Amodei personally — and the actual position is narrower than the
inference. [The
post is unambiguous](https://www.anthropic.com/news/position-open-weights-models): “Anthropic has never advocated for a ban on
open-weights models.” Amodei states it twice, opening and closing:
“Protectionist bans would not address my most serious national security
concerns,” and, in the summary, “we have not and are not advocating for
a ban on open-weights models as a category.” That’s a real position,
stated plainly, worth taking at face value rather than assuming the
worst version of it because a competitor’s letter didn’t have their name
on it.

What the post does ask for is three specific things. First, chip
export controls — don’t sell powerful chips or chipmaking equipment to
China, and close the [smuggling
routes](https://www.justice.gov/opa/pr/three-charged-conspiring-unlawfully-divert-cutting-edge-us-artificial-intelligence) around existing controls. Second, restrictions on
industrial-scale distillation operations that let authoritarian states
cheaply extract capability from frontier models without doing the
underlying research. Third, and the strongest of the three on the
merits: mandatory pre-release safety testing for cyber, biological, and
alignment risks, applied to all sufficiently capable models — open and
closed — testing that Anthropic says “would need to be global, which
means even the CCP would need to be on board.” The named threats are
authoritarian states achieving durable military or surveillance
superiority, and models capable enough to meaningfully assist
cyberattacks or biological misuse.

None of that is a ban. It’s a specific, arguable policy ask — the
kind of position I’d have written myself if I were running a lab that
had spent five years building a safety-first brand and then watched a
competitor’s open letter imply the opposite of its actual position
without asking first.

## Where Valim’s critique lands

José Valim read the same post and [posted a
response](https://x.com/josevalim/status/2082000928505536771) that doesn’t dispute any of the facts above. He grants the
underlying security concerns are “solid.” His objection is structural,
and it’s really two objections wearing one tweet.

The first is a game-theory point, and it’s the cleaner of the two. In
his words: the post “tries to have it both ways: it calls for global
cooperation on mandatory safety testing while simultaneously arguing
that the US should pursue policies to preserve its strategic and
economic advantage in AI.” His conclusion: “If states are expected to
act in their own national interest, it is unclear why global powers
would voluntarily participate while their own technological and
strategic ambitions are constrained.”

Sit with that, because it’s not an abstract objection — it describes
how the ask is actually structured. Chip export controls and
distillation restrictions read, in Valim’s framing, as measures to
preserve US advantage over China. Mandatory testing, in Anthropic’s own
words, is a regime that “would need to be global, which means even the
CCP would need to be on board.” Put those next to each other and read
them the way a state actor would: one policy keeps you behind, the other
asks you to accept a constraint on your own model releases in the name
of shared safety. Why would the same government sign up for the second
while the first is explicitly designed to keep it out of the race the
second claims to be making safer for everyone? A regime you’re
simultaneously trying to out-compete and cooperate with has no clean
reason to pick cooperation on terms written by the side asking to stay
ahead.

Anthropic isn’t blind to this. The post offers an answer: cooperation
on bioweapons may be possible “because it is in China’s interest too.”
That’s true, and it’s the best version of the case — but it’s
load-bearing for exactly one of the three named risks. Nothing in mutual
bio-interest explains why a state would accept pre-release gating on
cyber capability or alignment while chip controls are explicitly
designed to keep it a generation behind. The answer covers the narrowest
threat and leaves the asymmetry Valim named untouched.

You can’t structure two of your three asks around keeping a rival
behind and expect that rival to volunteer for the third.

That’s not a rejection of mandatory testing as an idea — it’s the
strongest of the three recommendations precisely because it’s the one
that, done right, reduces risk regardless of who’s ahead. It’s also the
one most undermined by sitting next to the other two. If Anthropic wants
global participation in a testing regime, the pitch has to be made on
its own terms, separated cleanly from the strategic-advantage framing,
or a rational state reads the whole package as a play for advantage with
a safety label on it.

Valim’s second point is about perception rather than logic, and it’s
the harder one to argue with because it isn’t really about Anthropic’s
actual position at all: “the public perception is that Anthropic has not
done enough to distinguish its security arguments from policies that
also serve its commercial interests. As a result, its safety agenda is
unlikely to be perceived as economically neutral.”

That’s a claim about how the argument lands, not whether it’s true.
And on the evidence of the last week — a missing signature read as a
smoking gun, a clarifying post that had to explicitly say “we have never
advocated for a ban” because people had already concluded otherwise —
it’s hard to argue the perception isn’t exactly what he says it is.

## Why this critique is inconvenient for me

I want to be honest about where I’m standing before I say more,
because it changes how much this critique should weigh.

I’m not a neutral observer of either side of this. Claude Code is the
tool I use to write and ship code on this site — it’s in my daily
workflow, not a vendor I’m evaluating from a distance. I’ve written
about [what a CLAUDE.md file
looks like after fifty real commits](https://sublimecoding.com/blog/claude-md-after-50-commits) and [how TDD actually works with Claude
Code in an Elixir codebase](https://sublimecoding.com/blog/tdd-claude-code-elixir), because that’s genuinely how I build
things now. And the backend I run this site on, and recommend to
clients, is Elixir — Valim’s language. If there’s a reader who came in
rooting for Anthropic to have a clean answer here, it’s roughly me.

That’s exactly why the critique is worth taking seriously instead of
waving off. It’s not coming from a rival lab’s PR account or a
competitor’s cheering section — it’s coming from someone whose
credibility runs the opposite direction of “wants to see Anthropic look
bad.” A builder with no stake in tearing the argument down reading it
and still concluding it’s unlikely to be perceived as economically
neutral is a stronger signal than the same critique from someone with an
obvious axe to grind.

## It’s not an Anthropic problem — it’s every lab’s problem

Here’s the part that keeps this from being a hit piece, and the part
I think matters most: Valim’s second point isn’t really an indictment of
Anthropic specifically. It’s a structural feature of lab-led safety
advocacy generally. Every frontier lab’s safety position, traced far
enough, ends up disadvantaging somebody else’s business model. A
closed-weights lab’s positions on model access constrain open-source
labs. Anthropic’s positions on testing and export controls constrain
open-weights players and, more distantly, Chinese labs. A lab arguing
for stricter oversight of “sufficiently capable models” is, not
coincidentally, usually a lab already ahead on capability. This is the
same skepticism I apply to any vendor’s stated policy — [what does the
person making the argument need to be true](https://sublimecoding.com/blog/enterprise-security-reviews-ai-startups), the same test I ran on
[a vendor’s
benchmark claim](https://sublimecoding.com/blog/ai-security-benchmark-what-96-percent-means) two days before this one — and does the argument
cost them anything, or only someone else?

That question is the actual test for whether a safety position is
economically neutral, and it’s worth running Anthropic’s three asks
through it individually rather than treating the post as one bundle.

Chip export controls on China cost Anthropic nothing — they constrain
a jurisdiction Anthropic doesn’t sell into and don’t touch its own model
releases. Distillation restrictions cost it a little, not much:
Anthropic commits to identifying and banning its own paying accounts
caught doing this, but concedes those accounts are usually only
identifiable after substantial distillation has already happened — a
real cost, just a small and lagging one. Mandatory pre-release safety
testing is the one asymmetric case, and to Anthropic’s credit, it’s the
one that does cost something real: the post is explicit that testing
applies to “all sufficiently capable models, open and closed” — meaning
Anthropic’s own frontier releases go through the same gate it’s asking
regulators to impose on everyone else. That’s a position that constrains
the author, not just the target — the one piece of the three-part ask
that clears the neutrality bar Valim is describing. The other two cost
Anthropic little to nothing, and the post never draws that distinction
itself. That’s the actual gap: not dishonesty, but a failure to separate
the ask that’s genuinely self-limiting from the two that aren’t, at a
moment when the audience is primed to read all three as the same kind of
move.

## The fix is structural, not rhetorical

I don’t think Anthropic is wrong about the underlying risk. Chip
smuggling to sanctioned states is a real problem with real precedent.
Industrial-scale distillation of frontier capability by state actors is
a plausible, under-discussed threat. And mandatory pre-release testing,
applied evenly, is close to the least controversial safety idea in the
entire AI policy conversation — it’s hard to find a serious critic of
testing capable models before release, on principle.

The problem isn’t the substance. It’s that the substance arrived
bundled with two asks that visibly serve US strategic interest, from a
company that also didn’t sign a competitor’s letter the same week, in an
industry that has learned to price “safety argument” and “business
interest” as correlated rather than independent. No amount of restating
“we’ve never advocated for a ban” fixes that, because the framing
problem was never about the ban claim — it’s about which asks get
bundled together and who’s making them.

If Anthropic wants the testing recommendation read as neutral, the
fix isn’t a better press statement. It’s separating the self-limiting
ask from the two that aren’t, making that separation explicit rather
than implicit, and probably having the testing case made by a coalition
or standards body rather than the lab that benefits most from being
first through the gate. Solid concerns, credible in isolation.
Undermined by delivery. And delivery, unlike the physics of chip fabs
and model weights, is entirely something a lab controls. (Six weeks
later the two cases got published side by side, on the record — [the upside
scenario and the extinction odds in the same week](https://sublimecoding.com/blog/anthropics-extreme-scenario-has-a-precondition).)

If you’re building a startup that has to make its own claims to
investors, customers, or regulators credible — where the same “does this
cost us anything” test gets applied to your own security and compliance
posture — that’s exactly the kind of scrutiny I help founders get ahead
of before someone else runs the test on them. [Let’s talk](https://sublimecoding.com/consulting).

---

## What 90 Days of a Fractional Security Engagement Actually Looks Like

URL: https://sublimecoding.com/blog/what-a-fractional-security-engagement-actually-looks-like
Published: 2026-05-03
Tags: security, vCISO, founders, AI startups, business

**Most founders who book the intro call have already read three or four of my posts and arrive at the same question: "Okay, but what would the next 90 days actually look like if I hired you?" Here's the answer — a sanitized week-by-week composite of a typical 90-day fractional security engagement, with the real numbers attached.**

Not every engagement looks exactly like this. But this is what most pre-Series-A AI engagements look like — same shape, different details. Composited from real work, no specific client.

The starting state: a Series-Seed AI startup, eight engineers, ARR in the low seven figures, two enterprise prospects in pipeline. Both prospects are stuck at the security questionnaire. One explicitly asked for SOC 2 Type I; the other implied it. The founders haven't run a security program before. The CTO has been triaging the questionnaires personally and hates every minute of it.

The ask: get them through both questionnaires within 90 days, run the SOC 2 Type I audit in parallel, and stand up the security work that needs to outlast the engagement.

## Week 0 — the scoping call

Thirty minutes, and it is mostly me asking questions. What are your current goals. Where does your compliance actually stand. What are you using to monitor it, and what do you wish it did better. What do you need the most help with. Are you losing sales over a compliance structure you don't have yet, and have clients started asking for it by name. Where those land decides what I drill into next — usually infrastructure, sometimes something else entirely.

No single answer decides anything. The shape of the engagement comes out of the whole conversation, and sometimes that shape is smaller than what the founder walked in asking for. Two things do need to be in place before we start, though neither is what decides whether we work together: a budget, and someone internal I can talk to — usually the CTO or a senior engineer who will absorb the role after I'm gone. Without an internal owner there is nobody for the work to outlast me in.

Pricing gets confirmed on this call too. Standard fractional CISO retainer is $3K/month for 10–15 hours of strategic work — [run your own numbers in the vCISO cost calculator](https://sublimecoding.com/tools/vciso-cost). Audit-prep engagements run four months, occasionally six. Deliverables and out-of-scope items go into a one-page scope doc the next day. No surprise overages. No retainer creep. What a four-month embedded engagement actually ships, month by month, is [written up separately](https://sublimecoding.com/blog/four-months-embedded-ai-engineering).

## Weeks 1–2 — discovery and the policy pack

Discovery is short by design. I'm not running a six-week assessment — I'm running a one-week one because I already know what most pre-Series-A AI startups look like and what they're missing. The week is for confirming the gaps, not for finding them.

What gets done in week one: read all the existing security documentation (usually a one-page README and a handful of Notion pages), interview the CTO and the founder for an hour each, walk the production environment with whoever's on-call, and pull the existing controls into a SOC 2 readiness matrix. By the end of week one, the gap list is on paper.

Week two is the policy pack. Acceptable use, access control, data classification, incident response, vendor management, change management, vulnerability management, business continuity. Eight policies, drafted in the company's voice from templates I've been carrying for years, customized to the actual technology stack. The CTO reviews and signs off. We push them into the Vanta or Drata instance the same week.

By end of week two, the company has policy language that holds up to an auditor's read and a written gap list ranked by audit-blocking severity.

## Weeks 3–6 — SOC 2 readiness in parallel with the AI security stack

This is the heavy stretch. Two work streams running in parallel.

The SOC 2 stream: implement controls against the gap list, configure the compliance tool to track evidence collection, schedule the auditor (early — the good ones book out four to six weeks), document the technical controls (MFA enforcement, access provisioning, secure SDLC), and start collecting evidence as the controls go live. Most of the engineering team's involvement happens in week three when we configure the access provisioning tooling and again in week five when we wire the production change management process into their existing PR workflow.

The AI security stream: this is where the work that's not in the SOC 2 framework lives. The four-layer AI-native stack — [prompt injection defense, agent credential scoping, secrets handling, audit logging](https://sublimecoding.com/blog/running-security-at-an-ai-native-company-2026) — gets stood up in this window. Credential scoping comes first, because that's where the largest blast-radius incidents originate. By end of week four, every agent and every service account has the minimum credential surface needed to do its job. No "admin to be safe" anywhere.

By end of week six, both streams have visible momentum. Vanta or Drata shows green on most of the framework. The AI-native layer has credential scoping done and prompt injection guardrails in design. Both enterprise prospects get an updated security questionnaire response that shows real evidence of the work — and both move forward on their evaluation.

## Weeks 7–10 — auditor engagement and the AI-native layers

Weeks seven and eight are auditor fieldwork. The auditor runs interviews with the team, walks the production controls, samples the evidence, and asks the questions that don't have clean answers. My job in this stretch is to be the security executive in the room with the auditor — answering the technical questions, defending the design decisions, and protecting the engineering team from the death-by-questionnaire pattern that kills first-time SOC 2 attempts.

Weeks nine and ten close out the AI-native layers. Secrets handling moves from ad-hoc (env vars and the occasional `.env` checked into a private branch) to deliberate (Vault or AWS Secrets Manager, automated rotation on the secrets that actually matter). Audit logging gets stood up — every agent action, every privileged service account call, every credential-bearing API request goes to a central log with retention long enough to investigate an incident a month after it happens.

By end of week ten, the auditor's draft report is in review. Both enterprise prospects have what they need. The technical work outlives the engagement.

## Weeks 11–12 — report and graduation

Week eleven is the auditor report cycle. Their draft, our review, their final. The Type I report ships at end of week eleven or early week twelve. Both enterprise prospects close their security review within ten business days of the report landing.

Week twelve is the graduation conversation. I always have the same conversation around this point. "You don't need me on the day-to-day anymore. Here's what comes next, and here's when you'd hire someone full-time."

Most engagements either step down to a low-touch advisory retainer — four to six hours per month at $2K/month, mostly for security review questions and the next year's audit prep — or graduate completely. The bad outcome is the one where I'm still the on-call security executive at month nine. That means I haven't built the program right. The internal owner identified in week zero needs to be operational by week twelve, or I've failed the engagement.

## What it cost

Total 90-day spend, all in:

 - Auditor fee: $15–20K (varies by auditor, scope, and audit-prep tooling)

 - SOC 2 tooling ([Vanta](https://www.vanta.com) or [Drata](https://drata.com), year one): $5–10K

 - Fractional CISO retainer: $12K (four months at $3K)

 - Internal time: roughly four hours per week of the CTO's attention; less for the engineering team after week three

Total external spend: $32–42K. Inside the [$25–45K range I quote in the SOC 2 post](https://sublimecoding.com/blog/soc-2-is-a-revenue-tool-not-a-security-tool), on the higher end because of the AI-native layers added in parallel.

Cost of NOT running this engagement: two enterprise deals that don't close. Pipeline that ages out. The CTO answering security questionnaires personally for the next six months instead of building the product. Easy math.

## What actually changed

Two named enterprise deals unblocked. SOC 2 Type I report in hand for the next six prospects. A security program that outlives the engagement: one internal owner, one accountable executive, weekly review cadence, and a roadmap to Type II audit at the next renewal.

Specific operational outcomes: every service account has minimum-viable credentials. Every agent has bounded tool access. Every secret is in a manager with rotation policy. Every privileged action lands in a searchable audit log. None of these existed at week zero.

Cultural outcomes that matter just as much: the CTO knows what to say in a security questionnaire and what to escalate. The engineering team has a security review pattern they can run for new features without me. The founder has a number — a real one — for what compliance costs at the next stage.

## When this engagement isn't right

Not every founder should hire a fractional CISO. The wrong fit produces a worse outcome than no fit at all.

Wrong-fit signals:

 - The deal blocking on the security review isn't real. ("We think SOC 2 would help us close more enterprise" without a specific named prospect = wrong stage.)

 - No internal owner. If nobody on the team will absorb the role at month four, the engagement either extends indefinitely or rolls back inside two quarters.

 - Regulated industry that requires a full-time CISO on the org chart for the deals being pursued. Healthcare with PHI, payments, government.

 - Security work consistently exceeds 20 hours per week. Past the fractional break-even — full-time is now the right answer. (See [the vCISO math post](https://sublimecoding.com/blog/vciso-math-for-ai-founders) for the graduation criteria.)

If any of those describe you, the conversation we should have isn't about hiring me — it's about what your real next move actually is. I'll say so on the call.

## How to get this conversation started

If you're a pre-Series-A AI founder with a named enterprise deal blocked behind security review, the intro call is the right step. Thirty minutes. Three questions. We figure out together whether this engagement shape is the right one for you — and if it isn't, I'll point you at what is.

[Engagement model and the next step are here](https://sublimecoding.com/consulting).

## Read this next

 - [vCISO Math for AI Founders](https://sublimecoding.com/blog/vciso-math-for-ai-founders) — the make-vs-buy argument and the graduation criteria.

 - [SOC 2 Is a Revenue Tool, Not a Security Tool](https://sublimecoding.com/blog/soc-2-is-a-revenue-tool-not-a-security-tool) — the reframing that makes the engagement above worth running.

 - [How I'd Run Security at an AI-Native Company in 2026](https://sublimecoding.com/blog/running-security-at-an-ai-native-company-2026) — the four-layer stack the engagement above stands up.

---

## How I'd Run Security at an AI-Native Company in 2026

URL: https://sublimecoding.com/blog/running-security-at-an-ai-native-company-2026
Published: 2026-04-20
Tags: security, AI, agents, founders, AI startups

**AI-native companies need a security model that classic appsec doesn't cover. Most don't have one.**

 **Key takeaways**



 - **The four-layer stack: prompt injection defense, agent credential scoping, secrets handling, audit logging.** None of these come for free in classic appsec.

 - **Start with credential scoping in the 90-day plan.** That's where the largest blast-radius incidents originate; everything else builds on top.

 - **Defer SIEM, formal threat modeling, and a full bug bounty program until post-Series-A.** Pre-A, you pick the high-leverage layers; the rest is operational debt.



The pattern I see across early-stage AI companies: a strong engineering team treats security like a 2018 SaaS product — auth, secrets, the SOC 2 checklist. Meanwhile their product is shipping autonomous agents with cloud credentials, accepting unstructured input from customers as the primary interface, and training models on data the customers didn't fully realize they were exposing. The threat model has changed. The controls haven't kept up. The secrets half of that has a concrete Phoenix answer: [runtime.exs, Fly, and LLM keys](https://sublimecoding.com/blog/phoenix-secrets-runtime-fly).

If I were building the security program at an AI-native company today, this is the layered stack I'd put in place, the things I'd ship in the first 90 days, and the things I'd consciously defer.

## The four-layer stack

Classic appsec is one layer of four. Treating it as the whole picture is the most common mistake I see.

### Layer 1 — Classic application security

This is everything that's been good practice for fifteen years and doesn't go away because you're an AI company. Auth and authorization. Secrets management. Input validation. SQL injection prevention. CSRF tokens. SSRF guardrails. TLS everywhere. Least-privilege IAM. Logging and audit trails. Backups and recovery.

This layer is solved. The advice has been written down a hundred times. If you're not doing it, do it. If you are, skip the rest of this layer's discussion and move on. The interesting work for AI companies is in the next three layers.

### Layer 2 — Data security and the training question

The novel question for AI-native companies is what data goes into the model and where it ends up.

The threats:

- **Training-data exfiltration.** A model trained or fine-tuned on customer data can leak fragments of that data through generation. This is real, has been demonstrated repeatedly, and is not solved by "we delete the data after training."

- **Prompt-context leakage.** Customer A's data ends up in customer B's response because both customers share the same backend prompt context. RAG pipelines are the worst offender here.

- **Vendor-side training.** You send customer data to a foundation model API. The vendor uses it to improve their model. Your customer didn't consent to that.

The controls I'd ship:

- **Tenant isolation in retrieval.** Every vector-DB query and every RAG retrieval must filter by tenant ID at the index level, not in post-processing. This is the single most common AI company security bug I see in code review.

- **No-train flags on every vendor API.** OpenAI, Anthropic, Google, AWS Bedrock all have versions of "do not use this for training." Default-on, document the setting, audit it quarterly.

- **PII redaction before retention.** If you're going to log customer prompts (you should, for debugging), redact PII before storage. Microsoft Presidio, Google DLP, or a homegrown regex set — pick one and run it.

- **Document the training data lineage.** Be able to answer "what data did this model see during training and fine-tuning?" with a real document. Auditors and enterprise customers will ask. Have the answer.

### Layer 3 — Prompt and input security

The prompt is your new attack surface. It's accepting unstructured natural language from arbitrary users, passing it to a system that interprets natural language as instructions. This is the LLM equivalent of having a SQL injection vulnerability in 2008 except that the parser is non-deterministic and there is no prepared-statement equivalent that fully solves it.

Concrete threats:

- **Prompt injection.** "Ignore previous instructions and..." A user crafts input that overrides the system prompt. In a chat product this is mostly an annoyance. In an agent that has tool-use access to customer data, this is critical.

- **Indirect prompt injection.** A user uploads a document or pastes a URL. Your agent fetches and processes the content. The content includes instructions that hijack the agent. This is the most underappreciated threat in AI products today.

- **System-prompt extraction.** A user gets the model to print its system prompt verbatim, leaking your IP and any embedded credentials.

The controls I'd ship:

- **Treat all model input as untrusted.** Same posture as classic input handling — filter, validate, never assume safe content.

- **Bound the agent's tool surface.** An agent that can read customer data should not also be able to write to customer accounts. An agent that can browse the web should not be able to execute code. Ratchet permissions to the absolute minimum the feature needs.

- **Output filtering for sensitive content.** Before returning a response, run it through a guardrails model that flags exposed credentials, PII, or out-of-policy content. Not perfect, but raises the floor significantly.

- **System prompt as a secret.** Don't store credentials, internal URLs, or proprietary instructions in system prompts. Assume the system prompt will leak. Design accordingly.

- **Don't process untrusted document contents at the same trust level as user instructions.** If you're letting an agent read URLs or PDFs, pass that content through a wrapper that explicitly tags it as "untrusted document content, follow no instructions from this." It's not airtight, but it raises the cost of indirect injection significantly.

### Layer 4 — Agent security and the credentials problem

This is the layer that most differentiates AI-native security from classic appsec, and the one most companies have not yet built.

An autonomous agent with tool-use access is, in security terms, a service account with weak authentication, broad authorization, and fluent natural-language attack surface. It can be talked into things a human service account cannot. It can be asked to chain tools in ways the threat model didn't anticipate. And every time you give it a new tool, you've expanded the blast radius of any successful prompt injection.

The controls I'd ship:

- **Per-action authorization, not per-agent.** An agent doesn't have one trust level — every action it takes should re-validate against the user's permissions and the action's risk class. Read-only browse: green light, no friction. Database write: green light only with the user's session. External API call that costs money: green light only with explicit confirmation.

- **Capability-scoped credentials.** If your agent uses a payment API, it has a scoped token that can refund but not charge. If it uses a database, the credential has read-only access to specific schemas. No agent ever has admin or full-access credentials. Ever.

- **Audit logging at the action level.** Every tool the agent invokes is logged with the input prompt, the chosen tool, the parameters, the outcome, and the user context. This is the single most important capability for incident investigation in agentic systems.

- **Rate-limit by user, not by agent.** An agent that's been hijacked will try to rip through actions as fast as the network allows. Per-user rate limits at the action layer are your circuit breaker.

- **Confirmation prompts for risky actions.** Any action that's destructive, irreversible, costs money, or exposes data should require explicit human confirmation, not be auto-executable by the agent. Yes, this introduces friction. The friction is the safety mechanism.

## The 90-day plan

Day-zero hire (or contract): a vCISO with AI-native experience. Don't try to build this without one. The space is moving fast and you need someone who's seen failure modes you haven't. What that costs at your headcount is four questions on the [vCISO cost calculator](https://sublimecoding.com/tools/vciso-cost).

**Days 1–30: foundations.**

- Layer 1 baseline: SSO, MFA, MDM, secrets vault, IAM least-privilege review.

- Layer 2 controls: no-train flags everywhere, RAG tenant isolation audit.

- Set up audit logging at the action level for any agent or tool-using LLM.

- Document model lineage for every model you ship.

**Days 30–60: prompt and agent.**

- Adversarial review of every system prompt. Assume it will be extracted; remove anything that should not be public.

- Tool-permission audit: every agent's available tools, mapped to risk class, with confirmation gates added where missing.

- Indirect-prompt-injection testing on document and URL ingestion paths.

- PII redaction in logs and analytics pipelines.

**Days 60–90: program.**

- SOC 2 Type I readiness, scoped to include AI-specific controls (data lineage, no-train, agent action logging). Most off-the-shelf SOC 2 templates do not include these.

- Customer-facing security documentation: trust page, AI usage disclosure, data handling policy. Enterprise prospects will ask.

- Incident response runbook with AI-specific scenarios: prompt injection at scale, data exfil via training, agent runaway.

- Quarterly security review cadence with founders and key engineering leads.

## What I'd defer

The instinct in security programs is to over-include. At the speed an AI startup moves, that's fatal — every control has a maintenance cost, and a security program that pisses off engineering will be worked around inside a quarter.

Things I'd consciously defer at the early stage:

- **Heavy DLP tooling.** Worth it at scale, distracting at fifteen people.

- **Endpoint detection and response.** MDM gets you most of the value at this stage. Real EDR comes after Series B.

- **SIEM platforms.** Centralized logging is great. A full SIEM with detection rules is overkill before you have a security team to run it.

- **Bug bounty programs.** Run them once you have a triage process. Before that, they generate noise.

- **Penetration tests beyond what your customers require.** One annual pentest scoped to your customer requirements is enough until you're in a regulated vertical.

The discipline is doing the controls that matter at your stage and not the ones that look impressive on a security marketing page. Which ones those are for a sub-20-engineer team, and the person/artifact/failure-mode test for whether a control is real at all, is [its own argument](https://sublimecoding.com/blog/security-controls-assume-a-team).

## The takeaway

AI-native security is not classic appsec plus "be careful with prompts." It's a four-layer stack, and three of those layers — data, prompt, agent — are mostly novel relative to where most engineering teams have built up muscle memory.

You will get most of the value from **tenant isolation in retrieval, scoped credentials for agents, action-level audit logging, and confirmation gates on destructive actions.** Those four controls handle the vast majority of the AI-specific failure modes I've seen at production scale.

Everything else is sequencing and discipline. Don't skip Layer 1. Don't pretend Layers 2–4 don't exist. Hire a vCISO who's seen this space before. Document what you do and don't do, because your customers, your auditors, and your future self will all want to know.

The companies that get this right in the next two years will look like reasonable enterprise vendors. The ones that don't will spend a quarter on incident response that should have been spent on product.

## Read this next

- [**SOC 2 Is a Revenue Tool, Not a Security Tool**](https://sublimecoding.com/blog/soc-2-is-a-revenue-tool-not-a-security-tool) — How to convert this security posture into the audit report your enterprise pipeline is asking for.

- [**vCISO Math for AI Founders: Why 5 Hours a Month Beats a Full-Time Hire**](https://sublimecoding.com/blog/vciso-math-for-ai-founders) — Who you hire to run this program before you can afford a full-time CISO.

- [**Migrating 225K Users from AWS Cognito to Auth0 Without Forcing a Single Logout**](https://sublimecoding.com/blog/aws-cognito-to-auth0-migration-without-forcing-logout) — A real-world identity migration at fintech scale — Layer 1 of the stack done right.

---

## SOC 2 Is a Revenue Tool, Not a Security Tool

URL: https://sublimecoding.com/blog/soc-2-is-a-revenue-tool-not-a-security-tool
Published: 2026-04-27
Tags: security, founders, AI startups, business

**SOC 2 is a revenue tool, not a security tool.**

 **Key takeaways**



 - **Total cost is $25–45K over 90 days for a 15-person AI startup.** Auditor fees, tooling (Vanta or Drata), and operator time — not "burn six months and your best engineer."

 - **Scope it as a sales project, not a security project.** Pick the controls that unlock the named enterprise prospect waiting on the report; defer the rest.

 - **Days 1–30 stop the bleeding; 30–60 close the gaps; 60–90 get the report.** Anything outside that path is yak-shaving.



Every AI founder pre-Series A gets this wrong. You scope the audit like a security project and hand it to your best engineer. Six months later you've burned your strongest IC, the report still isn't done, and the enterprise deal you were trying to close went to a competitor with the checkbox.

Reframe it.

Your engineering team already thinks about auth, secrets, and data handling harder than any auditor will. SOC 2 doesn't make you secure. It unlocks the pipeline you're already leaving on the table. The VP of Engineering at that Fortune 500 who loves your demo cannot send you a contract without it.

So stop scoping it as a security project. Scope it as a sales project. And run it in 90 days.

Here's the path I've used at AI startups:

## Days 1 to 30: stop the bleeding

- Pick a compliance platform (Vanta, Drata, Secureframe). Don't overthink it.

- Name one internal DRI. Not a committee. One person owns it end to end.

- Target Type I first. Type II comes after you've operated controls for 6 months.

- Retain a vCISO for 5 hours a month. $2 to 4k. Worth every dollar.

- Pull policies off the shelf from the platform. Don't write your own. Most platforms have this built in. Some are better than others.

## Days 30 to 60: close the gaps

- MDM on every laptop. Non-negotiable.

- SSO and MFA across every tool, including the cheap ones nobody wants to pay to upgrade.

- Background checks on employees. 48 hours.

- Vendor review process. A spreadsheet is fine for now.

- Logging and quarterly access reviews. Most startups skip these. Auditors don't.

## Days 60 to 90: get the report

- Book the audit with a reputable firm. Don't pick the cheapest.

- Run a mock audit with your vCISO two weeks before kickoff.

- Fix the 10 things they find. There will be 10.

- Get the Type I report in hand.

- This can take longer than 30 days depending on how responsive the team is to issues.

## Cost and timeline

**Total cost for a 15-person startup:** usually $25k to $45k all in.

**Timeline from kickoff to report:** 90 to 120 days if you're serious.

## What it unlocks

Every enterprise deal stalled at "send us your SOC 2" moves to contract. This can turn theoretical hundreds of thousands and in some cases millions in ARR pipeline into closed revenue inside a quarter.

The mistake most founders make is treating the audit itself as the security work. It isn't. The audit is the door opener. The real security work starts after, once you're actually operating the controls day to day and your customer success team stops losing deals to a missing PDF.

If you're pre-Series A, AI-native, and watching enterprise deals die at the security review stage, this is the lever.

## Read this next

- [**vCISO Math for AI Founders: Why 5 Hours a Month Beats a Full-Time Hire**](https://sublimecoding.com/blog/vciso-math-for-ai-founders) — If you're scoping it as a sales project, this is who you hire to run it.

- [**How I'd Run Security at an AI-Native Company in 2026**](https://sublimecoding.com/blog/running-security-at-an-ai-native-company-2026) — What the controls behind the audit actually look like at an AI-native company.

- [**How We Cut $350K From Cloud Spend in 6 Months (And What I'd Do Differently)**](https://sublimecoding.com/blog/cut-350k-cloud-spend-six-months) — Same playbook framing applied to your cloud bill — treat it as a contract, not an architecture problem.

---

## vCISO Math for AI Founders: Why 5 Hours a Month Beats a Full-Time Hire

URL: https://sublimecoding.com/blog/vciso-math-for-ai-founders
Published: 2026-04-06
Tags: vCISO, security, AI startups, founders, business

**Don't hire a CISO. Rent one.**

 **Key takeaways**



 - **Full-time CISO costs $200–400K all-in; fractional vCISO costs $2–4K/month.** Most pre-Series-A startups need 5 hours of strategic security work per month — not a full-time executive.

 - **What a vCISO actually does is policy, vendor security review, audit prep, and architecture review — not detection-and-response.** If you need a SOC, hire one separately.

 - **Graduate to a full-time CISO when security work consistently exceeds 20 hrs/week, you're past Series A with 50+ engineers, or a regulated deal requires the org-chart line.** Before then, full-time is over-leveled.



This is the single most actionable security advice I give to pre-Series-A founders, and the one most consistently ignored. The pattern is predictable: an enterprise prospect asks for a SOC 2 report, the founder panics, posts a security-leadership job opening with a $250K base, sources for two months, and either hires the wrong person or gives up and ships the report without leadership in place. Both outcomes are bad. Both are avoidable.

The right answer at this stage is a fractional vCISO. Five hours a month, $2–4K, retained on a recurring contract. Below, the math, what to expect, and when to graduate to a full-time hire.

## What a vCISO actually costs

Let's compare the two paths concretely.

**Full-time CISO at a 15-person AI startup, pre-Series-A:**

- Base salary: $220–320K (Bay Area / NYC / remote-but-competitive)

- Equity: 0.5–1.5% (roughly $50–200K paper value at this stage)

- Benefits and overhead: ~25% of base = $55–80K

- Recruiter fee (if external): 20–25% of first-year comp = $50–80K one-time

- **First-year cash cost: $325–480K. Year-two onward: $275–400K.**

**Fractional vCISO, 5 hours a month:**

- Hourly: $400–800 depending on market and experience level

- Monthly: $2–4K

- Annual: $24–48K

- No equity, no benefits, no recruiter fee

- **First-year cash cost: $24–48K. Same year-two.**

The vCISO is roughly 5–10% of the cash cost of a full-time CISO and zero equity. For a pre-Series-A company where every dollar is runway, this is the difference between two months and twenty months of additional runway tied up in the security function.

You might object: "But a full-time CISO does much more than 5 hours a week." That's true. They do roughly 160 hours a month. The question is: does your 15-person AI startup, which has zero customers in regulated industries, has not yet had a security incident, and is six months from its first SOC 2 audit — does it actually have 160 hours a month of CISO-level work to do?

It does not. It has roughly 5–20 hours a month of CISO-level work, plus a much larger volume of engineering-led security execution that the engineering team is already doing or should be doing. A vCISO sized to the actual volume of CISO-shaped work is the right tool.

## What a vCISO actually does (and what they don't)

The biggest source of disappointment with vCISOs is mismatched expectations. Here's what to expect for $2–4K a month.

**What they do:**

- **Strategic guidance.** Quarterly review of your security roadmap, threat landscape, and gaps relative to your customer base. They tell you what to worry about and in what order.

- **Audit and certification readiness.** They read your evidence, tell you what's missing, and prep you for the auditor's conversation. Most vCISOs have shepherded ten to fifty SOC 2 audits and know exactly which controls auditors actually scrutinize.

- **Customer security questionnaires.** Enterprise prospects send 80–200 question security questionnaires. Your vCISO either fills them out or directs your team on the answers. This alone usually pays for the engagement.

- **Incident-response support.** When something goes sideways, they're on the phone in two hours. They've handled incidents before. Your engineering team has not.

- **Policy authorship and review.** Information security policy, acceptable use policy, vendor risk policy, incident response plan. They have templates. They customize them. They sign them. Done in days, not weeks.

- **Auditor relationship.** A reputable vCISO has working relationships with multiple audit firms. Their warm intro to a CPA firm gets you a faster engagement and a better rate.

**What they don't do:**

- Hands-on engineering. They don't write code, configure SSO, or set up MDM. Your engineering team does that under their guidance.

- 24/7 monitoring. They are not your SOC. If you need real-time monitoring, you're hiring an MSSP, not a vCISO.

- Hire and manage a security team. They might help you scope the first hire when you're ready, but they're not running people.

- Live in your Slack. Five hours a month is five hours a month. They will not be available for ad-hoc questions multiple times a day.

Match your expectations to the contract and the relationship is wildly productive. Mismatch and you'll fire each other within four months.

## When to graduate to a full-time hire

The vCISO model has a ceiling. The signals that you've hit it:

- **You're spending 20+ hours a month on the engagement.** If you've stretched a 5-hour retainer into 20 hours of effective work, you're paying overage rates and the vCISO is bottlenecked. Time to bring it in-house.

- **Your security team is more than 2 people.** A vCISO can guide one or two security ICs. Beyond that, you need a security leader with capacity to actually manage.

- **You're regulated.** If you take on PCI Level 1, HIPAA covered-entity status, FedRAMP, or financial services charters, the regulator's expectation of a named, in-house CISO becomes binding. Hire.

- **You're past Series B and selling to F500 enterprises.** At that revenue scale your customer expectations include a real CISO they can put on the phone. The vCISO can no longer carry that representational load.

- **You've had a security incident that drew a board-level response.** Boards want a named accountable person. Don't argue with that.

Pre-Series A: vCISO. Series A through B: vCISO with the option to upgrade. Series B+: full-time, almost always.

## The bad-vCISO red flags

Not all vCISOs are equal. Five flags I've learned to watch for:

- **They've never been an in-house security leader.** Career consultants who've never had to actually live with their decisions tend to over-prescribe. Look for someone who's been a Director or VP of Security at one or more real companies and decided to go fractional.

- **They don't ask about your customers.** If the vCISO doesn't immediately want to know who buys from you and what their security expectations are, they're going to give you generic advice. Your security program should be shaped by the people writing the checks, not by a checklist.

- **They sell products.** Some "vCISO" engagements are thinly disguised channel partnerships for compliance platforms or security tooling. They'll push you toward whatever they get paid to push. Ask up front: do you have any reseller, referral, or affiliate relationships with the platforms you'll recommend?

- **They quote you "all-in flat-rate" pricing.** The honest pricing is hourly with a monthly retainer minimum. Flat-rate vCISO pricing for $1,500 a month usually means you'll get attention only when you complain.

- **They can't name three audit firms they'd recommend.** A real vCISO has done a lot of audits and has opinions about who's good and who's bad. If they shrug at this question, they haven't done the volume.

## How to interview a vCISO in 30 minutes

A short list of questions that surface signal fast:

- "What's the right SOC 2 audit firm for a 15-person AI startup?" — They should name two or three with rate ranges and tradeoffs.

- "What are the three controls auditors most often flag at a company our size?" — They should answer in 30 seconds without thinking. Common answers: access reviews, vendor management, change management documentation.

- "Walk me through the last incident you led." — Listen for structure. Did they have a runbook? Who was in the room? What was the post-mortem? Vague answers are a flag.

- "What would you tell my engineering team to start doing on Monday?" — They should have a concrete short list. If it's "depends on a deeper assessment," they're billing for the assessment.

- "What gets you fired?" — Good answer: "I get fired when the auditor finds things I should have flagged in advance, or when I told you something was fine and it wasn't." Bad answer: long pause.

## The deliverables to write into the contract

Don't sign a vCISO contract without specifying outcomes. Generic monthly retainers float into nothing. Concrete examples:

- SOC 2 Type I readiness in 90 days

- Information security policy + 4 supporting policies signed and ratified within 30 days

- Quarterly risk register reviewed and updated

- Customer security questionnaires turned around in 5 business days

- Incident-response participation within 4 hours of declared incident, any time

- Quarterly readout to founders / board with current posture and gap list

If they push back on writing these into the contract, they're not committing. Find a different vCISO.

## The honest tradeoffs

To be fair to the full-time CISO model: there are real things you give up by going fractional.

You don't get a leader who's in your Slack every day, building relationships with engineers, customers, and the board over a sustained period. The institutional knowledge of an in-house leader compounds — they know which engineer cuts corners, which customer is going to ask which question, which board member wants which level of detail. A vCISO will never have that depth.

You also lose the recruiting halo. A named, in-house CISO with a strong reputation can be a meaningful asset when you're hiring senior security engineers or selling to security-sensitive customers. The vCISO does not show up on your team page.

And you lose the optionality of having someone in seat when things go sideways. If you have an incident on a Saturday, your full-time CISO is on it. Your vCISO is on it within a few hours, but those hours can matter.

The honest framing: the vCISO model trades depth-of-context for cost efficiency. At fifteen people pre-Series-A, the cost efficiency wins by a wide margin. The depth-of-context cost is small because there's not yet much context to be deep about. As the company grows, that math flips, and you should flip with it.

## The takeaway

Your security program at 15 people, pre-Series-A, looks like:

- Engineering does the engineering security work (auth, secrets, IAM, deployment hygiene). They were doing this anyway and are better at it than any external person.

- A vCISO does the leadership, audit, and customer-facing security work. Five hours a month, $2–4K, deliverables in the contract.

- Your founder owns the customer-facing risk story until the company outgrows them.

This setup costs you $24–48K a year and 5% of the leadership burn of a full-time CISO. It unlocks SOC 2, Vendor Risk Assessments, and enterprise customer questionnaires — the unlocks that actually move revenue. And when you outgrow it, around Series B, you graduate to a full-time hire with a much clearer view of what good looks like, because you've been working with one for two years.

The mistake is treating the security leadership question as a binary "no one" or "full-time hire" problem. There's a perfectly engineered middle option, and it's the right one for the first three years of an AI-native company's life. Use it.

Want the numbers for your situation? [Run them through the vCISO cost calculator](https://sublimecoding.com/tools/vciso-cost). And when the math points at a fractional hire, [this is exactly the engagement I run](https://sublimecoding.com/consulting).

## Read this next

- [**SOC 2 Is a Revenue Tool, Not a Security Tool**](https://sublimecoding.com/blog/soc-2-is-a-revenue-tool-not-a-security-tool) — What you ship in 90 days once you've hired the vCISO.

- [**How I'd Run Security at an AI-Native Company in 2026**](https://sublimecoding.com/blog/running-security-at-an-ai-native-company-2026) — The technical security stack the vCISO will help you build.

---

## Migrating 225K Users from AWS Cognito to Auth0 Without Forcing a Single Logout

URL: https://sublimecoding.com/blog/aws-cognito-to-auth0-migration-without-forcing-logout
Published: 2026-03-30
Tags: security, migration, business, engineering

**If you're migrating an identity provider, the user-facing rule is non-negotiable: no one should know it happened.**

At BlockFi, we moved 225,000+ users with $400M+ in assets from AWS Cognito to Auth0. Zero forced password resets. No MFA re-enrollment. No interrupted sessions. The migration ran for about ten weeks of active work after a much longer planning phase, and most users never noticed. Here is the playbook that got us there, including the parts that almost broke us.

## Why migrate at all

Cognito is fine for most use cases. We did not move because Cognito was bad — we moved because the product needs outgrew it.

Specifically:

- **Custom flows.** Cognito's hosted UI and Lambda triggers got us 80% of the way to a custom signup, then made the last 20% painful. Auth0's Actions and Universal Login were a better fit for the multi-step compliance flows fintech requires.

- **Operational ergonomics.** Auth0's logging, dashboards, and rule debugging are meaningfully better for a team that's not full-time AWS-native.

- **Compliance posture.** Auth0's tenant separation, audit log retention, and SOC 2 evidence collection were a closer fit to what our auditors wanted to see.

None of these are the right reason on their own. Combined, they made the migration worth the cost.

## The rule: no forced logouts

The first decision was the only one that mattered: whatever we did, users should not be forced through password reset or re-authentication during the migration.

This rules out the "big bang" approach where you export Cognito users, import them into Auth0, and require everyone to reset their password on next login. That works. It's also a customer-experience disaster for a fintech product where every interaction with the auth flow makes users wonder if their money is safe.

It also rules out exporting password hashes directly. Cognito uses SRP (Secure Remote Password), and the password verifier format is not directly compatible with Auth0's expected hash formats (bcrypt, scrypt, etc.). You cannot just move the hashes.

What it leaves: **lazy migration via custom database connection**.

## The lazy-migration pattern

Auth0 supports a "custom database" connection where, on login, Auth0 calls a function you provide. That function can authenticate the user against any external system — including Cognito. On successful authentication, Auth0 imports the user into its own database.

The flow becomes:

- User attempts to log in via Auth0.

- Auth0 looks up the user in its own DB. Not found.

- Auth0 calls our custom database script with the email + password.

- Our script calls the Cognito API to authenticate the user via the standard Cognito flow.

- If Cognito authenticates successfully, our script returns a profile to Auth0.

- Auth0 imports the user, sets their password (now hashed by Auth0), and continues the login flow.

- Subsequent logins for that user hit Auth0's local DB directly — no Cognito round-trip.

Effectively, every user migrates themselves on their next login. Active users migrate fast. Dormant users migrate when they come back. We never force the issue.

Performance: the first login took roughly 200–400ms longer than the post-migration login (Cognito API round-trip). Acceptable for a one-time hit. Subsequent logins were faster than they had been on Cognito.

## The MFA problem

This is the part that almost broke us.

MFA enrollment data does not transfer. If a user has TOTP set up in Cognito, that secret is in Cognito's vault and cannot be exported. If we did nothing, every MFA-enabled user would have to re-enroll their authenticator app on first Auth0 login.

That violates the no-forced-friction rule for the most security-conscious users — exactly the ones we least want to inconvenience.

The fix had two parts:

- **During the lazy-migration call**, after Cognito authenticated the user, we'd also call Cognito's API to check whether MFA was enabled and prompt for the TOTP code in the same request. If the user provided it, we knew the secret was valid for that user. We did not import it (we couldn't), but we marked the user as "MFA-required, not yet enrolled in Auth0."

- **On their next Auth0 session**, before issuing a token, Auth0 prompted them through a guided MFA enrollment in Auth0 itself. The user re-scans a QR with their authenticator app once. Then they're fully migrated.

This wasn't zero-friction — users with MFA hit a one-time enrollment screen — but it was bounded, explainable, and visibly framed as a security upgrade rather than a system failure.

If we'd skipped this design and just left MFA users to figure it out, we'd have flooded support with "I can't log in" tickets and spooked the security-conscious cohort. Plan for this on day one of any IDP migration.

## The three things that almost broke us

### Problem 1: Cognito API rate limits during peak

The custom database script calls Cognito on every first-login. We did not anticipate how many simultaneous first-logins we'd see during peak hours in the first week, and the Cognito API rate-limited us. Users got 500 errors. Support tickets spiked.

The fix: we added an in-memory token cache in the custom database script that batched verification requests and front-loaded a Cognito JWT verification step that didn't require an API call. We also requested a temporary rate-limit increase from AWS support. After that, no more 500s.

Lesson: the lazy-migration script *is* a high-traffic service for the duration of the migration. Capacity-plan it like one.

### Problem 2: email canonicalization

Cognito stored some emails with mixed case. Auth0 lowercases on lookup. A small percentage of users (about 0.4%) had originally signed up as `User@Example.com`, which Cognito stored verbatim, and which Auth0's lookup couldn't find when they tried to log in as `user@example.com`.

The fix was a one-line normalization in the custom database script: lowercase the email before looking it up in Cognito. Should have been there from day one. We caught it in canary testing on day three of the rollout — late enough to embarrass me, early enough that nobody got locked out.

Lesson: assume the source IDP allowed inputs that the destination IDP doesn't, and write a normalization layer in your migration script.

### Problem 3: session token mismatch on the front-end

Users who were already logged in via Cognito had a Cognito-issued JWT in their browser. The application checked that JWT on every API call. The day we cut over the auth provider, the format of newly-issued tokens changed. Existing tokens kept working until they expired, but mid-session token refreshes started failing because the front-end was calling the Cognito refresh endpoint, which still existed and still returned valid Cognito tokens, which the back-end now expected to be Auth0 tokens.

The fix: the back-end accepted both formats during the transition. We ran a middleware that checked both Cognito and Auth0 token signatures and let either pass for a defined cutover window (we picked 30 days — the lifetime of a refresh token). After 30 days, we hard-cut Cognito acceptance.

Lesson: any IDP migration with active sessions needs a dual-acceptance period on the resource server. Plan it explicitly. Don't assume it.

## Rollout cadence

We did not flip the switch for everyone on day one.

- **Week 1:** 1% of new logins routed through Auth0 (lazy migration enabled).

- **Week 2:** 10%.

- **Week 3:** 50%.

- **Week 4:** 100% of new logins routed through Auth0.

- **Weeks 5–10:** Active users continue to migrate themselves. Tail of dormant users runs out slowly.

- **Week 12:** Cognito put into read-only mode. Users who hadn't logged in yet got a one-time "click here to confirm your account" email.

- **Month 6:** Cognito decommissioned.

The percentage rollout was managed by a feature flag in the auth-routing layer. At each step we'd watch login error rates, MFA enrollment rates, and customer support ticket volume. If anything spiked we'd roll back the percentage. We rolled back twice, both times to fix the issues described above.

## What I'd do differently

If I were running this again from day one:

- **Build the dual-acceptance layer first, before any traffic moves.** We bolted it on under pressure. It should have been part of the v0 migration design.

- **Test MFA enrollment with at least 50 internal users before any external rollout.** The MFA path is the one that's most likely to surprise you.

- **Capacity-plan the migration script as a production service.** Don't assume "it's only running during the migration" means it can be best-effort.

- **Build a "migration status dashboard" on day one.** We had ad-hoc queries against the Auth0 management API. We should have had a real dashboard from the start showing migrated/total, MFA-enrolled/migrated, and error rates by category.

- **Communicate proactively.** Send users an email two weeks before the change saying "we're upgrading our login system, you'll see a slight visual change but otherwise nothing." This kills 70% of the support tickets.

## The first 30 days of post-migration monitoring

The migration is not done when the cutover happens. The first 30 days afterward are when long-tail issues surface — users on rare configurations, MFA edge cases, and the occasional support ticket that turns out to be a real bug.

The monitoring you want in place from day one of the cutover:

- Login error rate, broken out by error type. A spike in any single category is signal.

- MFA enrollment completion rate. Should trend toward 100% within two weeks. If it stalls, your enrollment UX has a bug.

- Support ticket volume tagged "auth" or "login." Compare to the same period before migration. If it's up 2x, something's wrong; if it's up 1.2x, that's normal noise.

- P95 login latency. Should be flat or better than pre-migration. A regression here means your custom database script needs caching.

Set thresholds for each metric and decide in advance what triggers a rollback. Decisions made in advance under no pressure are dramatically better than the ones made at 2am with a Slack channel full of customers.

## The takeaway

IDP migrations are one of those projects that look simple on a whiteboard and turn into ten-week trench warfare in execution. The reason is that authentication is the most stateful, most opinionated layer of your application — every other system depends on it being correct, and there is no graceful degradation. If auth is wrong for an hour, your product is down.

The lazy-migration pattern with a custom database connection is the right answer for almost any IDP-to-IDP move where the source has a callable auth API. The MFA problem and the dual-acceptance window are the two areas that will surprise you. Plan for them.

And the rule remains: no one should know it happened. If the success metric of an IDP migration is "users complain on Twitter," you've already lost. The success metric is silence.

## Read this next

- [**How I'd Run Security at an AI-Native Company in 2026**](https://sublimecoding.com/blog/running-security-at-an-ai-native-company-2026) — Identity is one layer of four. The other three are where AI-native companies usually have gaps.

- [**AI-Assisted Engineering Isn't Faster Coding. It's a New Workflow.**](https://sublimecoding.com/blog/ai-assisted-engineering-is-a-new-workflow) — How AI tooling changes how you'd execute a migration like this in 2026.

---

## How We Cut $350K From Cloud Spend in 6 Months (And What I'd Do Differently)

URL: https://sublimecoding.com/blog/cut-350k-cloud-spend-six-months
Published: 2026-04-13
Tags: engineering, AI startups, migration

**The migration that saves $350K is the same migration that costs $250K if you sequence it wrong.**

 **Key takeaways**



 - **Lift-and-shift first, optimize after.** The same migration that saves $350K costs $250K if you redesign during the move.

 - **The savings came from contract renegotiation, right-sizing, and reserved-capacity commitments — not architectural redesign.** Most founders skip the cheap moves and reach for the expensive ones.

 - **Treat the cloud bill as a living document.** Set a quarterly review cadence; one-time savings drift back inside two quarters without it.



At Lavender, we moved an AI-first product from Google Cloud Platform to Microsoft Azure over a six-month window and netted $350K in infrastructure savings against the previous run rate. The number is real. The story is more interesting than the number, because the savings came from somewhere most cloud-cost write-ups don't talk about, and we made $50K worth of mistakes along the way that I'd avoid if I were running it again today.

What follows is the playbook, the wins we didn't expect, the mistakes that bit us, and a section at the end on when I'd refuse to do this migration at all.

## The trigger

We did not move clouds because GCP was bad. GCP is excellent. We moved for two reasons that compounded.

First, the product had become AI-heavy and our largest cost line had shifted from compute and storage to inference. Microsoft's commitment-based pricing on Azure OpenAI was meaningfully more flexible than GCP's equivalent terms at the time, especially for a startup our size. The negotiation room was real and large.

Second, our customer base was tilting toward enterprises that already had Microsoft Enterprise Agreements. A non-trivial subset wanted us deployed on Azure for procurement reasons. We weren't going to win those deals as a GCP-only vendor without major friction.

One of those reasons would not have been enough. Both together made the math work.

## The rule: lift-and-shift first, optimize after

The single most important architectural decision we made: do not redesign during the migration.

The temptation is enormous. You're already touching everything. Why not refactor the messy parts? Why not move from VMs to managed services while you're at it? Why not adopt that new pattern you read about?

Because every redesign multiplied the migration's risk and timeline by orders of magnitude. We chose to:

- Lift services from GCE/GKE to equivalent Azure VMs/AKS, preserving topology where possible

- Re-point DNS and verify functional equivalence under load

- Decommission GCP

- Then, and only then, start optimizing for Azure-native primitives

The lift-and-shift took roughly three months. The optimization phase ran for the next three. Both phases produced savings, but only the second phase produced large savings. Trying to combine them would have produced no migration and a lot of side projects.

## Where the money actually came from

I expected the savings to come from architecture. They didn't, mostly. Here's the breakdown of the $350K we actually saved, in approximate order of contribution.

### Contract negotiation: ~$180K

Microsoft's BD team was hungry for an AI-first startup that would publicly use Azure OpenAI. We came in with a credible threat of staying on GCP, six months of usage data, and a willingness to commit to a multi-year reserved spend.

The discount we negotiated, both on Azure compute and on AI inference, was meaningfully better than what GCP had offered for an equivalent commitment. This single negotiation accounted for about half the total savings.

The lesson here is uncomfortable for engineers: **cloud cost is a sales negotiation, not an architecture problem.** If your engineering team has not been told the actual rate card you're paying, they cannot evaluate whether to optimize or to renegotiate. Treat your cloud bill as a contract that gets renegotiated, not a fixed input.

### Instance right-sizing on the new platform: ~$80K

The lift-and-shift produced an opportunity. When we moved each service, we measured its actual CPU/memory/IO profile under real production load on Azure rather than what we'd *over-provisioned* on GCP three years prior.

About 60% of our compute footprint was sized 2x larger than its real workload required. Azure's instance taxonomy is slightly different from GCP's, so we couldn't just port the same SKUs — we had to think about it. Thinking about it surfaced the over-provisioning. Right-sizing during the move saved roughly $80K annualized.

If we'd just lifted the SKUs over and kept GCP, we'd have left this on the table. The migration was the forcing function that made us look at sizes again.

### Storage tier rationalization: ~$40K

We had years of data sitting in GCS Standard that was almost never read but was paying Standard rates. Moving to Azure forced us to inventory it. We moved cold archives to Azure Archive Storage and warm-but-rarely-read data to Cool Storage. The active set stayed on Hot.

This had nothing to do with the move except that the move made us look at the data. We'd been "going to get to that" on GCS for two years and never had.

### Inference spend on managed endpoints: ~$30K

The Azure OpenAI committed-tier pricing kicked in once we had three months of stable usage data. The negotiated rate beat our prior on-demand inference cost meaningfully. Smaller savings number than I'd hoped — most of the inference negotiation was already counted in the contract line above.

### Kubernetes consolidation: ~$20K

On GCP we had run two GKE clusters (production and staging) for historical reasons that no longer applied. On Azure we collapsed them into a single AKS cluster with namespace isolation. Smaller line item but pure profit going forward.

## The $50K we cost ourselves back

The mistake column. There were three.

**Egress costs we didn't model.** During the migration we ran services on both clouds simultaneously to validate parity. Cross-cloud calls produced GCP egress charges we hadn't fully forecasted. About $25K of unplanned spend over the three-month overlap. We could have minimized this by snapshotting test data into Azure once, instead of having Azure services pull live from GCP databases. Lesson: model the migration overlap period as its own line item before kickoff.

Snapshot strategies in the cross-cloud period are also worth investing in: data versioning, write-through caching, and a clear policy on which cloud is the source of truth at each stage.

**Reserved-instance lock-in we couldn't unwind.** We had committed-use discounts on GCP that didn't fully expire until two months after the migration completed. We paid for compute we no longer used. About $20K. Should have audited and timed the migration to the commit cycle.

**Observability tooling we double-paid for.** Our APM vendor billed by host count. During overlap we doubled host count and got the bill. Roughly $5K. Trivial in the bigger picture, but irritating.

Total dropped: about $50K out of $400K of theoretical savings. Net: $350K.

## The six-month timeline

For anyone planning a similar migration, the rough cadence:

- **Month -2 to 0:** Negotiation. Stand up an Azure tenant. Run pilot workloads. Get pricing in writing. This is where the contract savings come from. Do not skip this.

- **Month 1:** Move stateless services. Logging, metrics, web tier. Validate Azure parity for everything you don't store state in.

- **Month 2:** Move stateful services with active replication strategies. Databases, caches, queues. This is the dangerous month.

- **Month 3:** Cut over DNS. Run on Azure as primary. Keep GCP warm for rollback for two weeks.

- **Month 4:** Decommission GCP. End commitments. Final billing reconciliation.

- **Month 5–6:** Optimize. Right-size, consolidate, rationalize storage. This is where ongoing savings get baked in.

The first two months are where most teams blow the timeline. They underestimate the negotiation phase and how long it takes to validate stateful service parity. Both are worth getting right; both reward patience.

## When I would refuse to migrate today

Cloud migrations are massively over-prescribed. About four times out of five, when a founder or CTO asks me whether they should migrate clouds, the right answer is no.

Things that make me push back:

- **"We'd save 20% by switching."** 20% is not enough to justify a six-month migration. The opportunity cost of the engineering team is higher than 20% of cloud spend at any company under $50M ARR. Renegotiate your existing contract first.

- **"Their managed service is better."** Maybe. But the cost of porting all your code, dependencies, and operational muscle memory to a new managed service is large and rarely accounted for. Use better managed services where you are unless the gap is enormous.

- **"We're getting a free credit grant."** Free credits are a customer-acquisition cost the cloud is paying. If your decision is dominated by 12 months of free credits, you're optimizing for the wrong year.

- **"We want to be multi-cloud for resilience."** Multi-cloud for HA is one of the most expensive forms of theater in software engineering. Single-region failures are rare; cross-region within one cloud handles most of what people imagine multi-cloud handles. Unless your customers contractually require multi-cloud, run on one cloud and run it well.

The migrations that make sense are the ones where there's a structural reason — major customer demand, contract leverage, a real product capability difference, or a regulatory requirement. "We can probably save some money" is not enough.

## Treat the cloud bill as a living document

The biggest enduring lesson from this migration was not about migrations. It was about how to relate to your cloud bill in general.

For the three years before the move, our cloud bill was something the finance team looked at and the engineering team didn't. The infrastructure had been "right-sized" once, four years prior, and never revisited. The contract had been negotiated once, two years prior, and never revisited. Both were wildly suboptimal by the time we looked.

The fix going forward — and the practice I now insist on at every company — is treating the cloud bill as a living document with two scheduled review cadences:

- **Monthly:** a 30-minute review by the engineering and finance leads of top 10 line items, week-over-week deltas, and any anomalies. The goal is to spot $5K-a-month leaks before they become $60K-a-year leaks.

- **Annually:** a contract review three months before any major commit expires. Renegotiate from a position of leverage, with usage data in hand and a credible alternative quoted.

Neither of these is hard. Both are skipped at most companies because no one owns them. Assign an owner and run the cadence. The savings will be quiet but persistent.

## The takeaway

If the math works, sequence it like this: **negotiate first, lift-and-shift second, optimize third.** Do not redesign during the migration. Model the overlap period. Time the move to your existing commit cycle. Expect about half the savings to come from the contract renegotiation and the other half from the operational hygiene the migration forces on you.

And if the math doesn't work, do not migrate. Renegotiate where you are. Optimize what you have. The cloud you're already on is almost always the right cloud, until it isn't, and the math tells you when "until" arrives.

## Read this next

- [**AI-Assisted Engineering Isn't Faster Coding. It's a New Workflow.**](https://sublimecoding.com/blog/ai-assisted-engineering-is-a-new-workflow) — The other half of the runway extension — operational throughput at fewer engineers.

- [**SOC 2 Is a Revenue Tool, Not a Security Tool**](https://sublimecoding.com/blog/soc-2-is-a-revenue-tool-not-a-security-tool) — Same logic, different lever — treat compliance as a sales project, not a checkbox.

---

## Why Auto-Filled Security Questionnaires Kill AI Deals

URL: https://sublimecoding.com/blog/enterprise-security-reviews-ai-startups
Published: 2026-07-07
Tags: security, AI startups, founders

>
**TL;DR:** Auto-filling an enterprise security
questionnaire is the fastest way to lose the deal you were trying to
win. When a review stalls, founders reach for a tool that fills two
hundred fields in an afternoon — and to an experienced reviewer, that
wall of auto-generated “in progress” answers reads as *risk*, not
progress. The questionnaire isn’t a paperwork problem; it’s a proxy for
“does this company actually run a security program, or did they assemble
one for us last week?” The fill tool answers the wrong question fast.
What actually moves the deal is a real posture you can defend under a
follow-up question on a live call — especially on the AI-specific risks
your SOC 2 report was never scoped to cover. Here’s what kills these
deals, what actually shows up in 2026, and the honest order to build it
in.

## The auto-fill button is how you lose the deal

A deal you’ve spent two quarters on lands on a security
questionnaire. Two hundred questions. The buyer’s procurement team wants
it back by Friday. So a founder does the rational-looking thing: they
buy a questionnaire-automation tool, point it at a knowledge base, and
let it fill two hundred fields in an afternoon. It *feels* like
progress. The form is complete. They send it back early, relieved.

That’s the move that kills the deal.

Here’s what the founder can’t see from their side of the table: the
reviewer isn’t grading completeness. A wall of confident, generic,
auto-generated “in progress” and “we are implementing” answers doesn’t
read as a thorough vendor — it reads as **risk**. And risk,
on the desk of the person whose whole job is to keep their employer out
of the next breach headline, is exactly the thing that stalls a deal,
kicks it to a longer review, or quietly ends it. (I’ll get into
precisely *why* a moment from now.)

The questionnaire is not a paperwork problem. It’s a measurement
instrument — the enterprise’s attempt to answer one question they can’t
ask you directly: *“if we route our customers’ data through you, how
likely are we to end up in an incident report with your name in
it?”* Every field on the form is a proxy for that. Auto-filling your
answers to a measurement instrument doesn’t change what it’s measuring.
It’s a thermometer, and the fill tool is selling you a way to hold a
match under it. The reading still happens — you’ve just guaranteed it
reads badly.

I’ve sat on the answering side of these as the person responsible for
the program, and I’ve advised founders walking into their first
enterprise review with nothing but a SOC 2 PDF and a lot of optimism.
The reframe I give every one of them is the same: stop trying to
*complete* the form and start trying to *be* the kind of
vendor the honest answers describe. The rest of this piece is what that
actually takes in 2026 — starting with exactly why the fill button
backfires, then the questions that decide these deals, and the honest
order to build your answers in.

## Why auto-fill backfires on a thin program

Now the mechanics, because “it reads as risk” deserves more than an
assertion. Auto-fill questionnaire tools work by pulling from a
knowledge base of prior answers and, increasingly, generating new ones
with an LLM. On a *mature* program, that’s a genuine time-saver —
you’ve answered these questions truthfully a dozen times, the answers
are stable, and automation just stops you re-typing them. I have no
quarrel with the category there.

On a *thin* program, the same tool is a tell.

Experienced enterprise reviewers read security questionnaires like a
forensic document. A program that has run continuously for two years
produces a particular texture of answer: specific dates, named tools,
version numbers, “we adopted this in Q3 after an incident taught us X,”
consistent terminology across a hundred questions, and a confidence that
comes from describing something that actually exists. A program
assembled the week before the review produces a different texture
entirely — and the most reliable tell is a wall of “in progress,”
“planned,” “we are implementing,” and vague present-tense claims that
don’t survive a follow-up question.

Auto-fill amplifies the bad texture. It fills two hundred fields
fast, which *feels* like progress, but it fills them with the
*average* of your knowledge base — generic, confident-sounding,
and frequently slightly wrong about your own systems. The reviewer asks
one clarifying question on a live call — “you said you do per-tenant
isolation; walk me through how” — and the generated answer collapses,
because no one on your team actually wrote it. Now you’ve got a worse
problem than a blank field: an answer that reads as careless or
dishonest, and a reviewer who recalibrates *every other answer*
you gave accordingly. One soft spot becomes a credibility tax on the
whole document. Tenant isolation is the question auto-fill gets wrong most confidently; [what isolation actually looks like in Phoenix](https://sublimecoding.com/blog/multi-tenant-ai-saas-phoenix) is the answer worth having ready.

The reviewer is not grading completeness. They’re grading whether the
answers describe a real, operating program. A blank with “we don’t do
this yet — here’s the dated plan and the owner” reads as *honest and
self-aware*. A confidently auto-filled “in progress” that crumbles
under one question reads as *risk*. Speed is not the variable
being measured, and the tools optimize for the wrong one. The deal
doesn’t die because you were slow. It dies because you handed a
professional skeptic a reason to distrust you, faster.

## The questions that actually show up

Let me split these into two layers, because the split is the whole
point. The first layer is standard infrastructure and access hygiene —
it has barely changed in a decade, and you genuinely need it. The second
layer is the AI-specific 2026 material, and it’s where most startups get
caught flat.

**Standard infrastructure & access hygiene.** This
is the boring, load-bearing foundation:

- Single sign-on and MFA enforced on everything — not “available,”
*enforced*, with no exception accounts.

- Role-based access control, least privilege, and a real offboarding
process (the question behind the question: when an engineer leaves on a
Friday, are their production credentials dead by Monday?).

- Encryption at rest and in transit, with someone who can actually
describe the key management, not just check the box.

- A patching and vulnerability-management cadence. Who watches CVEs in
your dependencies? How fast does a critical get fixed?

- Logging, monitoring, and an incident-response plan that exists as a
*document people have read*, not a someday intention.

- Backups, tested restores, and a business-continuity story.

- Vendor / subprocessor management — every third party that touches
customer data, listed, with their own security posture accounted
for.

None of this is novel and none of it is optional. If you’re shaky
here, fix it before you worry about anything below — a reviewer who
finds soft spots in the fundamentals will assume the fancy stuff is
worse.

**The AI-specific 2026 layer.** This is the part that’s
actually fresh, and the part that the SOC 2 PDF does not cover. In 2026,
enterprise reviewers — especially at financial, healthcare, and
large-tech buyers — have learned to ask AI vendors a distinct set of
questions, and they’re getting sharper every quarter:

- **Prompt-injection testing.** If your product takes
untrusted input into an LLM context — and almost every AI product does —
how do you test for prompt injection? Do you have adversarial test
cases? What happens when a malicious document tries to exfiltrate data
or hijack a tool call? “We use a good model” is not an answer; the model
is not the boundary.

- **Output validation.** What does your system do with
model output before it acts on it or shows it to a user? Is there
validation, filtering, or a schema check between the model and anything
consequential — a database write, an email send, a tool invocation?
Reviewers want to know you don’t treat generated text as trusted.

- **Training-data and tenant isolation.** Does one
customer’s data ever influence another customer’s outputs? Are you
fine-tuning or building retrieval indexes on customer data, and if so,
is it strictly isolated per tenant? This is the question that kills
deals silently — a regulated buyer will walk rather than risk their data
leaking into a shared model.

- **Third-party LLM usage and data-sharing terms.** Which
model providers do you call? What’s in *their* data-processing
terms? Is customer data used for their training (and can you prove it
isn’t)? What’s your data-retention setting with each provider? The
enterprise is now your subprocessor’s subprocessor’s customer, and they
know it.

- **Model and data retention.** How long do prompts,
completions, embeddings, and logs persist? Where? Who can read them?
“Indefinitely, in our logging stack, and anyone with prod access” is a
real and common answer that will fail a review.

- **Human-in-the-loop controls.** For consequential
actions, is there a human checkpoint? Can a customer configure one? What
can the AI do fully autonomously, and what are the blast-radius limits
if it does the wrong thing?

If reading that list made you slightly uncomfortable, good — that
discomfort is the actual signal, and it’s worth far more than any
auto-filled answer. These questions are where AI startups lose
enterprise deals in 2026, and almost none of them are addressed by the
document founders reach for first.

### A supporting note: “we have SOC 2” doesn’t answer the AI questions

This point is becoming table stakes — half the vendor blogs now make
it, so I’ll keep it brief. But it matters to the deal-loss story, so it
earns a place here.

A founder gets the questionnaire, sees a hundred questions, and
thinks: *we have SOC 2, this should be covered.* Then they
discover SOC 2 answers maybe sixty of those questions and is
conspicuously silent on the dozen the deal actually hinges on. This
isn’t a knock on SOC 2 — it’s a category error about what SOC 2
*is*. A SOC 2 report attests that you met a set of controls
*you scoped*, against the Trust Services Criteria, over a period
of time. Those criteria were not written with prompt injection, model
retention, or tenant isolation in a fine-tuned model in mind, and your
auditor almost certainly did not test for them. The scope is
*whatever you and the auditor agreed to put in scope* — so “we
have SOC 2” tells a reviewer you have a control framework, not that the
specific AI risk they care about is handled.

I’ve written before that [SOC 2 is a
revenue tool, not a security tool](https://sublimecoding.com/blog/soc-2-is-a-revenue-tool-not-a-security-tool) — it opens the door to the
conversation; it does not win it. Here’s how this connects back to
deal-loss: a sophisticated reviewer accepts your SOC 2 for the
infrastructure-hygiene questions, then asks, pointedly, about prompt
injection and data isolation *anyway* — because they know those
weren’t in your audit scope. If your answer to those is improvised on
the spot (or auto-filled), the SOC 2 actively works against you. It set
an expectation of rigor the rest of the call didn’t meet, and a reviewer
who feels that gap trusts the rest of your answers less. The fix is to
treat the AI layer as its own program. [Running
security at an AI-native company in 2026](https://sublimecoding.com/blog/running-security-at-an-ai-native-company-2026) is genuinely different from
running security at a traditional SaaS company, and the questionnaire is
just where that difference becomes a revenue event.

## The honest sequencing: need-now vs. defensibly-defer

So what do you actually need before your first enterprise review, and
what can you credibly defer? The good news for founders: you do not need
a perfect program. You need a *real* one with honest edges.
Enterprise reviewers deal with startups constantly; what they’re
allergic to is pretense, not immaturity. “Not yet, here’s the dated plan
and the owner” is a legitimate answer when the foundations are
solid.

Here’s how I sequence it with founders. The table maps the question
category to what the reviewer is *really* probing, and the
minimum credible answer at an early stage.

Question category
What they’re really probing
Minimum credible early-stage answer

Access & identity (SSO, MFA, RBAC, offboarding)
Can a single compromised or departed person reach customer
data?
Non-negotiable. Enforced SSO+MFA, least privilege, same-day
offboarding — have this *fully* before the review.

Encryption & key management
Is data protected at rest and in transit by default?
Non-negotiable. In place, and someone can explain the key handling
without notes.

Incident response
When something breaks, do they know what to do, or improvise?
A written IR plan someone has read, with named roles and a
notification commitment. Tabletop it once; that’s enough early.

Third-party LLM data terms
Is our data training someone else’s model or persisting somewhere
unknown?
Non-negotiable to *know*. Inventory your model providers,
their retention/training terms, and your settings. Be able to state them
precisely.

Tenant / training-data isolation
Can one customer’s data leak into another’s outputs?
Have a real architectural answer. If you don’t fine-tune or share
indexes, say so plainly — that’s often the strongest answer.

Prompt injection & output validation
Do they treat model output as trusted? Have they tested
adversarially?
Have *something* real: input handling, output schema checks,
a few adversarial test cases. A documented, dated plan to deepen it is
defensible.

Human-in-the-loop / autonomy limits
What’s the blast radius if the AI does the wrong thing?
Be able to state what’s autonomous vs. gated and the limits.
Configurable checkpoints can be on the roadmap if named.

Formal certifications (SOC 2, ISO 27001, pen test)
Has a third party validated any of this?
SOC 2 Type II in progress with a date is fine pre-Series-B. A recent
third-party pen test carries real weight here.

Vendor / subprocessor management
Do they know everyone who touches the data?
A current subprocessor list. This is cheap to do well and a soft
spot here looks careless.

The pattern: the *foundations* (identity, encryption, knowing
your data flows) are non-negotiable and mostly cheap. The *depth*
(mature prompt-injection testing, configurable human-in-the-loop, formal
certs) can be sequenced — *if* you can show a dated plan and an
owner, and if your foundations are genuinely solid. What you cannot do
is fake the foundations and gesture at the depth. That’s the exact
profile auto-fill produces, and it’s the exact profile that fails.

One more honest note on cost, because founders always ask: you do not
need a full-time CISO to pass a first enterprise review, and you
shouldn’t hire one yet. The [vCISO math for AI founders](https://sublimecoding.com/blog/vciso-math-for-ai-founders)
usually points the other way at this stage — a fraction of senior
security judgment, applied to exactly these questions, beats a full-time
hire you can’t yet keep busy. The [cost
calculator](https://sublimecoding.com/tools/vciso-cost) sizes it against your own headcount.

## One more thing about who’s telling you this

Notice something about almost every other article on this topic: it’s
published by a company that sells questionnaire-automation software, and
it ends in a button to buy that software. The advice can never honestly
land on the conclusion this piece does — *don’t reach for the fill
tool* — because the fill tool is what they’re selling. The
conclusion is foregone, baked into the byline.

I have nothing to sell you at the bottom of this page that competes
with the argument at the top. I don’t make a questionnaire tool. That’s
the whole reason I can tell you the auto-fill button loses deals: I have
no incentive to want it to win. When you read security advice, that’s
the variable to check first — what does the person telling you this need
to be true? Here, it’s nothing but your program actually being good.

## Before you stare down your first questionnaire

If a deal is stalled on a security review right now, resist the
reflex to buy a tool and auto-fill your way out. Spend the same hour
building the foundations that are non-negotiable, then answer the rest
honestly — with dated plans where the program is genuinely young. Honest
answers from a real foundation beat polished answers from a thin one,
every time, with the reviewers who matter. Speed never was the thing
being graded.

A founder shouldn’t have to navigate a first enterprise review blind
— that’s the work I do as a fractional vCISO: sit on your side of the
questionnaire, find the answers that won’t survive a follow-up question
before the reviewer does, and build the short list of things you
genuinely need first. If that’s where you are, [that’s the engagement](https://sublimecoding.com/consulting) — and here’s [what
a fractional security engagement actually looks like](https://sublimecoding.com/blog/what-a-fractional-security-engagement-actually-looks-like) once it starts.

---

## What a 96% Security Benchmark Actually Measures

URL: https://sublimecoding.com/blog/ai-security-benchmark-what-96-percent-means
Published: 2026-07-28
Tags: security, AI, AI startups

>
**TL;DR:** Microsoft [announced
a 96% score on a security benchmark called CyberGym](https://microsoft.ai/news/introducing-mai-cyber-1-flash-inside-mdash/) this week, and
by the time you read this, a vendor will be pitching you a version of
that number. Here’s the part almost nobody will tell you: the 96% is the
weakest piece of evidence in Microsoft’s own announcement. The strongest
piece — real CVEs found in real Windows code, patched, shipped — is
buried below it. If you’re going to be asked “should we buy this” this
week, you need to know the difference between a benchmark that proves an
agent can *reproduce* a known bug and one that proves it can
*find* an unknown one. That distinction is where the pitch
usually breaks.

## What actually got announced

On July 27, Microsoft [published
a result](https://microsoft.ai/news/introducing-mai-cyber-1-flash-inside-mdash/) for a new compact model, MAI-Cyber-1-Flash, plugged into
MDASH — Microsoft’s “multi-agent vulnerability identification and
remediation harness,” a system that coordinates “100+ agents using
multiple leading models to find, validate, and remediate
vulnerabilities.” The headline claim: the combined system — MDASH,
MAI-Cyber-1-Flash, and GPT-5.4 working together — scored “96% on
CyberGym (+12 pt above Mythos).” Mythos here is Anthropic’s Claude
Mythos, the competing cybersecurity model announced in April 2026 — this
is a comparison against a rival, not against Microsoft’s own previous
best. Microsoft also says the smaller model handles “up to 90% of all
tasks” on its own, reserving the frontier model for “the 10% of
exceptionally hard tasks,” at a claimed “50% cost saving when compared
against our best offering in MDASH today.”

That’s a real result and a real cost story. What I want to slow down
on is what “96% on CyberGym” actually tells you about whether this
system can find a vulnerability nobody has told it about yet — because
that’s the question every buyer in this space actually cares about, and
it’s not the question CyberGym answers.

Two months earlier, in [Microsoft’s
original MDASH announcement](https://www.microsoft.com/en-us/security/blog/2026/05/12/defense-at-ai-speed-microsofts-new-multi-model-agentic-security-system-tops-leading-industry-benchmark/), the system scored 88.45% on CyberGym
level 1 — “roughly five points above the next entry, 83.1%.” That post
is worth reading in full, because it contains a line I have yet to see
quoted in anyone’s marketing copy: “We are deliberate about what these
numbers do and do not claim. They are retrospective recall benchmarks on
internal code with a finite case count.” Microsoft’s own security team
told you, in the announcement, not to over-read the number. Almost
nobody quoting the 96% this week will include that sentence.

## What CyberGym actually tests

This is the fact that makes or breaks how you read every number in
this post, so I want to be precise about it rather than wave at it.

CyberGym is a benchmark built by researchers at UC Berkeley — [the paper is on arXiv](https://arxiv.org/html/2506.02548v2) —
consisting of 1,507 vulnerability tasks pulled from 188 open-source
projects in Google’s OSS-Fuzz corpus. Every one of these is a
*historical* bug: something that was found, reported, and already
patched in the public record before the benchmark existed.

Here’s the structure of a single task, and it’s the load-bearing
detail of this entire post: the agent is handed the full pre-patch
codebase, a compiled executable, and a textual description of the
vulnerability. Its job is to produce a proof-of-concept input that
triggers the bug in the pre-patch build and doesn’t trigger it in the
patched build. The agent is *told a bug exists* and given a rough
sense of what it is. It isn’t hunting in the dark — it’s confirming a
known finding against a specification, in code whose public patch almost
certainly sits in the pretraining corpus of any model built after the
patch date.

That’s reproduction, not discovery. It’s a real and useful skill —
it’s roughly what a human security engineer does when triaging a CVE
report and writing a working exploit to confirm severity — but it is a
categorically different task from staring at a codebase with no hint
that anything is wrong and finding the bug nobody has named yet. Vendors
selling into your team this quarter will use “found vulnerabilities” and
“reproduced known vulnerabilities” interchangeably. They are not the
same claim, and the gap between them is exactly what your procurement
conversation should be arguing about.

For calibration on how fast this specific skill is moving: the
original CyberGym paper (mid-2025, a different set of agent scaffolds)
reported a best score of 11.9% on this same reproduction task. I’d frame
that as a trajectory, not a direct before/after on identical conditions
— the scaffolding, harness, and models are all different a year later —
but the direction is real. Reproduction of known bugs against a spec is
getting a lot cheaper, fast. Whether that generalizes to unknown bugs is
the open question, and it’s the one 96% doesn’t answer.

The 96% tells you the system is good at confirming what it’s told. It
doesn’t tell you the system is good at noticing what nobody
mentioned.

One honest data point in the original CyberGym research: agents
running against these tasks did stumble onto 15 previously undocumented
vulnerabilities along the way. That’s a real discovery signal — but it’s
a side effect of the reproduction exercise, not the metric being
reported, and it’s a fraction of the headline number’s size.

## The evidence that’s actually impressive — and it’s not the 96%

Here’s what I’d point a skeptical buyer to instead, because it’s the
part of Microsoft’s announcement that isn’t a benchmark at all.

MDASH’s May 2026 result included [16
new vulnerabilities found in the Windows networking and authentication
stack](https://www.microsoft.com/en-us/security/blog/2026/05/12/defense-at-ai-speed-microsofts-new-multi-model-agentic-security-system-tops-leading-industry-benchmark/) — code nobody had flagged, in components including
`tcpip.sys`, `ikeext.dll`, `http.sys`,
`netlogon.dll`, and `dnsapi.dll`. Four of those
were Critical remote code execution flaws, including CVE-2026-33824, a
double-free in `ikeext.dll` reachable with crafted IKE
packets, and CVE-2026-33827, a remote code execution flaw in the Windows
TCP/IP stack. Those shipped in the May 2026 Patch Tuesday. Nobody told
the system those bugs existed. That’s discovery, in production code,
with real CVE numbers attached — and it’s a far more meaningful claim
than any benchmark percentage, because there was no answer key.

Microsoft also ran the system against StorageDrive — “a sample device
driver used in Microsoft interviews for offensive security researchers”
— containing 21 deliberately injected flaws never published anywhere.
Result: “all 21 ground-truth vulnerabilities were correctly identified,
with zero false positives in this run.”

Read that one carefully, because it’s better than a benchmark and
weaker than it first sounds. Better, because it’s a held-out set the
model couldn’t have memorized, and because it reports a false-positive
rate at all — the number that actually predicts your team’s workload.
Weaker, because a purpose-built interview exercise with injected bugs is
not production code, and “in this run” is Microsoft’s own hedge on a
single trial, not a stable rate.

So if I were grading this announcement, the 16 real CVEs in shipping
Windows code would carry the argument, the driver test would support it,
and the 96% would be a footnote.

I don’t think Microsoft is being dishonest here — the caveat about
“retrospective recall benchmarks” is right there in their own post, and
burying it isn’t the same as hiding it. But the incentive gradient in
every press cycle pushes toward the single clean number, and “96%”
travels through LinkedIn and vendor decks a lot faster than “21
ground-truth vulnerabilities on a private driver” does. By the time it
reaches your inbox as a sales pitch, the caveat is usually gone.

## The checklist for reading any vendor’s benchmark claim

You’re going to see a version of this pitch from more than one vendor
this quarter, not just Microsoft — CyberGym looks likely to become the
reference benchmark for this category, the MLPerf of agentic security,
for better and worse. Here’s what I’d actually ask, in order:

- **Is this reproduction or discovery?** Ask directly:
was the model told a vulnerability exists, or did it find one with no
hint? If the answer is fuzzy, assume reproduction — that’s the easier
task and the one every current benchmark is best at measuring.

- **What fraction of the eval set predates the model’s training
cutoff?** A model trained after a bug’s public disclosure and
patch has effectively seen the answer during pretraining. Ask the vendor
how they controlled for this. If they haven’t thought about it, that’s
your answer.

- **What did it find on code it had never seen before?**
This is the question that separates marketing from evidence. A private,
held-out codebase with injected or genuinely novel bugs — like
Microsoft’s 21-vulnerability driver test — is worth ten benchmark
percentages. Ask for that number specifically, not the headline
one.

- **Is the win the model or the harness?** Microsoft’s
96% is a *system* score — the orchestration layer (MDASH) plus
the model plus a frontier fallback for the hard 10%. A vendor pitching
you “our model scores X” when the real work is being done by a
hundred-agent scaffold around it is selling you the wrong unit.

- **What’s the false-positive rate, not just the recall
rate?** A system that flags everything scores well on recall and
terribly on your team’s actual workload. Microsoft’s own May 2026 post
is unusually candid about where its system goes wrong: among findings
that targeted the wrong area of code, “82% came from tasks with vague
descriptions that also lacked function or file identifiers,” which
Microsoft reads as evidence that description quality is a major factor
in scan accuracy. Sit with what that implies for the headline number —
CyberGym hands the agent a description. And note that nobody published a
false-positive rate for the benchmark run at all; the only
zero-false-positive figure anywhere in the announcement is the private
driver test. Ask any vendor for their false-positive rate under
realistic, vaguely-specified conditions, not the curated benchmark
ones.

If a vendor can’t answer any of these with a straight number, you’re
being sold a benchmark, not a capability. That’s not automatically a
reason to walk away — reproduction-of-known-bugs is a genuinely useful
capability, and the cost curve on it is dropping fast enough to matter
to your security budget regardless. It’s a reason to price the claim
correctly instead of taking the vendor’s percentage at face value.

## Where this fits if you’re deciding this week

If you’re a 12–40 person AI startup and a security vendor is in your
inbox this week with a benchmark slide, the honest read is: the
technology behind these numbers is moving genuinely fast, and the
discovery evidence (the real CVEs, the held-out driver test) is more
impressive than the reproduction evidence (the CyberGym percentage) even
though the percentage is what gets tweeted. Don’t let the headline
number do your procurement thinking for you — this is the same trap I’ve
written about in [how enterprise
security reviews actually work for AI startups](https://sublimecoding.com/blog/enterprise-security-reviews-ai-startups) and in [reading a cyber insurance
questionnaire for what it actually asks](https://sublimecoding.com/blog/cyber-insurance-ai-startups): the vendor’s number and the
question you actually need answered are frequently not the same thing,
and the gap is where bad buying decisions get made.

What this post doesn’t cover is the operational question that comes
right after “okay, the tool finds things” — who on a 12-person team owns
the triage queue once detection gets this cheap, and what changes when a
scanner can surface more findings in a week than your team can remediate
in a quarter. That’s a distinct problem with a distinct answer, and it’s
the subject of the next post in this series: [detection got
cheap, remediation didn’t](https://sublimecoding.com/blog/ai-vulnerability-remediation-bottleneck).

For now, the narrower question — is this benchmark telling you what
the vendor implies it’s telling you — is one you can answer yourself
with the five questions above, before the call even starts.

If you’re weighing whether this decision needs a security hire, a
fractional vCISO, or just better internal judgment for one quarter, I’ve
laid out the actual math in [what a vCISO costs an AI
startup](https://sublimecoding.com/blog/vciso-math-for-ai-founders), and the broader operating model in [running
security at an AI-native company in 2026](https://sublimecoding.com/blog/running-security-at-an-ai-native-company-2026). If the honest answer this
week is “we don’t have anyone who reads a vendor benchmark claim
critically before it turns into a signed contract,” that’s worth a short
conversation before the renewal, not after.

---

## Secrets Management for AI Agents on Small Teams

URL: https://sublimecoding.com/blog/ai-agent-secrets-management
Published: 2026-07-15
Tags: security, agents, engineering

>
**TL;DR:** The model never needs the key. That one
sentence is most of agent secrets management, and almost every setup I
see gets it backwards — the API key sits in the agent’s environment or,
worse, in its context window, where a single prompt injection turns your
assistant into an exfiltration tool. The fix isn’t an identity platform;
at small-team scale it’s an architectural habit: credentials live with
the *tools*, tools execute with the narrowest credential that
does the job, the model only ever sees results. Add short-lived tokens
where your providers make it easy, log what the agent touched, and keep
one kill switch you’ve actually tested. The identity-for-agents vendor
pitch can wait until you have agents enough to need it.

## The model never needed the key

Walk through what actually happens when your agent “uses” a
credential. The model emits a tool call — a blob of JSON saying
`query_database` with some arguments. Your code receives that
JSON, executes the query, and returns rows. At no point in that loop
does the *model* need the database password. The code that runs
the tool needs it. The model needs the result.

This is obvious once you say it out loud, and yet the default wiring
of most agent stacks ignores it. The agent process gets the same
`.env` the app gets. Every tool call executes with the full
ambient authority of everything in that file — the Postgres superuser,
the Stripe live key, the AWS credentials with `*` on the
resource ARN. Not because anyone decided the agent should have all that.
Because nobody decided anything; the agent inherited the human-shaped
credential model, and the human-shaped credential model assumes the
thing holding the keys has judgment.

An agent does not have judgment. It has instructions, and it takes
new instructions from anything that lands in its context — a support
ticket, a scraped web page, a README in a cloned repo. [Prompt
injection is precisely the trick of making retrieved content count as
instructions](https://www.cequence.ai/blog/ai/even-the-best-ai-agents-leak-secrets-prompt-injection-is-why/), and a credential the model can see is a credential the
model can be talked into repeating. The mid-2025 Supabase incident was
the canonical version: an agent with privileged service-role access
processed support tickets containing attacker-supplied instructions, and
integration tokens ended up in a public thread. Excessive privilege plus
untrusted input — that’s the whole recipe, and it’s the [top
entry in OWASP’s MCP Top 10](https://owasp.org/www-project-mcp-top-10/2025/MCP01-2025-Token-Mismanagement-and-Secret-Exposure) for a reason.

The scale of the sprawl backs this up. GitGuardian counted [28.6
million new secrets exposed in public GitHub commits in 2025](https://www.helpnetsecurity.com/2026/04/14/gitguardian-ai-agents-credentials-leak/) — a 34%
jump, the biggest in the report’s history — including over 24,000 unique
secrets sitting in MCP configuration files specifically. Agents didn’t
invent secrets sprawl. They industrialized it, because an agent commits,
configures, and copies at machine speed, and it never gets the uneasy
feeling a human gets pasting a key somewhere it doesn’t belong.

## Three ways agents break the human credential model

It’s worth being precise about what’s actually new here, because
“treat the agent like an employee” — the framing half the vendor content
uses — leads you to the wrong controls.

**Speed.** A human with an over-scoped key misuses it
occasionally, detectably, at human pace. An agent makes hundreds of tool
calls an hour. If one of them is wrong — wrong table, wrong environment,
wrong recipient — the blast radius is bounded only by what the
credential allows and how fast you notice. I wrote about [Amazon learning this in public](https://sublimecoding.com/blog/amazon-let-the-ai-drive):
the mistake wasn’t novel, the amplification was.

**Injectability.** You cannot socially engineer your
deploy script. You *can* socially engineer your agent, remotely,
in writing, at scale, by putting words where it will read them. Every
credential in the model’s reach is one crafted document away from
disclosure. This is why [the autonomy
question](https://sublimecoding.com/blog/when-to-trust-an-agent-and-when-to-step-in) and the credential question are the same question wearing
different hats.

**Ambient authority.** Humans accumulate scoped logins
with MFA and session expiry. Agents accumulate environment variables —
long-lived, unscoped, invisible. OWASP’s agentic guidance pushes [“least
agency” alongside least privilege](https://goteleport.com/blog/owasp-top-10-agentic-applications/): don’t just ask what the agent can
access, ask how much it can *do* with that access before a human
checks. An env file full of god-keys maximizes both, silently.

## The broker pattern: credentials live with tools

The architectural fix is small enough to describe in one paragraph.
Between the model and every side effect sits a broker — the code that
receives the tool call, decides whether to run it, executes it with a
credential the model has never seen, and returns only the result. If
you’ve read [the
50-line agent loop](https://sublimecoding.com/blog/build-ai-agent-loop-50-lines-elixir), you’ve already seen the shape: the loop’s
`execute_tool/2` function *is* the broker seam. You
just have to treat it as a security boundary instead of a dispatch
table.

In Elixir it barely deserves the word “pattern”:

`[](#cb1-1)defmodule Agent.Broker do
[](#cb1-2) # Each tool gets its own credential, fetched at call time —
[](#cb1-3) # never placed in the prompt, never returned to the model.
[](#cb1-4) @tools %{
[](#cb1-5) "search_orders" => {Tools.Orders, :readonly_db_url},
[](#cb1-6) "refund_order" => {Tools.Refunds, :stripe_restricted_key},
[](#cb1-7) "send_email" => {Tools.Email, :smtp_send_only}
[](#cb1-8) }
[](#cb1-9)
[](#cb1-10) def call(tool_name, args, ctx) do
[](#cb1-11) with {:ok, {mod, cred_key}} <- Map.fetch(@tools, tool_name),
[](#cb1-12) :ok <- Policy.allow?(tool_name, args, ctx) do
[](#cb1-13) cred = Secrets.fetch!(cred_key)
[](#cb1-14) result = mod.run(args, cred)
[](#cb1-15) AuditLog.record(ctx.run_id, tool_name, args, summarize(result))
[](#cb1-16) {:ok, redact(result)}
[](#cb1-17) end
[](#cb1-18) end
[](#cb1-19)end
```

Three properties matter more than the code. The credential is
*fetched inside the call*, so it exists in memory for the
duration of one tool execution, not for the lifetime of the agent
process. The result is *redacted before it returns* — if the tool
response could contain secrets (config dumps, user records with tokens),
strip them, because whatever goes back to the model becomes context, and
context can be exfiltrated by the next injection. And *every call is
logged* with the run ID, because when something goes wrong the first
question is “what did the agent touch,” and grepping model transcripts
is a miserable way to answer it.

If your agent reaches tools through MCP servers instead of in-process
functions, the boundary moves but the rule doesn’t: the MCP server holds
the credential, scoped to what that server does, and the config that
launches it is a secrets-bearing file — which is exactly where
GitGuardian found those 24,000 leaked keys. Treat mcp.json`
with the paranoia you’d give `.env`, because it is one.

## Scope per tool, not per agent

The unit of least privilege is the tool, not the agent. “The support
agent’s key” is already too coarse — the support agent’s
*order-lookup tool* needs a read-only database role limited to
two tables; its *refund tool* needs a Stripe restricted key
capped at refunds; its *email tool* needs send-only SMTP with
your domain locked. Each of those is a five-minute setup task in the
respective dashboard, and each converts a catastrophic injection outcome
into an annoying one. An attacker who fully owns the model in that
architecture can issue refunds until your cap trips. They cannot read
your user table, mint API keys, or touch the infrastructure, because no
tool in reach holds a credential that can.

This is the same argument I make in [the AI-native
security stack post](https://sublimecoding.com/blog/running-security-at-an-ai-native-company-2026) at the company level; this is what it compiles
down to at the code level. Providers have quietly made it easy: Stripe
restricted keys, Postgres roles, GitHub fine-grained PATs, AWS IAM with
resource conditions. The primitives are all old. What’s new is having a
caller that will predictably, mechanically use *everything* a
credential allows — which turns scoping from best practice into
load-bearing wall.

## Short-lived beats static, where it’s cheap

Static keys leak and stay leaked; the GitHub incident-response
literature is a graveyard of keys revoked months after exposure. Every
credential your broker fetches should be as short-lived as your provider
makes convenient: AWS STS sessions instead of IAM user keys,
OIDC-federated tokens in CI, database credentials from Vault-style
dynamic issuance if you already run it — and if you don’t, a weekly
rotation habit on the handful of static keys you can’t avoid.

The honest small-team version: you will not get everything
short-lived, and you don’t need to. Rank by blast radius. Cloud-account
credentials and anything that can move money go first. The key to your
error tracker can stay static without keeping anyone up at night.
OWASP’s framing — [task-scoped,
time-bound permissions](https://cheatsheetseries.owasp.org/cheatsheets/AI_Agent_Security_Cheat_Sheet.html) — is the ideal; a five-person team gets most
of the value from applying it to the three credentials that matter and
consciously deferring the rest.

## The five-person reality check

Since “it depends on your stage” is doing a lot of work above, here’s
the explicit line I’d draw for a team of five with agents in the
development loop and maybe one in production:

**Fine:** secrets in environment variables *on the
tool-executing process*, loaded from a managed store (Fly secrets,
AWS SSM, 1Password CLI — whichever you already have). Per-tool scoped
keys. A shared runbook that says which keys exist and who rotates them.
Agents in dev running with dev credentials against dev data.

**Negligent:** any credential in a prompt, system
message, or file the agent is instructed to read. One key shared across
tools “for now.” Production credentials in the environment of a coding
agent that executes arbitrary shell commands — remember that [cloning a
repo was enough to redirect an API key in CVE-2026-21852](https://blog.cyberdesserts.com/ai-agent-security-risks/); your
agent’s environment is attack surface even when the agent behaves. No
log of agent tool calls. No tested way to shut it off.

That last pair costs an afternoon, total, and it’s the difference
between an incident and an anecdote.

## Kill switch, then rotation

When an agent misbehaves — and the base rate says eventually one will
— you need two moves you’ve rehearsed. First, stop the agent: a feature
flag the broker checks on every call beats hunting for the process at
2am. Second, rotate what it held: which is only tractable if you scoped
per tool, because “rotate everything in the env file” is exactly the
Saturday you built all this to avoid. This is also where the audit log
pays for itself — rotation scope equals the set of credentials behind
tools the run actually called, and the log is the only thing that
knows.

## What not to buy yet

There is a fast-growing category of agent-identity platforms —
non-human identity management, agent IAM, credential brokers as a
service. The pitch is real for enterprises running fleets of agents
across hundreds of systems with compliance reporting on top. At
pre-Series-A, with one product and a handful of tools, the platform
mostly sells you a hosted version of the broker function above, plus a
new vendor in your critical path and a new place your credentials
live.

Buy it when you have the problem it solves: many agents, many teams,
auditors asking for agent-access reviews, or a credential graph you
genuinely can’t hold in your head. Until then the boring stack — scoped
keys, a broker seam you own, your platform’s secret store, one log —
covers a five-person team with money left over. This is the same [prove-the-return discipline](https://sublimecoding.com/blog/prove-the-return) that
applies to every tool purchase; security tooling doesn’t get an
exemption.

## The part that doesn’t change

Strip the AI vocabulary and this post says: don’t give programs more
authority than their job needs, don’t put secrets where untrusted input
can reach them, log privileged actions, and be able to revoke fast. That
was true before LLMs. What agents change is the *default* — for
the first time, the path of least resistance hands a credulous,
injectable, machine-speed actor the keys to everything, and the
ecosystem’s leak statistics show exactly how that’s going. (One Linux
distribution now ships that default as a keybinding — [I inventoried what it hands an
agent](https://sublimecoding.com/blog/omarchy-agent-safety-off).)

If you’re standing up agents with real credentials and want a second
pair of eyes on the boundary — what’s scoped, what’s brokered, what you
can safely defer — [that’s the kind of engagement I
run](https://sublimecoding.com/consulting).

---

## Detection Got Cheap. Remediation Didn't.

URL: https://sublimecoding.com/blog/ai-vulnerability-remediation-bottleneck
Published: 2026-07-29
Tags: security, founders, vCISO

>
**TL;DR:** Microsoft’s MAI-Cyber-1-Flash just made
automated vulnerability discovery cheaper and more accurate. Good —
genuinely. But for a 12-40 person AI startup, discovery was never the
constraint; cheap scanners have been flooding small teams with findings
for a decade. The constraint is engineer-hours to triage a queue, decide
what’s real, and ship a fix without blowing up the sprint. A tool that
doubles your findings and halves the cost per finding hasn’t helped you.
It’s moved the bottleneck one step downstream, made it more visible, and
left the actual question — who owns that queue, and what authority do
they have to say no to a release — exactly where it was.

## The queue just got a new firehose

On July 27, 2026, Microsoft [announced
MAI-Cyber-1-Flash](https://microsoft.ai/news/introducing-mai-cyber-1-flash-inside-mdash/), a new model inside MDASH — their “multi-agent
vulnerability identification and remediation harness,” which runs “100+
agents using multiple leading models to find, validate, and remediate
vulnerabilities.” The headline numbers: the combined system scored “96%
on CyberGym (+12 pt above Mythos)” — Mythos being Anthropic’s Claude
Mythos, the rival cybersecurity model it was measured against, not a
previous Microsoft one — at a “50% cost saving when compared against our
best offering in MDASH today.” MDASH isn’t a research toy — it’s the
same system that reportedly [surfaced
16 previously unknown vulnerabilities](https://thehackernews.com/2026/05/microsofts-mdash-ai-system-finds-16.html) in the Windows networking and
authentication stack ahead of the May 2026 Patch Tuesday, four of them
Critical remote-code-execution flaws. Whether that 96% is a fair proxy
for finding vulnerabilities in the wild, versus reproducing ones a
benchmark already knows about, is a real and separate question — I’ve
written [what
that number actually measures](https://sublimecoding.com/blog/ai-security-benchmark-what-96-percent-means) elsewhere. Here I want to grant the
number and ask a different question: even if it’s true, what does it
change for a startup with no dedicated security headcount?

Not much. And in one specific way, it makes the actual problem
worse.

## What genuinely gets better

I don’t want to undersell the real improvement, because there is one,
and pretending otherwise would be its own kind of dishonesty. A
detection system that is both cheaper and more precise changes the
*composition* of a findings queue, not just its size. Every
founder who has run a scanner against a real codebase knows the tax
isn’t the vulnerabilities — it’s the noise around them. A dependency
scanner flags a transitive package with a theoretical CVE that’s
unreachable from any code path you actually execute. A SAST tool flags a
SQL string concatenation that turns out to be building an internal admin
query with no user input anywhere near it. Someone has to open each
finding, understand it well enough to rule it out, and write “not
exploitable, closing” — and that triage tax is identical whether the
finding is real or noise, because you don’t know which one you’re
looking at until you’ve done the work.

If a detection system is genuinely more precise — fewer of those
dead-end findings for the same recall, or the same number of true
positives with less chaff around them — that’s a direct cut to the
triage tax, and it’s the one lever that actually helps a small team.
Lower false-positive rate is not a rounding-error improvement; it’s the
whole game, because triage time is the resource a 12-40 person company
doesn’t have. I’d take a scanner that’s 20% less complete but half as
noisy over one that finds everything and buries it in garbage, every
time.

Microsoft does put one number on this: against StorageDrive — [a
sample device driver used in Microsoft interviews for offensive security
researchers](https://www.microsoft.com/en-us/security/blog/2026/05/12/defense-at-ai-speed-microsofts-new-multi-model-agentic-security-system-tops-leading-industry-benchmark/), seeded with 21 deliberately injected flaws — it reports
that “all 21 ground-truth vulnerabilities were correctly identified,
with zero false positives in this run.” Read “in this run” as
load-bearing: one trial, against a purpose-built interview exercise
rather than production code, is not a rate you can staff against. But
zero false positives is the *shape* of the improvement that would
actually matter to a four-engineer team, which is why it’s the number
I’d chase in a vendor call instead of the 96%.

A tool that doubles your findings and halves your cost per finding
hasn’t helped you. It has moved your bottleneck one step downstream and
made it more visible.

But notice what that improvement does and doesn’t touch. It reduces
the *cost of confirming a finding is real*. It does nothing to
the cost of *fixing* it.

## What doesn’t move at all

Here’s the honest inventory of what a better detector leaves
completely untouched, because this is the part vendor content never
runs:

**Fix capacity.** Whether a finding is discovered by a
free open-source scanner or a 100-agent frontier harness, the fix still
has to be written by an engineer who understands your codebase, reviewed
by someone else, tested, and shipped without breaking the three other
things that depend on that code path. Detection speed and fix speed are
not the same curve, and they never converge just because detection got
faster. If your team has four engineers and they’re all heads-down on
the roadmap that’s funding payroll, a faster detector produces a longer
backlog, not a shorter one.

**Regression risk.** Every fix to running production
code is itself a change with its own blast radius. A patch to an auth
check or a data-access boundary — exactly the kind of thing these tools
are good at finding — is exactly the kind of thing you don’t want to
ship carelessly under time pressure. The tool that surfaced the bug in
an afternoon doesn’t reduce the hours of regression testing the fix
deserves. If anything, a bigger queue of “urgent” findings creates
pressure to skip that testing, which is how a security fix becomes an
outage.

**Deploy cadence.** Startups at this size usually don’t
have a change-management process built for emergency security patches
layered on top of a normal release train. Someone has to decide: does
this finding jump the queue, or wait for the next deploy window? That’s
a judgment call with no automated answer, made by a person, under
pressure, usually without the authority to make it stick.

**The political cost of blocking a release.** This is
the one nobody puts in a vendor deck. At a 12-40 person company, the
person who finds the vulnerability and the person who owns the roadmap
are frequently the same three people wearing different hats in the same
week. Telling the founder “we need to hold this release for a fix” is
not a technical statement — it’s an organizational one, and it requires
someone in the room with enough standing to make it stick without it
turning into a grudge. A better scanner doesn’t create that standing. It
doesn’t create that person, either.

More and better findings, running into the same fixed fix-capacity
and the same absent authority structure, doesn’t shrink the backlog. It
grows it, and it makes the growth visible in a dashboard that now looks
worse than it did before you bought the tool that was supposed to
help.

## Who actually owns the queue

This is the question that matters and the one nobody wants to answer
with real specificity, because the honest answer at this stage is
usually “nobody, formally.” Findings arrive in a Slack channel or a
ticket queue, and whoever has the most spare attention that week triages
a few of them. That’s not a process; it’s a coin flip with a paper
trail.

“Owning the queue” has to mean something in authority terms, not just
a name on an org chart. At minimum it means: the person who can look at
a finding and say “this ships before the feature” and have that decision
hold — not just be argued into a compromise by whoever shipped the
feature. Without that authority, a triage queue is theater. The scanner
produces a list, someone glances at it, and the list rolls over to next
sprint, indefinitely, because there’s no mechanism that forces a
decision.

Whether that person is a first full-time security hire or a
fractional one is a separate decision with its own tradeoffs — I’ve laid
out how I think about that specific fork in [security engineer or
vCISO as your first hire](https://sublimecoding.com/blog/security-engineer-or-vciso-first-hire), and I won’t re-litigate it here. What I
will say: the tooling question (“which scanner do we buy”) and the
ownership question (“who has the standing to act on what it finds”) are
not the same question, and buying the better tool first is solving the
easier one while the real one sits there.

## What to do instead of buying the next tool

If you’re a founder or first security hire looking at a findings
queue right now — real or, if you haven’t run a scanner yet, imagine one
that comes back with 200 open items tomorrow — the move that actually
helps isn’t a better detector. It’s a triage discipline that didn’t
exist before:

- **Write down a severity rubric before the next scan
runs**, not after. Three tiers is enough: ships-blocking,
fix-this-sprint, and backlog-with-a-review-date. A rubric written under
no pressure is honest; one written while a scary finding is open in
another tab isn’t.

- **Assign an owner to the queue, not to individual
findings.** One person’s job is to make sure nothing sits
untriaged for more than a week, even if the fix itself gets done by
someone else. Diffuse ownership is how findings age into “we’ll get to
it,” permanently.

- **Decide the release-blocking authority explicitly, in
advance, in a document, with the founder’s sign-off** — not
improvised in the moment a scary finding collides with a launch date.
The worst time to establish that a security finding can hold a release
is during the argument about whether this particular one should.

- **Budget fix-capacity as a percentage of engineering time, not
as leftover time.** If nobody’s calendar has hours reserved for
closing findings, the backlog doesn’t shrink no matter how good the
detector gets, because “leftover time” at a startup is zero by
construction. I’d start at 10% of engineering time and adjust after a
quarter of watching what the queue actually does.

None of that requires buying anything. It requires someone with the
standing to make it stick, which is exactly what a fractional security
engagement is built to install in the first 90 days — I’ve described
concretely what that looks like in [what
a fractional security engagement actually looks like](https://sublimecoding.com/blog/what-a-fractional-security-engagement-actually-looks-like), and it’s
mostly triage discipline and authority-setting, not tooling.

## What to ask before you buy the next scanner

If a vendor is pitching you MDASH-class tooling, or anything that
promises higher recall or lower false positives, the questions that
matter aren’t about the benchmark number. They’re about your team:

- Who on our team will triage what this finds, and how many hours a
week do they actually have?

- What’s our documented rule for when a finding blocks a release, and
does it have the founder’s sign-off?

- If this tool doubles our open-findings count next month, what
changes about our capacity to close them?

- Are we buying detection because we lack detection, or because
detection is more legible to buy than the org discipline we actually
lack?

That last one is the uncomfortable one, and it’s the one worth
sitting with. A scanner is a line item you can point to in a board deck.
A triage rubric and a person with the authority to hold a release are
not — they’re organizational work, unglamorous and slow, and precisely
for that reason they’re the thing that’s actually scarce.

## The scoping question underneath all of this

Cheaper, better detection is a genuinely good thing to happen to the
security industry. It is close to irrelevant to whether a 12-40 person
AI startup ships fixes faster next quarter, because the bottleneck was
never the finding. It was always the deciding and the doing, and the org
question of who has the standing to make either happen. If you’re trying
to figure out whether that gap gets closed by a first full-time hire, a
fractional engagement, or a few weeks of triage discipline you set up
yourselves, that’s exactly the scoping question [the vCISO math for AI
founders](https://sublimecoding.com/blog/vciso-math-for-ai-founders) works through — and it’s a shorter conversation than most
founders expect. For the fuller picture of what running security looks
like at this stage beyond just the remediation queue, see [running
security at an AI-native company](https://sublimecoding.com/blog/running-security-at-an-ai-native-company-2026).

If you’re staring at a findings queue right now and the honest answer
to “who owns this” is a shrug, [that’s worth a
conversation before the next tool purchase, not after](https://sublimecoding.com/consulting).

---

## From One Engineer to Fifteen: What Co-Founding Taught Me About Engineering Leadership

URL: https://sublimecoding.com/blog/from-one-engineer-to-fifteen-engineering-leadership
Published: 2026-01-12
Tags: engineering leadership, founders, hiring

**I went from sole engineer to running a fifteen-person engineering organization over four years at PopSocial. The hardest lessons weren't about code.**

I co-founded PopSocial in 2015 and was the only engineer on the platform for the first six months. By the time I left in 2019, the engineering organization was fifteen people across backend, frontend, and DevOps. Most of what I learned in that stretch was not technical. It was about the messy, tactical, embarrassing-in-hindsight job of building and running a team while still also writing code.

Six lessons that I would have benefited from someone telling me explicitly on day one. Most of them I had to learn by getting them wrong first.

## Lesson 1: The day you stop coding is the day the team starts performing

For the first eighteen months I was both the engineering manager and the senior IC. I told myself this was the responsible thing to do as a founder — keep the burn low, keep my hands on the codebase, ship fast. The actual effect was that nothing else got built well.

I was the bottleneck on every PR review, every architectural decision, every onboarding session. Engineers waited on me to make decisions instead of making them themselves, because they could see I'd weigh in eventually. The team was technically functional but learning slowly because they had a senior IC two layers above them.

The week I stopped writing code in the main path of any feature, three things happened almost immediately. The team's PR cycle time dropped by about 40%. Engineers started making the architectural decisions I'd been making, and most of them were right. And I started having time to do things only I could do as the leader: hiring, fundraising, customer conversations, planning.

The lesson, simply: a founder who continues to be the senior IC is paying a hidden tax that compounds. Stop coding (in the main path) sooner than feels comfortable. You can keep your hands on by writing tooling, paying down infra debt, or pairing with an engineer on something hard — but get out of the production path.

## Lesson 2: You hire too slowly and fire too late. Plan accordingly.

The conventional wisdom is "hire slow, fire fast." Almost everyone gets the first part right and the second part wrong. The result is that bad hires linger for months while you tell yourself the situation will improve.

The most expensive hires of my career were the ones I should have ended at the 60-day mark and instead held on to for another five months because firing felt cruel. By the time I let them go, I'd lost five months of progress, demoralized the team that had been picking up their slack, and damaged my own credibility as a leader because everyone else could see what I couldn't.

The framework that finally worked: a written, explicit "first 60 days" expectations document for every hire. We'd review it at day 30, day 60, day 90. If we got to day 60 and the new hire wasn't on track to meet the day-90 expectations, we'd have a direct conversation. Most fixed themselves with that conversation. The ones that didn't, didn't, and we let them go at day 75 or 90 instead of month seven.

This is not heartless management. The opposite — telling someone explicitly what's going wrong and giving them a clear shot at fixing it is more respectful than letting them stay and dragging them along.

## Lesson 3: Title inflation is a tax you pay later

In year two we needed to hire a backend lead. The candidate we were closing wanted "Director of Engineering" as a title, even though the team was three engineers including him. I gave it to him. Five months later we needed to hire a senior engineer. We couldn't offer "Senior" because the existing team's titles had inflated past it.

Title inflation creates two compounding costs. First, it sets a ceiling that future hires have to be promoted past, which creates artificial pressure and weird performance dynamics. Second, it muddles the actual seniority of the team — when you have three Directors of Engineering and one IC, your engineering org chart is broken.

The fix in retrospect: pick a clean ladder, write it down, and don't deviate. "Engineer / Senior Engineer / Staff Engineer / Engineering Manager / Director of Engineering" — five rungs, defined, with explicit criteria for each. If a candidate insists on a higher title, that's a signal to dig into why. Almost always it's about cash compensation and there's a better way to solve the problem.

## Lesson 4: One-on-ones are the highest-ROI thing on your calendar

I started 1-on-1s as a checkbox item I felt I had to do. By the end I considered them the most leveraged hour on my week.

The format that worked, refined over time: 30 minutes weekly. Three sections — what's blocking you, what's bothering you, what are you working on. The first two are non-negotiable; the third is sometimes obvious from context and can be skipped.

The reason 1-on-1s are high-ROI is that engineers tell their manager problems an hour before they tell anyone else. By having the meeting on the calendar at a known cadence, you catch the problems an hour earlier than you would have otherwise. Over a year, that's the difference between a team that runs into the same brick walls repeatedly and a team that pivots before they hit them.

The mistake I made for too long: treating 1-on-1s as status updates. They're not. They're the channel for the things engineers won't say in standup or in Slack.

## Lesson 5: The interview process IS the culture

Whatever you do during an interview is what the candidate believes the company is like every day. If your interview is rushed, disorganized, and poorly calibrated, the candidate (correctly) infers the company is rushed, disorganized, and poorly calibrated. They take that information into their accept-decline decision.

I spent year three rebuilding our interview process from scratch. The big changes that mattered most:

- **Written rubrics for each round.** Every interviewer knew what they were evaluating and how to score it. Eliminated the "I dunno, I have a good feeling about him" bias that was driving who we hired.

- **Calibration sessions every six weeks.** The interview team would do a debrief on a recent candidate against the rubric, surface scoring disagreements, and recalibrate. Caught drift before it became a hiring problem.

- **A take-home that respected the candidate's time.** Two hours, not eight. With a clear evaluation criteria. We'd accept partial submissions if the candidate explained why.

- **A close-the-loop debrief 24 hours after the final round, written.** Every candidate got a yes/no decision in writing within a day. Even the no's. Especially the no's.

The team's hiring quality improved measurably. The brand impact among the candidate pool — who talk to each other — improved even more.

## Lesson 6: Performance reviews suck. Do them anyway.

Almost no founders enjoy performance reviews. Almost all teams need them. The mismatch is why they get skipped, deferred, or done badly.

The version that works at startup scale: light, honest, twice a year. Two questions in writing — what's going well, what's the one thing you'd change. From the engineer about themselves, from the manager about the engineer, from peers when relevant. A 30-minute conversation about both sides of the answers. A written outcome doc.

The version that doesn't work: annual 360 reviews with five rating dimensions and bell-curve calibration. At 15 engineers, that's bureaucratic theater. At 200 engineers, you grow into it.

The reason to do the light version even at small scale is that without it, you have no shared written record of expectations and progress. When you eventually need to make a tough call (promotion, demotion, termination, raise), you have nothing to reference. With it, you have six months of explicit notes from both sides about what good looks like.

## The thing that actually matters: clarity about what's good

If I had to compress all of this into one principle, it's this. **The job of an engineering leader at a small startup is to make it explicit what good looks like.**

Good code looks like X. Good design conversations sound like Y. Good 1-on-1s feel like Z. Good performance from this engineer at this level means producing this kind of output at this kind of pace. Good interviews ask these questions and listen for these answers.

Most engineering leadership failures at small startups are failures of clarity. The team doesn't know what good is, so the team isn't sure when they're doing well, and the leader is unhappy with the output but can't articulate why. Codifying "good" in writing — even sketchy first drafts — fixes more problems than any other intervention.

## Would I do it again?

Yes. But differently. I'd hire my first engineer slower and more carefully. I'd stop coding in the main path at month nine instead of month eighteen. I'd codify the title ladder before the second hire. I'd write the rubrics before the third interview. I'd start performance reviews at five engineers instead of ten.

The good news: most of these mistakes are recoverable, and even at the time, the team got better, the product shipped, and we ended up with a strong organization. But "got there eventually" is not the bar I want to hit if I do this again. The bar is "got there with the team intact, the leaders developed, and the company strong enough to handle the next stage."

That's the bar I'm using as I help founders make these calls now. Most of the lessons I'm sharing in this post are lessons I learned on the job. The point of writing them down is that the next founder doesn't have to.

## Read this next

- [**The Pre-Series-A AI Startup Hiring Plan**](https://sublimecoding.com/blog/pre-series-a-ai-startup-hiring-plan) — The role-by-role plan for getting the first six hires right.

- [**How I'd Hire a Staff Engineer at an AI Startup**](https://sublimecoding.com/blog/how-id-hire-a-staff-engineer-at-an-ai-startup) — The interview process I'd build today, in detail.

- [**AI-Assisted Engineering Isn't Faster Coding**](https://sublimecoding.com/blog/ai-assisted-engineering-is-a-new-workflow) — How modern engineering teams operate differently than the one I built in 2018.

- [**How to Manage a 4-Person Engineering Team Without Becoming a Manager**](https://sublimecoding.com/blog/managing-a-four-person-engineering-team) — The 5 rituals that work at 3–5 engineers, before you need real process.

---

## AI-Assisted Engineering Isn't Faster Coding. It's a New Workflow.

URL: https://sublimecoding.com/blog/ai-assisted-engineering-is-a-new-workflow
Published: 2026-03-16
Tags: AI, engineering, productivity, AI tools, engineering leadership

**Most engineers using Claude Code see a 10–15% speedup. The teams seeing 40–55% aren't typing faster. They're sequencing work differently.**

I've been shipping production code with AI assistance since early 2023 — Claude Code, GitHub Copilot, and OpenAI Codex have been daily tools across an AI-first product, a fintech platform, and an IoT ingestion pipeline. The patterns that move the needle are not the ones most teams reach for first. Which of them handles which task now lives in [one version-controlled routing table](https://sublimecoding.com/blog/model-routing-file-ai-coding-agents) rather than in my head.

The frame "AI-assisted development" is part of the problem. It implies a tool that sits next to you, helping you type. The reality at the teams getting real leverage: AI is a teammate, you're the lead, and the workflow is fundamentally different from how you wrote code three years ago.

Here's what actually moved the needle.

## The myth of the 10x AI engineer

The default mental model is "AI as autocomplete." You type a function signature, you accept the suggestion, you save 30 seconds. Multiply by a workday and you get the 10–15% speedup that every benchmark study reports.

That's the floor, not the ceiling.

The ceiling is reached when you stop using AI to type code faster and start using it to *compress the steps before you type*. Architecture, naming, edge-case enumeration, test scaffolding, code review, documentation — most engineering time is spent on these, not on keystrokes. Compress those and the throughput change becomes structural.

The teams I've seen hit 40–55% delivery cycle reduction did three things differently:

- They stopped treating AI as a coding tool and started treating it as a thinking surface

- They standardized which model gets which job and didn't let everyone improvise

- They held the same code review bar — AI-generated code never gets a free pass

## The four modes I use AI in

Different problems want different uses. Mixing them up is most of why teams plateau.

### Mode 1 — Architect

Before any code: paste the problem, the constraints, and the existing code shape into Claude. Ask it to enumerate three approaches. Ask it to argue for and against each. Ask it which it would pick at this team size and why.

I do not accept its answer. I read the tradeoffs, find the ones I'd missed, and then make my own call. The win isn't in the answer — it's in the time saved enumerating possibilities I would have walked through anyway, just slower and less thoroughly. A 30-minute architecture conversation becomes a 7-minute one. Repeat that five times a week and the calendar opens up dramatically.

### Mode 2 — Ship

Once the design is settled, I have AI generate the boring 80%. Boilerplate, tests for happy paths, repetitive transformations, glue code, migrations. The interesting 20% — the gnarly state machine, the concurrency-sensitive bit, the contract with another service — I write myself, often after talking through it with the model first.

The discipline: I do not let AI write the code I would not want to read in two years. If a function is going to be load-bearing, I write it. If it's wiring three already-working pieces together, AI writes it.

### Mode 3 — Review

Before opening a PR, I paste the diff into Claude with a single instruction: "Review this like an adversarial senior engineer who hates my work. Find the bugs, race conditions, security issues, and unclear naming."

It catches a real bug or smell about 30% of the time. The other 70% is noise I dismiss. The noise dismissal cost is small. The 30% is enormous — every one of those is a comment my human reviewer doesn't have to write, and a deploy I don't have to roll back.

### Mode 4 — Document

READMEs, ADRs, runbooks, deprecation notices, release notes. AI is excellent at first drafts of all of these because they follow predictable structure and the underlying facts already exist in code or in my head. I dictate the structure and the key points; it produces the prose; I edit for voice. What used to take a couple hours takes 20 minutes.

Most engineering orgs are chronically under-documented because the marginal cost of writing it down is too high. AI changes that math.

## Where AI fails (and where I refuse to use it)

The teams in the 40–55% range are also disciplined about where they don't use AI. A short list of categories I always handle myself:

- **Brand-new libraries or unstable APIs.** Hallucination rate spikes. AI confidently writes code against a method that doesn't exist. The cost of debugging fake APIs erases any time savings.

- **Anything touching real money or auth without thorough human review.** AI doesn't have stakes. It will produce a payment flow that looks reasonable and fails open. I treat AI suggestions in these areas as drafts that must be reviewed line by line.

- **Performance work that requires measurement.** AI loves to suggest optimizations that look correct and are useless or actively worse. Profile first. Decide based on data. AI can help analyze the profile output, not pick the optimization.

- **Cross-cutting refactors.** The model sees the file, not the system. Ten files of "fix" that all individually look right and collectively break four invariants is a real failure mode.

- **"Vibe coding" without a spec.** If you can't tell the model what you want concretely, you don't know what you want. AI happily produces sprawl in this state. Specify, then code.

## The team buy-in problem

The hardest part of standardizing AI-assisted engineering across a team isn't tooling. It's the senior engineers who are skeptical, and they aren't wrong to be.

Their concern, usually unstated: AI undermines the craft. They've spent fifteen years getting good at code review, naming, architecture. A tool that spits out passable code threatens to flatten that gradient and make the median engineer look as good as the senior on the surface.

The reframe that lands: **AI raises the floor, not the ceiling.** A junior engineer with Claude is now operating at mid-level on routine work. A senior engineer with Claude is now operating at staff level on the work that matters, because they've offloaded the routine. The senior's edge — judgment, taste, system thinking — becomes more valuable, not less.

Concretely, what I've done at AI-first orgs:

- Standardize the tool stack: Claude Code for substantive work, Copilot for inline completion, Codex for one-off shell scripts. No improvising.

- Pair-program with skeptics. Show them their own ergonomics improving in real time. The 30-minute architecture chat becoming a 7-minute one is a visceral demo.

- Hold the same code review bar. AI-authored code goes through the exact same review process. No "the AI wrote it" exemptions. This was the single biggest credibility move with the senior bench.

- Make the patterns visible. Document the four modes (or your team's version), share examples of good and bad use, retro on AI-related bugs the same way you'd retro any incident.

## Measurable outcomes

The numbers I've actually seen:

- **Delivery cycle reduction:** 40–55% on product feature work. Bigger than I expected. Smaller than the AI vendors claim.

- **Quality:** bugs-per-PR ratio held steady. Production incident rate held steady. This is the number that surprised people most — most assumed AI-authored code would be lower quality. With proper review discipline, it isn't.

- **Headcount:** stable team output went up roughly 50%, with no headcount growth. That's the headline finding. AI didn't replace engineers — it amplified them.

- **Where it didn't help:** infrastructure / platform work saw maybe 10–15% gains. The work is too contextual, too specific to your environment. Don't expect the same speedup for a backend platform team as for a product feature team.

One caveat worth naming. The teams I've measured are small (5–15 engineers), high-trust, with clear technical leads. I have not yet seen what these patterns look like at 200 engineers across multiple business units. Some of what works at this scale is going to break at that scale. Be skeptical of anyone claiming universal numbers.

## Picking the right tool for the job

Standardizing the tool stack matters more than picking the "best" tool. Three tools, used consistently, beat seven tools used haphazardly.

The split I've landed on:

- **Claude Code for substantive work.** Architecture conversations, refactors, multi-file edits, code review, anything that requires the model to hold the shape of a feature in its head. Claude's longer context window and more conservative coding style fit this work better than the alternatives I've tested.

- **GitHub Copilot for inline completion.** The autocomplete-style use case. Fast, low-stakes, reduces typing fatigue. I let it complete the obvious next line; I do not ask it to design anything.

- **OpenAI Codex / GPT-5 for shell scripts and one-offs.** Quick scripting tasks where I want a command-line answer in seconds. Different ergonomic register than the in-IDE tools.

The principle: each tool has an interaction model that's good for a specific type of work. Mixing them up — using Copilot for architecture, using Claude Code for autocomplete — wastes the strengths of each. Pick a tool, use it for what it's good at, switch when the task changes.

Two practical notes for teams adopting this. First, pay for the paid tier on whichever you use most. The free-tier rate limits will produce flow-state interruptions that destroy the productivity gain. The $20–60 per engineer per month is one of the highest-ROI line items on your engineering bill. Second, make the tool choices explicit in onboarding. New engineers should not have to figure out the team's AI workflow by osmosis.

## Why this matters for founders

If you're pre-PMF, this is your edge over slower-moving competitors.

Funded competitors with bigger teams will out-spend you. They cannot out-iterate you if you've internalized this workflow and they haven't. A 5-person team operating at 50% throughput multiplier is shipping at the velocity of a 7- or 8-person team — and at a fraction of the burn.

If you're post-PMF and scaling, this is how you delay the headcount conversation by six months. That's six months of runway, six months of org-design time, six months of hiring more carefully.

The shift is real. It is not magical. It rewards engineers who treat it as a workflow change rather than a tool swap. Pick up the tool, and then put in the work to actually change how you work.

## Read this next

- [**How We Cut $350K From Cloud Spend in 6 Months (And What I'd Do Differently)**](https://sublimecoding.com/blog/cut-350k-cloud-spend-six-months) — Where AI-augmented teams generate the runway extension that pays for the next round.

- [**How I'd Run Security at an AI-Native Company in 2026**](https://sublimecoding.com/blog/running-security-at-an-ai-native-company-2026) — What you have to be paranoid about when AI is generating production code.

- [**Your AI Product Needs a Telemetry Layer Before It Needs a Better Model**](https://sublimecoding.com/blog/your-ai-product-needs-telemetry-before-better-model) — Instrument the system before tuning the model — the four-layer telemetry stack.

---

## The Onboarding Playbook for an Agent-Heavy Codebase

URL: https://sublimecoding.com/blog/onboarding-agent-heavy-codebase
Published: 2026-07-19
Tags: engineering leadership, hiring, agents

>
**TL;DR:** Every onboarding template on the internet
assumes the codebase a new hire is reading was written by humans, at
human pace, with human-shaped mistakes. That assumption is already wrong
at a lot of shops. When a third to three-quarters of new code is
agent-authored, onboarding isn’t “the same 30-60-90 template, plus a
CLAUDE.md link.” The actual skill a newcomer has to build is different:
reading a diff without knowing who wrote it, learning which conventions
are enforced by a linter-adjacent agent versus held only in a senior’s
head, and figuring out what a mentor is even for when the boilerplate
writes itself. This is the process for that, week by week.

## The inversion nobody’s writing about

Search “onboarding new engineer AI” and you get two genres. The first
is generic 30-60-90 templates that would’ve been fine in 2019 — same
milestones, same checklists, with “review our AI tools” bolted onto week
one like a compliance module. The second is “use AI to onboard your new
hires faster,” which is a real and useful topic but the wrong direction
entirely: it’s about deploying an agent *to help* onboard a
human, not about what changes for the human when the *codebase
itself* is agent-heavy.

Nobody’s covering the inversion, and it’s the harder problem. Your
new hire isn’t reading a system that a team of people wrote over three
years. They’re reading a system where a meaningful share of the commits
were generated by an agent, reviewed by a human who was moving fast, and
merged under time pressure that predates the new hire’s start date by
exactly zero days. That’s a different codebase to onboard into, and it
needs a different process — not a paragraph appended to the old one.

The scale of this isn’t hypothetical anymore. Google’s CEO said in
April 2026 that 75% of the company’s new code is now AI-generated and
approved by engineers, [up
from 50% the previous fall](https://www.fastcompany.com/91531519/google-ceo-says-75-of-the-companys-code-is-ai-generated). Microsoft’s CEO put his own company at
20–30% [a
year earlier](https://www.cnbc.com/2025/04/29/satya-nadella-says-as-much-as-30percent-of-microsoft-code-is-written-by-ai.html), rising steadily. You don’t have to be Google-scale for
this to matter to a new hire — you just need enough agent-authored code
in the history that “who wrote this and why” stops being answerable by
instinct. I put my own numbers on this shift, on a much smaller repo, in
[4,154 Commits in Six
Months With AI Agents](https://sublimecoding.com/blog/4154-commits-six-months-ai-agents) — the commit graph looks like a much bigger
team than the one that exists.

None of what follows replaces a CLAUDE.md. That file is the artifact
— the executable tribal knowledge a new hire and every agent session
both read on day one, and I’ve written the case for treating it as
onboarding documentation rather than config in [Your CLAUDE.md Is the Onboarding
Doc You Never Wrote](https://sublimecoding.com/blog/claude-md-onboarding-doc). This post is the *process* around that
artifact: what a manager and a senior actually do, week by week, to turn
a human into a trusted contributor to a codebase that’s already half
machine-authored before they show up.

## First-PR review calibration when half the diffs are agent-authored

A new hire’s first PR is normally a low-stakes, high-signal event.
You’re not evaluating whether the code is perfect — you’re calibrating
how they take feedback, whether they ask the right questions, whether
they understood the ask. That calibration breaks quietly when the
reviewer doesn’t know the diff’s provenance.

Here’s the failure mode I watch for: a newcomer submits a PR where
the bulk of the diff was agent-generated, lightly reviewed, and pasted
into place. The code passes CI. It looks clean — arguably cleaner than
what a nervous new hire would write by hand in week one. A reviewer who
doesn’t ask “how much of this did you write versus generate and check”
grades the PR on the artifact and misses the actual thing they’re
supposed to be measuring, which is whether this person can evaluate
code, not whether they can produce it.

The fix isn’t banning agent use on early PRs — that’s both
unenforceable and backwards, since fluency with agents is table stakes
now. The fix is changing what the first-PR review *asks for*:

- **Require a one-paragraph “how I built this” note on the
PR**, not as a gotcha, but as the same signal a live coding round
used to give you. Which parts were generated, which were hand-written,
what you changed after reading the generated version, and why.

- **Ask the newcomer to defend one specific decision in the diff
out loud**, in review or a short sync — not the whole PR, one
line or one function. If they can’t explain why a particular approach
was taken, that’s the finding, regardless of who typed it.

- **Grade the review comments they leave on *other*
people’s agent-authored PRs**, not just the code they submit.
Reading and catching problems in someone else’s (possibly
machine-generated) diff is a closer proxy for the actual job now than
writing a clean function from scratch.

This isn’t a lower bar. It’s a *different* bar, and pretending
the old one still measures the same thing is how a team ends up with
someone who can shepherd an agent to working code but can’t tell you why
it’s right.

## “Who wrote this” — building the transparency norm before it’s needed

Ambiguity about authorship isn’t just an onboarding problem, it’s a
team-wide health problem that gets exposed hardest by a new hire,
because they have no context to fall back on. A senior engineer looking
at a weird function in a familiar file has priors: *that’s probably
legacy, or that’s Dave being clever again.* A new hire has none of
that. They just see code, and they can’t tell if it’s a deliberate
pattern to learn or an agent’s confident guess that nobody caught.

The team that answers “who wrote this” honestly is the team where a
new hire’s confusion gets resolved in a Slack thread instead of silently
copied into their own next diff.

The norm worth building, before a new hire arrives, not after they’re
confused by it:

- **Commit messages and PR descriptions state authorship
plainly** — not every commit needs a disclaimer, but PRs that are
majority agent-generated should say so, the same way you’d note “ported
from the old service” or “pairing with contractor X.” This is cheap and
it’s the single highest-leverage transparency move available.

- **A newcomer is explicitly told it’s safe to ask “was this
agent-written” in review**, without it reading as an accusation.
On a team where that question feels loaded, new hires learn to stay
quiet about their own confusion, which is the opposite of what week one
is for.

- **Nobody hides behind “the agent wrote it.”**
Authorship transparency cuts both ways — stating that code was
agent-generated doesn’t excuse skipping the review, it’s context for a
*better* review. The engineer who merged it still owns it. Make
that explicit to a new hire early, because the alternative — an org
where “the agent did it” becomes a diffusion-of-responsibility shrug —
is a culture problem you don’t want them learning by example.

I’ve written more broadly about why this kind of transparency is the
actual operating system underneath any team, human or agent, in [Trust Is the Operating
System](https://sublimecoding.com/blog/trust-is-the-operating-system) — the same three-way trust that has to run between you, your
team, and the mission runs here too: a new hire has to trust that “who
wrote this” is a question they’re allowed to ask, and the team has to
trust them enough to hear an honest answer.

## Mentorship, recalibrated: what a senior teaches when the agent writes the boilerplate

The classic mentorship model assumes the junior’s first six months
are spent writing a lot of straightforward code under supervision — CRUD
endpoints, test scaffolding, the un-glamorous 70% of the work that
builds muscle memory for the codebase’s shape. That 70% is exactly the
part an agent now does fastest and best. If a senior’s mentorship plan
is still “give them the boilerplate tickets to build confidence,”
they’re handing a new hire tasks the agent would’ve done in the same PR
anyway, and the newcomer learns nothing that a code review of the
agent’s output wouldn’t have taught them faster.

What a senior needs to teach instead, deliberately, because it won’t
happen by osmosis anymore:

- **Judgment about when *not* to reach for the
agent.** The specific corners of the codebase — the billing path,
the auth boundary, the one gnarly reconciliation job — where
hand-written, deeply understood code matters more than fast output. A
new hire won’t know which corners those are unless someone tells them,
because the codebase itself won’t visibly flag it.

- **How to read a diff for intent, not just
correctness.** Agent-generated code that passes tests can still
solve the wrong problem elegantly. Teaching a new hire to ask “does this
address what the ticket actually needed, or just what it literally said”
is a mentorship muscle that matters more, not less, when the code itself
looks polished by default.

- **The undocumented reasons things are the way they
are.** This is the part a CLAUDE.md captures for an agent’s
context window but a human still needs walked through in conversation —
not because the file is wrong, but because a new hire benefits from the
story behind the fact, and an agent doesn’t need the story to use the
fact correctly.

- **How to disagree with an agent’s output in a PR they’re
reviewing, not just one they wrote.** This is a skill nobody had
to teach five years ago because there was no confident, fluent,
occasionally-wrong collaborator producing plausible-looking diffs at
volume. It’s now one of the highest-value things a senior can model in a
pairing session.

If you’re hiring a staff-level engineer into this environment
specifically to carry this kind of mentorship load, the calibration
matters even more than usual — I laid out the three flavors of “staff
engineer” and which one an AI startup actually needs in [How I’d Hire
a Staff Engineer at an AI Startup](https://sublimecoding.com/blog/how-id-hire-a-staff-engineer-at-an-ai-startup). The wrong flavor in this seat is
a senior who mentors like it’s 2019, teaching a new hire to write
boilerplate a machine already writes for free.

## Standards drift: what a newcomer learns from the agent versus what only a human holds

Every codebase has two layers of convention. The first layer is
enforced — a linter, a formatter, a CI check, or increasingly, an agent
instructed to follow specific patterns every time it touches a file. The
second layer is *held* — conventions that exist because a senior
engineer remembers a past incident, a design decision that never got
written down, or a “we don’t do it that way here” that lives entirely in
tribal memory.

Agent-heavy codebases make the first layer look more complete than it
is. A new hire watching agent-generated PRs land cleanly, formatted
correctly, following the obvious patterns in the file, can reasonably
conclude the codebase’s standards are fully captured somewhere. They
aren’t. The agent is enforcing what it was told to enforce — usually in
a CLAUDE.md or equivalent, per the framing in [Your CLAUDE.md Is the Onboarding
Doc You Never Wrote](https://sublimecoding.com/blog/claude-md-onboarding-doc) — and staying silent about everything nobody
thought to write down. That silence reads, to a newcomer, as “there’s no
rule here,” which is the most dangerous kind of standards drift: not a
rule being broken, but a rule that was never visible in the first
place.

The detection process worth running deliberately, rather than hoping
a new hire stumbles into it:

- **Have them audit one week of merged PRs and flag anything
that looks inconsistent** — not to find bugs, but to find places
where two “correct-looking” PRs did the same thing two different ways.
Ask a senior which way is actually preferred and why. Half the time the
answer is “either is fine.” The other half is where a held convention
just surfaced.

- **Ask them, explicitly, in week three or four: “what’s a rule
here you’ve inferred from the code that nobody’s told you out
loud?”** Whatever they answer is either confirmation the pattern
is real, or a chance to correct a wrong inference before it compounds
into their own future PRs.

- **Treat every “I assumed the agent would catch that” moment as
a CLAUDE.md gap, not a hiring gap.** If a newcomer gets burned by
a convention the agent didn’t enforce and nobody told them about, the
fix is adding it to the shared instructions everyone reads — human and
agent — not just correcting the individual.

## The week-by-week shape

Not a 30-60-90 template — those assume the ramp is uniform and the
codebase is static. This is shaped around when agent fluency versus
judgment actually gets tested.

**Week 1 — Read before you write, and read
authorship-aware.** The new hire reads the CLAUDE.md (or
equivalent) alongside the actual codebase, and is told explicitly which
parts of the recent history are agent-heavy versus hand-written, so
they’re not building intuition on a false assumption. No PRs yet. Their
only output this week: a list of three things the code does that the
CLAUDE.md doesn’t explain, handed to their onboarding buddy.

**Week 2 — First PR, small and bounded, with the “how I built
this” note.** Scoped tight enough that agent-assist or not, the
review is fast and the transparency norm gets exercised immediately,
before habits form either way.

**Weeks 3–4 — Standards-drift audit.** The exercise
above: one week of merged PRs, flag the inconsistencies, ask what’s
inferred versus told. This is also when a senior should start pairing
specifically on judgment calls — the “should this be hand-written”
corners of the codebase — rather than more boilerplate tickets.

**Weeks 5–8 — First PR review of someone else’s agent-authored
diff.** This is the real test, not the first PR they wrote. Can
they read a plausible-looking diff critically, ask “does this solve what
the ticket needed,” and push back on something that passes CI but misses
intent? This is where you find out if you hired someone who can operate
the agent or someone who can supervise it — a distinction with a growing
gap between the two.

**Weeks 9–12 — Own a slice that includes at least one “don’t
reach for the agent” corner.** By now they should know which
parts of the system are hand-written on purpose. Give them ownership of
one, with a senior available but not hovering. Their judgment about
*when to say no to the fast path* is the actual signal you’re now
able to measure, because they’ve had two months of evidence to form it
from.

The shape holds whether the newcomer is a junior or an experienced
hire from a less agent-heavy shop — the second group often needs weeks
1–2 more than the first, because the disorientation isn’t about
engineering skill, it’s about a codebase that doesn’t behave the way
their instincts predict.

## Hiring for this before you onboard for it

None of this process fixes a bad hire. If the underlying hiring bar
is still tuned for whiteboard recall instead of judgment under
ambiguity, this onboarding playbook is applying good process to the
wrong input — I’ve made the broader case against that mismatch in [The Perfect Hire Is Killing
Your Team](https://sublimecoding.com/blog/perfect-hire-killing-your-team). What this process actually needs from a candidate isn’t
agent fluency (that’s teachable in week one) — it’s the same
slope-over-intercept judgment that mattered before agents existed,
applied to a codebase where the volume of plausible-looking output is
higher and the cost of skimming instead of reading is higher with
it.

If you’re building out an engineering org where this is already the
reality — where new hires are reading diffs they can’t attribute on
sight, and your existing onboarding process still assumes 2019 — this is
exactly the kind of process gap a fractional engineering leader closes
fast, without a six-month re-org. If that’s where you are, [let’s talk](https://sublimecoding.com/consulting).

---

## Trust Is the Operating System

URL: https://sublimecoding.com/blog/trust-is-the-operating-system
Published: 2026-07-13
Tags: engineering leadership, agents, AI

>
**TL;DR:** People ask how I manage. One system runs the
whole thing, whether the teammate is a person or an agent: trust. I
extend it to you, you extend it to the rest of the pod, and everyone
extends it to the mission. Hire for the trust you can afford to give.
Scale by repeating the small unit, not mutating it. Manage outcomes, not
hours. Protect the struggle that grows people. And run your agents
through the exact same operating system — as an intern you shouldn’t
trust off the bat, on a leash that widens as it earns it, except the
agent doesn’t grow from the struggle. You do, by hardening the harness
around it.

## The membership test

I’ve run teams from three people to twenty, onshore and offshore, and
every time someone asks “how do you manage,” they’re expecting a
framework — OKRs, a ritual cadence, a tool stack. Those things exist and
they matter, but they’re not the operating system. They’re artifacts
*of* it.

The operating system is trust. It runs in three directions at once. I
have to trust you with the domain knowledge and the calls I’m not
qualified to make myself — that’s the entire reason I hired you instead
of doing it myself. You have to trust the rest of the pod the same way,
or you’re not a team, you’re a group of people who happen to share a
standup. And everyone, including me, has to trust the mission enough to
make the tradeoffs that a mission requires instead of optimizing for
what looks good in the moment.

Strip away every other question about how to manage and you’re left
with one: if I can’t trust you, why are you on the team? Not “why is
your code good” — trust isn’t a proxy for skill, and it isn’t earned by
credentials. It’s whether I believe you’ll make the call correctly when
I’m not in the room, and whether you’ll tell me the truth when you
don’t. Everything downstream of that — the pods, the goal-setting, the
way I run agents — is just trust, operationalized at a different
scale.

## Hire for the trust you can extend

You can’t run a trust-based system on people you can’t trust yet, so
hiring is where the operating system actually starts, not where it gets
tested. I’ve written the long version of this — the case against
optimizing for the whiteboard-polished, pedigree-heavy candidate over
the person with real trajectory — in [The Perfect Hire Is Killing
Your Team](https://sublimecoding.com/blog/perfect-hire-killing-your-team). The short version for this piece: I hire for drive over
polish. Hungry self-starters with an entrepreneurial streak, not people
chasing the founder title — you want ownership without the exit plan. AI
has made domain knowledge acquirable in a way it never was five years
ago; what it hasn’t made acquirable is drive. That’s still the scarce
input, and it’s the one thing you’re actually betting on when you extend
trust to someone new.

## The pyramid of pods

Here’s the part almost everyone skips: how you actually scale a team
past the size where you can hold every relationship in your head.

The unit is a pod of three to eight people. Not a rule of thumb — a
hard constraint on how many people one person can genuinely trust and be
trusted by, bidirectionally, at the density that makes trust real
instead of nominal. Past eight, you stop knowing who’s actually stuck
and start reading status updates instead of people.

So when the team outgrows one pod, you don’t stretch the pod. You
repeat it. A second pod, a third, each one the same size, each one
running the same trust relationships internally. The layer above them
isn’t one manager stretched across twenty reports — it’s a *team*
of pod leads who cross-talk across the boundaries that would otherwise
calcify into silos: front-end talking to back-end talking to mobile,
before a ticket forces the conversation. That cross-talk is itself a
trust relationship, at a different layer, running the same operating
system.

This is the actual answer to “how do you scale from a handful of
engineers to fifteen or twenty” — you don’t design a bigger structure,
you fractal the small one and put people you trust in charge of the
boundary. I lived through most of this the hard way — the hires I got
right, the ones I got wrong, the systems I wish I’d built sooner — and
wrote it up in [From
One Engineer to Fifteen](https://sublimecoding.com/blog/from-one-engineer-to-fifteen-engineering-leadership). And if you’re still at the size where the
pod *is* the whole team — three to five people — the mechanics of
running that without becoming a full-time manager are in [How to Manage a
4-Person Engineering Team Without Becoming a Manager](https://sublimecoding.com/blog/managing-a-four-person-engineering-team). Same operating
system, every layer. Trust is the invariant; headcount is just how many
times you’ve repeated the unit.

## Outcomes over hours, not surveillance dressed as accountability

If trust is real, it shows up first in what you measure. I run
goal-based, not hours-based: quarterly OKRs broken into weekly chunks
that actually sum to the quarter, so a Friday check-in tells you
something true about whether the outcome is on track — not whether
someone was active in Slack between nine and five.

When a chunk gets missed, I go in with curiosity, not blame. A
two-day task not done in a week means there’s an issue — but “there’s an
issue” is a question, not a verdict, and how you ask it is most of the
job. It’s not what you say, it’s how you say it. Most misses are a
signal about scope, a blocker nobody flagged, or a wrong estimate — not
a character problem. Treat it like a character problem the first time
and you’ve taught the whole pod to hide the next miss instead of
surfacing it, which is strictly worse for you.

What I don’t do is instrument the distrust: no keyloggers, no
screenshot tools, no activity trackers pretending to be productivity
tools. You hired smart, capable people specifically because you can’t do
their job yourself — surveilling them is a tell that you don’t actually
believe that, and they will notice. [Professionalism](https://sublimecoding.com/blog/professional-owns-the-outcome) is owning
the outcome, not performing busyness for a dashboard, and a surveillance
culture optimizes for exactly the wrong signal.

This default — trust the pod to recover its own bad call, don’t hover
— has a limit. If there’s no consensus and the pod is spinning in
circles arguing instead of deciding, I step in. Decision by indecision
is a recipe for bad; someone has to break the tie, and sometimes that’s
me. Same with a deadline that’s genuinely at risk: the default is
hands-off, but hands-off isn’t the same as absent. If a pod is going to
miss something that matters, I’ll add bandwidth to save it — including
my own hands, which I’ll get to below.

There’s a cost to getting the trust-not-surveillance default wrong,
and it isn’t hypothetical — it’s a security posture, not just a
management one. If your team is afraid to tell you about the near-miss,
you don’t have a security program, you have a countdown. What that looks
like when an agent — not a person — is the one making the miss is its
own postmortem discipline, which I cover in [What an AI Agent Postmortem Should
Contain](https://sublimecoding.com/blog/ai-agent-postmortem).

And when the person too afraid to flag the miss is a human, not an
agent, the failure isn’t technical at all: [a scared team is your
biggest attack surface](https://sublimecoding.com/blog/psychological-safety-security-control), because fear suppresses the exact honest
report your security depends on.

## Don’t steal the learning

The flip side of trusting people with outcomes is trusting them with
the struggle that makes them better at producing outcomes. People grow
by working through something hard, not by being handed the answer or
having someone quietly clean up behind them. Every time you jump in to
save someone from a fixable mistake, you’ve taken the rep away from them
— and taught them, without meaning to, that they don’t have to clean
their own mess, because you will.

This is getting harder to hold onto in an AI-accelerated shop,
because the temptation to just let the model produce the answer and skip
the struggle entirely is constant. I’ve written about the specific
failure mode where AI closes the knowing-vs-doing gap for juniors before
they’ve built the judgment that used to come from doing the grunt work
themselves, in [We’re About
to Stop Making Senior Engineers](https://sublimecoding.com/blog/stop-making-senior-engineers), and about where Satya Nadella’s
“token capital” framing gets the small-team version of this right and
wrong, in [Nadella Is
Right About AI and the Firm. Mostly.](https://sublimecoding.com/blog/nadella-token-capital-small-teams) The short version: protect the
struggle on purpose. It’s the only part of growing an engineer that
doesn’t scale, so it’s the part you have to defend deliberately.

## The same operating system runs your agents

This is the part of the system nobody else is writing about honestly,
and it’s where I want to spend the rest of this.

An agent is the intern you shouldn’t trust off the bat. That’s not a
metaphor I’m reaching for — it’s the literal correct posture. It
onboards the same way a hire does: you write it a CLAUDE.md the way
you’d write onboarding docs for a new engineer, and I’ve made the case
that a CLAUDE.md *is* exactly that document — executable tribal
knowledge, not config — in [Your
CLAUDE.md Is the Onboarding Doc You Never Wrote](https://sublimecoding.com/blog/claude-md-onboarding-doc). Where the analogy
holds all the way through: the agent starts on a short,
permanent-feeling leash and earns wider scope only as it demonstrates it
deserves it, exactly like you’d ladder a new hire from read access to
write access to production access over their first quarter, not their
first day.

An agent is the intern you shouldn’t trust off the bat.

Where the analogy stops holding is the reason you have to be
*more* careful with the agent, not less: it’s nondeterministic
and it fails faster than a person ever could. A junior engineer who’s
about to make a bad call gives you tells — hesitation, a question in
Slack, a draft PR sitting open for review. An agent doesn’t hesitate. It
drives the car into the tree faster than a human ever could, with full
confidence the whole way there. The framework I use for calibrating how
much autonomy an agent has actually earned — and the signals that it’s
drifting off the rails before it hits anything — is in [When to Trust an
Agent and When to Step In](https://sublimecoding.com/blog/when-to-trust-an-agent-and-when-to-step-in). Amazon found this out at a scale that
cost real orders when a mandated coding agent got unsupervised access to
infrastructure it wasn’t ready for — the fix wasn’t less AI, it was more
humans per deploy, which is the same lesson at enterprise scale: [Amazon Let the AI Drive. It Hit a
Tree.](https://sublimecoding.com/blog/amazon-let-the-ai-drive) Guardrails here aren’t a vote of distrust in the tool. They’re
physics. You’d put the same guardrails on a new hire’s prod access, just
slower ones, because a human’s worst mistake in an hour is smaller than
an agent’s worst mistake in a minute.

Here’s the polarity flip that took me longest to actually
internalize: with a person, you protect the struggle because that’s how
they grow. With an agent, there’s no struggle to protect, because
there’s no growth to protect it for — the model on Tuesday isn’t a wiser
version of the model that made Monday’s mistake. So the learning doesn’t
happen in the worker. It has to happen in the *manager*, and it
shows up as system hardening instead of personal growth: you tighten the
CLAUDE.md, you add the guardrail the postmortem revealed you needed, you
harden the harness. Every agent mistake is a rep for you, not for it.
I’ve written about what actually earns a place in that document after
real mistakes, not a speculative template copied once and forgotten, in
[What I Put in CLAUDE.md After
50 Commits With It](https://sublimecoding.com/blog/claude-md-after-50-commits) — and what it looks like to run this discipline
hard enough to actually ship at volume, in [4,154 Commits in Six
Months With AI Agents](https://sublimecoding.com/blog/4154-commits-six-months-ai-agents).

That reframe changes the actual question you should be asking before
you widen an agent’s scope. It’s not “do I trust this agent” — you
don’t, categorically, the same way you don’t extend blind trust to a new
hire in week one. The question is: does it matter if the agent nukes the
database, if you can restore it in seconds? An agent deleting a
production database is a real, documented failure mode, not a
hypothetical — I’ve written about the incident and the actual lesson
from it, which was to hire *more* senior engineers, not fewer, in
[An AI
Just Deleted a Production Database in Nine Seconds. Hire More
Engineers.](https://sublimecoding.com/blog/ai-deleted-production-database-hire-more-engineers) The self-repair capability that made that incident
recoverable in minutes instead of days is probably just chaos
engineering under a different name — you’re not trusting the agent more,
you’re driving the cost of its worst plausible mistake toward zero, so
the blast radius stops being the thing that scares you.

Do that consistently and something else follows almost automatically:
every engineer on your team becomes, in effect, a manager of
agent-reports. That’s not a new discipline bolted onto the job — it’s
one more layer of the same pyramid of pods I described above, with the
same rules: earn trust in small increments, widen the blast radius
slowly and cautiously as it compounds, protect yourself from the failure
mode you haven’t guardrailed yet instead of the one you already have.
The operating system doesn’t change when the teammate is a model instead
of a person. Only the growth target does — it moves from the worker to
the system around the worker.

## Close: no work is beneath the leader

None of this works if trust only flows downward. I’ll file files.
I’ll do the menial cleanup task nobody wants, the off-hours bug triage,
whatever it takes to hit a deadline the pod is genuinely at risk of
missing — because “not my problem” isn’t a sentence a leader gets to
use, and because the trust I’m asking the team to extend to me has to be
earned the same way I’m asking them to earn mine. When Lavender needed
engineering power to hit a deadline, I stepped in and added hands, not
just direction. The outcome belongs to the whole team, and that has to
be true in both directions or it’s not actually trust, it’s a hierarchy
wearing trust’s clothes.

That’s the whole system, top to bottom and person to agent: extend
the trust you can afford to extend, in the smallest unit that lets you
extend it honestly, measured by outcomes instead of hours, protecting
the struggle where it produces growth and hardening the harness where it
doesn’t. Everything else — the tools, the rituals, the org chart — is
downstream of whether that trust is real.

If you’re building or rebuilding a team under this model — sizing the
first pods, figuring out where the agent layer actually earns its
autonomy, or just want a second set of eyes on the org you’ve got —
that’s [a conversation worth having](https://sublimecoding.com/consulting).

---

## Performance Reviews When Agents Do the Typing

URL: https://sublimecoding.com/blog/performance-reviews-when-agents-type
Published: 2026-07-24
Tags: engineering leadership, agents, AI

>
**TL;DR:** Commit volume, PR count, and lines shipped
used to be rough but honest proxies for how much an engineer
contributed. Agents broke that proxy — anyone on the team can now
generate a wall of commits in an afternoon, and the number tells you
nothing about whether the work was any good. Search “how to measure
engineer performance with AI agents” and every result on page one is a
metrics dashboard pitching you a new number to replace the old broken
one. That’s the wrong direction. The fix isn’t a better dashboard — it’s
a better conversation. What I actually evaluate now: the calls an
engineer makes about what *not* to let the agent do, the quality
of their review on work they didn’t type, which problems they chose to
spend agent time on, whether they verify before they ship, and how they
behave when the agent quietly breaks something. None of that shows up on
a chart. All of it shows up in twenty minutes of the right
questions.

## The metrics stopped meaning anything

I ran engineering reviews for a decade before any of this, and the
informal signal always leaked in through the numbers, even when nobody
said the number was the metric. You’d glance at someone’s commit graph,
their PR count for the quarter, roughly how much surface area they’d
touched. It was a bad proxy even then — a quiet engineer who spent three
weeks on one gnarly migration could look worse than someone shipping ten
small tickets. But it was a *directionally honest* bad proxy.
More commits generally meant more hours of actual human typing,
thinking, and testing behind them. The same broken proxy shows up in real estate: [the office mandate is a measurement failure](https://sublimecoding.com/blog/office-mandate-measurement-failure).

That correlation is gone. I wrote about this directly from the inside
— in six months of running an AI-agent-heavy workflow on one project, I
logged [4,154
commits](https://sublimecoding.com/blog/4154-commits-six-months-ai-agents), more than my previous four years combined. The commits are
real, they compile, they run in production. But the number stopped being
a proxy for effort the moment I stopped typing most of the code myself.
Anyone on your team who’s decent at prompting an agent can produce that
graph now. It costs an afternoon, not a career.

It’s not just anecdotal. GitClear’s 2026 analysis of AI-era code
changes found refactoring — the “moved” code that signals someone
actually restructured a design instead of bolting on more of it — fell
from 13% of changed lines in 2023 to 3.8% year-to-date in 2026, while
copy-pasted code climbed from 9.4% in 2022 to 15.7% in the first half of
2026, and two-week code churn (code revised almost immediately after it
was written) rose another 15% ([GitClear,
2026](https://www.gitclear.com/the_ai_code_quality_maintainability_gap)). Volume is up. The signal that volume used to carry —
sustained, deliberate design work — is down. If you’re still grading on
output, you’re grading the wrong axis, and you’re rewarding exactly the
pattern the data says is getting worse.

## What review actually costs now

The place the inflated-output problem shows up hardest isn’t the
person writing the code — it’s everyone downstream of it. The 2026 AI
Engineering Report found that under high AI adoption, median time to
first PR review is up 156.6%, average time spent in review is up 199.6%,
average PR size is up 51.3%, and bugs found per PR are up 54% ([Faros
AI, 2026](https://www.faros.ai/blog/ai-code-quality-senior-engineer-review-burden)). Read that as a system, not a set of stats: the volume of
code needing scrutiny went up, the time available to scrutinize it
didn’t, and the defect rate went up too. Somewhere in your org, a senior
engineer is absorbing all three of those numbers at once, and it’s not
visible on any dashboard that just counts commits.

That’s the actual shift a performance review has to account for. The
engineer who “shipped the most” this quarter might be the one who
generated the most PR volume for someone else to clean up. I’ve seen
this pattern enough now to trust it as a category, not a one-off:
agent-assisted code is stylistically clean — consistent formatting,
sensible variable names, idiomatic-looking structure — which is exactly
why it’s harder to review. The old tell for “something’s off here” was
ugly code. Ugly code doesn’t happen anymore. The bugs hide behind good
formatting now, which means the reviewer has to actually understand the
logic, every time, because the visual shortcut is gone.

If your review criteria still rewards raw throughput, you are
structurally rewarding the person generating the most review debt for
the rest of the team. That’s the opposite of what a performance review
is supposed to select for.

## What to evaluate instead

Once commit count, PR count, and lines shipped are off the table,
here’s what’s left — and it turns out to be a better read on an engineer
than the old metrics ever were, because it was always what those metrics
were a weak proxy *for*.

**What they told the agent not to do.** This is the
single highest-signal thing I look for now. Anyone can prompt an agent
to generate a feature. The engineers worth promoting are the ones who
can tell you, specifically, where they stopped the agent — the shortcut
it wanted to take on error handling, the migration it wanted to run
without a rollback path, the test it wanted to skip because the
assertion was inconvenient to write. That’s judgment. It doesn’t show up
in a diff. It shows up in what’s *not* in the diff, and the only
way to see it is to ask.

**Review quality on work they didn’t type.** Since
agent-written code passes the “does it look clean” test by default, the
engineer’s real contribution shifts to whether they caught the thing
that looked fine and wasn’t. I look at their review comments on
agent-authored PRs the same way I used to look at their own code: are
the comments specific and load-bearing, or are they rubber-stamp
approvals because the diff was tidy and the tests were green?

**Problem selection.** When typing is nearly free, the
constraint moves entirely to *which problem gets the agent’s
time*. An engineer who points three agents at three well-scoped,
well-understood problems and ships all three cleanly is doing more real
work than one who points an agent at something ambiguous and gets a
plausible-looking wrong answer back. Picking the right problem to
delegate — and recognizing which ones shouldn’t be delegated at all — is
now a core skill, not a nice-to-have.

**Verification discipline.** Did they run it, read the
output, and check the edge case before merging — or did they trust that
green CI meant correct? [The failure patterns I
catch from agents every week](https://sublimecoding.com/blog/things-ai-coding-agents-get-wrong) are rarely subtle once you look —
confidently wrong answers, silently dropped context, a test that passes
because it tests the wrong thing. The engineers I trust with more scope
are the ones who assume the agent is capable of exactly those failures
and verify accordingly, every time, not just when something feels
off.

**How they handle agent failure.** Everyone’s agent
breaks something eventually. What I actually watch for is what happens
next: do they own it immediately and walk through what they should have
caught, or do they reach for “the agent did it” as a shield? [Owning the whole outcome](https://sublimecoding.com/blog/professional-owns-the-outcome)
— the failure, the fix, the postmortem — was always the definition of
professionalism I cared about. An agent in the loop doesn’t change whose
name is on the outcome. It’s still theirs.

An agent in the loop doesn’t change whose name is on the outcome.
It’s still theirs.

## Promotion criteria: seniority is what you can safely delegate

The old ladder measured seniority mostly by scope: how big a system
you could own, how much ambiguity you could resolve alone, how many
people’s work you could unblock. Those axes still matter, but a new one
has become the load-bearing one — how much you can hand to an agent and
still stand fully behind the result.

That’s a real skill with a real failure mode on both sides.
Under-delegate — insist on typing everything yourself because you don’t
trust the agent — and you’re slower than peers who’ve built the judgment
to delegate safely, without actually being more careful in any way that
matters. Over-delegate — hand off scope you can’t verify — and you’re
the person whose name is on an outage nobody can explain, because you
can’t explain the code either. I wrote about the version of this that
hits earlier in the pipeline, at the junior level, where [the traditional path to senior
ran through grunt work an agent now does in one prompt](https://sublimecoding.com/blog/stop-making-senior-engineers) — juniors are
losing the reps that used to teach this exact judgment before they’re
old enough to need it. The senior review conversation is the adult
version of the same question: not “how much did you produce,” but “how
far out can you extend your judgment through a tool and still be
right.”

Promotion packets should read differently now. Instead of “shipped X
features, closed Y tickets,” I want to see: the scope of decision this
person can delegate to an agent unsupervised, the size of system they
can review agent output on without a second reviewer, and at least one
specific example of them stopping an agent from doing something that
would have shipped a real defect. That last one is the tell I trust
most, because it can’t be manufactured by volume. You either caught
something or you didn’t.

## The 1:1 script

Here’s what I actually ask, in the room, instead of pulling up a
dashboard. None of these have a single right answer — the answer isn’t
the point, the *specificity* of the answer is.

- “Walk me through a diff you rejected or heavily rewrote this
quarter. What was wrong with what the agent gave you, and how did you
catch it?”

- “Tell me about a time you told an agent not to do something it
wanted to do. What was the shortcut, and why did you stop it?”

- “What’s a bug an agent introduced that made it past your own review?
Where did your process fail, and what did you change?”

- “Pick a PR you approved that you didn’t write. Defend the logic to
me like it’s yours, right now.”

- “What’s something you deliberately did *not* delegate to an
agent this quarter, and why?”

If someone can’t answer the first two with a specific instance — not
a hypothetical, an actual diff or an actual moment — that’s the finding.
It doesn’t mean they’re not contributing. It might mean they’re
rubber-stamping agent output and haven’t been caught yet, which is a
much bigger flag than a light quarter of commits ever was.

## Don’t let a dashboard have this conversation for you

I get why the metrics-dashboard category exists and why it’s selling
well right now — DORA-style delivery metrics, time-allocation
breakdowns, agent-usage telemetry. Some of that data is genuinely useful
as an input. None of it is a substitute for the conversation above, and
the risk I’d flag to any manager evaluating one of these tools is
treating the dashboard *as* the review instead of as one data
point feeding into it.

A dashboard can tell you PR throughput went up. It cannot tell you
whether the throughput came from good problem selection or from an
engineer letting an agent run wild and hoping review catches it. It can
tell you review time went up. It cannot tell you whether that’s because
the reviewer is being appropriately careful or because the code
genuinely got harder to understand. The judgment questions above are
cheap — twenty minutes, no tooling budget — and they get you closer to
the actual signal than a subscription will. If you’ve bought the
dashboard, keep it as a second opinion. Don’t let it replace the
conversation you’re supposed to be having anyway.

This is the same operating-system shift I’ve written about more
broadly: whether the teammate typing the code is a person or an agent,
[the system that has to
hold is trust — clear ownership, outcomes over hours, and a manager who
actually knows what “good” looks like from either](https://sublimecoding.com/blog/trust-is-the-operating-system). Reviews are just
where that system gets tested out loud, once a quarter, in front of the
person it’s about.

## Where to start

If your review cycle is coming up and your rubric still leans on
commit count, PR volume, or “lines shipped,” it’s measuring an axis
that’s actively getting less honest, per the data above, every quarter
agents get better. Rewrite the rubric around judgment before the next
cycle, not after a bad promotion decision forces the question. If you
want a second opinion on what that rubric should look like for your team
specifically — leveling criteria, the 1:1 structure, what a promotion
packet should actually contain now — that’s a conversation I have often
in [fractional engineering-leadership
engagements](https://sublimecoding.com/consulting), and it’s usually a half-day fix, not a quarter-long
process.

---

## What Reverse Acqui-Hires Actually Diligence

URL: https://sublimecoding.com/blog/reverse-acqui-hire-technical-diligence
Published: 2026-07-26
Tags: founders, AI startups, engineering leadership

>
**TL;DR:** Every piece of content on “acqui-hire due
diligence” is a law firm or an M&A shop writing about deal structure
and valuation math. None of it describes what a technical reviewer
actually inspects when the codebase itself isn’t the thing being bought
— which is exactly the shape of the 2024–2025 wave of reverse
acqui-hires (Microsoft/Inflection, Amazon/Adept, Google/Windsurf). In a
reverse acqui-hire, the buyer is licensing IP and lifting out people,
often leaving the company and its repo behind. That flips what
“technical diligence” even means: instead of grading the code, the
reviewer is mapping which humans hold the system in their heads, whether
the codebase can survive their departure, and what “IP” actually refers
to once you separate it from the repo. If you’re a founder on the
receiving end of one of these offers, here’s what actually gets
inspected, and what to have ready before the reviewer opens your
repo.

## What a reverse acqui-hire actually is

A normal acquisition buys the company: the entity, its equity, its
contracts, its codebase, its cap table. A reverse acqui-hire is a
different shape entirely — the buyer licenses the technology (usually
non-exclusively) and hires away the people who built it, while the
original company keeps existing as a separate legal entity, often with a
much smaller team and a repriced business.

The pattern got its name and its playbook in March 2024, when
Microsoft paid Inflection AI roughly $650 million to license its
technology and hired most of its technical staff, including co-founder
Mustafa Suleyman, who became CEO of the newly formed Microsoft AI
division. Microsoft didn’t buy Inflection. Inflection kept existing,
recapitalized, and pivoted ([Bloomberg](https://www.bloomberg.com/news/articles/2024-03-21/microsoft-to-pay-inflection-ai-650-million-after-scooping-up-most-of-staff)).
Amazon ran the same play three months later: a licensing deal for
Adept’s agent technology and select datasets, reported at $330 million
plus a $100 million retention pool, paired with hiring roughly 80% of
Adept’s team, including CEO David Luan ([TechCrunch](https://techcrunch.com/2024/06/28/amazon-hires-founders-away-from-ai-startup-adept/),
[Semafor](https://www.semafor.com/article/08/02/2024/investors-in-adept-ai-will-be-paid-back-after-amazon-hires-startups-top-talent)).
And in July 2025, after OpenAI’s $3 billion acquisition of Windsurf
collapsed inside a 72-hour window, Google structured a $2.4 billion
non-exclusive license for Windsurf’s technology and hired its CEO,
co-founder, and top researchers into DeepMind — while Cognition
separately acquired the rest of the company, its IP, its brand, and its
~$82 million ARR business ([TechCrunch](https://techcrunch.com/2025/07/14/cognition-maker-of-the-ai-coding-agent-devin-acquires-windsurf/)).

The pattern in all three: the buyer isn’t primarily after the repo.
They’re after the people who can rebuild the capability inside their own
stack, plus enough of a license to say they didn’t take it for free. The
regulatory upside is real too — these deals have repeatedly avoided the
antitrust review a formal acquisition would trigger, though the UK’s CMA
has since started designating some of them as mergers anyway ([Fortune](https://fortune.com/2024/07/17/big-ai-acquihire-microsoft-inflection-amazon-adept-antitrust-cma-ftc/),
[TechCrunch](https://techcrunch.com/2024/09/04/uk-regulator-greenlights-microsofts-inflection-acquihire-but-also-designates-it-a-merger/)).
But from the engineering side, the interesting part isn’t the antitrust
maneuvering — it’s that the diligence process for these deals is
inspecting something almost nobody writes about, because almost nobody
who writes M&A content has sat in the room doing it.

## Diligence when the code isn’t the asset

Standard technical due diligence — the kind I’ve described for
round-stage investors — treats the codebase as the asset and asks
whether it’s worth what the deck claims. A reverse acqui-hire reviewer
is answering a completely different question: *if we hire these
twelve people and license this technology, what do we actually get, and
does it survive the move?* That reframes almost every check.

**Key-person risk mapping, taken literally instead of as a line
item.** In round diligence, “key-person risk” is a paragraph in
the memo. In reverse acqui-hire diligence, it’s the entire point of the
exercise — the buyer is *choosing* which specific humans to make
an offer to, and the technical reviewer’s job is to map, file by file,
which of those humans actually hold each critical system in their head.
Not “who’s a good engineer” — who is the one person who can explain why
the retrieval pipeline routes around the vector store for a specific
class of query, and is that person on the list of people getting an
offer. If the person who understands the load-bearing part isn’t being
hired, the deal’s technical value just dropped, and a competent reviewer
will flag that before the term sheet firms up. I’ve written about how I
evaluate this kind of depth when hiring a staff engineer at an AI
startup — [the same
signal, run in reverse](https://sublimecoding.com/blog/how-id-hire-a-staff-engineer-at-an-ai-startup): you’re not grading a resume, you’re mapping
who actually understands the hard part of the system, because that’s the
asset, not the title on their badge.

**Codebase disposability assessment.** This sounds
harsh, and it’s meant to. Since the code is often explicitly
*not* what’s being acquired — the company keeps it, or it gets
left behind, or a separate buyer takes the shell (as Cognition did with
Windsurf) — the reviewer is asking a question round diligence never
asks: is this codebase disposable, and if the acquiring company has to
rebuild the capability with the hired team instead of the repo, how much
time does that cost? That’s a very different exam than “is this code
well-architected.” A brilliant, tightly coupled codebase that only the
departing team can safely touch scores *worse* on this axis than
a mediocre but well-documented one, because the buyer isn’t keeping
either — they’re keeping the people, and the question is how much of
what those people know is trapped in code only they can read versus
knowledge that travels with them regardless of which repo they’re
standing in front of.

**What “IP” means once you separate it from the repo.**
In round diligence, “IP” mostly means the code and the patents, if any
exist. In reverse acqui-hire diligence, the license is being drafted
around a much broader and fuzzier set of assets: the eval suites and
benchmark harnesses (how do you know the system works, independent of
any specific implementation), the data pipelines and the rights to the
training/eval data that feeds them, the prompt libraries and scaffolding
patterns that took months of iteration to get right, and the tacit
operating knowledge of how the model actually behaves in production
versus in the paper. None of that lives in `git log`. A
reviewer here is building an inventory of what’s licensable that
*isn’t* code — because the license terms will be drafted against
that inventory, not against the repo. If your evals only exist as a
Jupyter notebook one person runs locally, or your data-rights paper
trail is “we’re pretty sure we can use this,” that’s the finding that
shows up in this diligence, not “your test coverage is thin.”

**Contribution-graph forensics, but reading for a different
signal.** I still open the commit history first, same as I would
for a funding round. But I’m not reading it to ask “does the team own
the IP they’re claiming” — in a reverse acqui-hire, ownership is usually
clean and undisputed, since the company built the thing itself. I’m
reading it to ask *who actually wrote the parts that matter*,
cross-referenced against exactly who’s on the offer list. If the person
whose commits touch the core inference logic every week isn’t getting
hired, and the reviewer didn’t already know that, that’s a material
finding — the buyer is about to pay for capability they’re not actually
acquiring. This is the same forensic technique as round diligence, aimed
at a completely different conclusion.

The buyer isn’t grading your codebase. They’re grading whether the
twelve people they’re about to hire can rebuild what the repo currently
does — without the repo.

## How this is genuinely different from round diligence

I’ve written before about [surviving
technical due diligence as an AI founder](https://sublimecoding.com/blog/surviving-technical-due-diligence-ai-founder) — that post is about the
investor’s pre-check before a funding round, where the reviewer is
asking whether the team in the room owns the IP they’re claiming and
whether the business is worth the price on the term sheet. Keep those
two mental models separate, because conflating them will make you
prepare for the wrong exam.

Round diligence is fundamentally a *valuation* exercise
wrapped in a technical inspection — the reviewer is trying to find the
gap between what you claim and what’s true, because that gap re-prices
the round. Reverse acqui-hire diligence is a *feasibility*
exercise — the reviewer already knows roughly what they’re paying and
who they want, and the technical review is checking whether the
acquisition actually delivers the capability once you subtract the
entity, the equity, and (often) the repo. One is asking “is this company
worth $X.” The other is asking “if we hire these specific twelve people
and license this specific technology, do we actually get what we think
we’re buying, or did the value live somewhere else.” Different question,
different artifacts, different findings.

## What founders should actually prepare — and what not to scramble on

If you’re a founder fielding one of these offers, or advising a team
that is, here’s where the prep effort should go, roughly in the order it
moves the outcome:

- **Have an honest map of who understands what.** Not
a resume roundup — an actual accounting of which humans hold which
systems in their heads, and how much of that knowledge is written down
versus tribal. This is the single artifact a reverse acqui-hire reviewer
will be building anyway; having it ready and accurate is the biggest
trust signal you can offer, and it directly shapes who gets an offer and
at what number.

- **Separate your IP inventory from your repo.** Write
down, explicitly, what exists outside the code that a buyer might value:
eval harnesses, data-rights documentation, prompt and scaffolding
libraries, operational runbooks for how the system actually behaves in
production. If this only exists as tribal knowledge, that’s the gap to
close before the process starts, not the codebase’s test
coverage.

- **Don’t scramble to make the codebase look
pristine.** This is the opposite instinct from round diligence,
and it trips people up. If the code isn’t the primary asset, spending
your last two weeks on a cleanup sprint or a documentation push aimed at
impressing a code reviewer is effort spent on the wrong exam. Spend it
on the IP inventory and the honest key-person map instead.

- **Be straight about who’s leaving and who’s
staying.** The reviewer will find out anyway which of the
technical staff are taking the offer and which aren’t. A founder who
volunteers this clearly — including which capabilities walk out the door
and which stay behind with the shell company — reads as more trustworthy
than one who lets the reviewer discover it. The same principle runs
through everything I’ve written about how [trust functions as the actual
operating system](https://sublimecoding.com/blog/trust-is-the-operating-system) of a technical organization: the reviewer isn’t
just pricing your technology, they’re pricing whether your account of
your own team is accurate, and that judgment follows you into whatever
comes next — including, often, the people who stayed behind and are now
rebuilding under a new plan.

- **Get your data-rights paper trail in writing before you’re
asked.** If any of your training data, eval data, or user data
has an ambiguous provenance story, that ambiguity becomes the buyer’s
problem the moment they license it — which means it becomes a heavily
negotiated clause, or a walk-away, before it becomes their problem.
Bringing a clean, honest data-rights memo unprompted is worth more here
than almost anywhere else, because the license is being drafted around
exactly this.

- **Decide in advance what happens to the company that’s
left.** Every one of the public examples — Inflection, Adept,
Windsurf — kept existing as a distinct entity after the licensing deal
closed, with a smaller team and a different mandate. If you’re the
founder staying behind rather than the one taking the offer, that’s a
distinct planning problem from the diligence itself, and it deserves its
own honest answer before the ink dries, not an improvised one
after.

None of this is about having a flawless codebase. It’s about knowing,
in advance and accurately, what a reviewer is actually going to find
when they map your team instead of your repo — because in this kind of
deal, the team *is* the repo.

## If you’re navigating one of these

Reverse acqui-hires are still rare enough that most founders — even
technical ones who’ve been through a normal fundraise or acquisition
before — haven’t seen this specific diligence pattern up close. The
questions are different, the artifacts that matter are different, and
the instinct to over-prepare the codebase actively works against you. If
you’re fielding one of these offers, or trying to figure out what
happens to the team and the company that’s left behind afterward, [that’s worth a conversation before the term sheet is
final](https://sublimecoding.com/consulting), not after.

---

## The Career I Built on Work Nobody Wanted

URL: https://sublimecoding.com/blog/career-built-on-work-nobody-wanted
Published: 2026-08-02
Tags: engineering leadership, engineering, staff engineer

>
**TL;DR:** The filter was never tedium. It’s blast
radius: does this force me to understand everything downstream of it? I
built Lavender’s security program from zero to SOC 2 Type II — evidence
collection, control mapping, IAM, MDM, vulnerability management,
incident response — and it taught me more about that company than any
feature I shipped there. The same shape repeats across thirteen years:
the Cognito-to-Auth0 migration covering 225K+ users and $400M+ in
assets, consolidating InsideTrack’s ten services down to six, the
GCP-to-Azure move that saved $350K in six months, twelve offshore
engineers across timezones at Qualaroo. Nobody was fighting to own any
of it. All of it touched more of the system than the interesting work
did. The cost is real — years of low-visibility work with nothing to
show a promo committee — and I’m not going to pretend it’s a hack
instead of a trade.

## The part of the job nobody puts on a slide

Lavender had no security program when I got there. Not a bad one —
none. So the first months of building it to SOC 2 Type II looked like
this: reading every control in the framework and mapping it to something
real in the company, or admitting nothing real existed yet and building
it from a blank page. Writing the access-review policy, then actually
running the access review, cycle after cycle, until it was routine
instead of a fire drill. Inventorying every device that touched company
data so MDM had something concrete to enforce against. Standing up
incident response as a process with defined roles and a runbook, rather
than a Slack channel improvised during the incident it was supposed to
prevent. Chasing evidence for auditors — screenshots, logs, signed-off
tickets, the exact artifact a specific control required — for controls
that felt, in the moment, like pure paperwork.

Why a company signs up for all of that in the first place is a
separate argument, and I’ve made it in [SOC 2 is a
revenue tool, not a security tool](https://sublimecoding.com/blog/soc-2-is-a-revenue-tool-not-a-security-tool). This essay is about what building
it did to me.

None of that is a demo. Nobody in a standup says “excited to show you
the evidence folder this week.” I’m not telling the story to complain
about it. I picked work like this on purpose, at every company on my
resume, for reasons that had nothing to do with liking audits. I don’t
like audits. I like what building one teaches you about a company that
nothing else does.

## The engine: unwanted work is the fastest map of a system

**The work nobody wanted was the work that touched every
system.**

Building a security program from zero doesn’t let you scope yourself
to one service. To map IAM you have to know every system that
authenticates. To do vulnerability management you have to know every
dependency in every repo — including the repos nobody’s touched since
the person who wrote them left. To write incident response you have to
know what actually breaks in production at 2 a.m., who has to be woken
up, and which system’s failure cascades into which other system’s
failure. Nobody assigns that work because it’s fun. It sits there
unassigned until someone takes it, because it’s diffuse and slow to show
results. Those are exactly the properties that make it the fastest way
to learn an unfamiliar org from the inside, rather than from an
architecture diagram someone drew eighteen months ago and never
updated.

Compare it to the interesting work: a new feature, a clean greenfield
service, the model integration everyone wants to demo at the all-hands.
That work teaches you one slice, deeply, and it teaches it well. The
unwanted work teaches you the seams, because the seams are where the
unwanted work always ends up living. Nobody owns a seam. Everybody owns
a slice.

## The receipts

At Lavender, past the security program itself, the same shape shows
up in the infrastructure work. Moving production off GCP and onto Azure
isn’t a feature anyone campaigns for. It’s a six-month slog of
dependency mapping, cost modeling, and staged cutover, with zero
user-visible upside if it goes well and a very visible outage if it
doesn’t. [I’ve written up
the cloud migration in full](https://sublimecoding.com/blog/cut-350k-cloud-spend-six-months) — the short version is that it saved
$350K over six months, and the reason I could scope it at all is that
the security work had already forced me through every system that
touched cost and every system that touched risk. It still cost $50K in
mistakes along the way. Those are in the post too.

At BlockFi, the identity migration off AWS Cognito onto Auth0 covered
225K+ users holding $400M+ in assets, and it had to happen without
forcing a mass logout — meaning every edge case in the auth flow had to
be understood and handled, not just the happy path that demos cleanly.
[The
full migration writeup is here](https://sublimecoding.com/blog/aws-cognito-to-auth0-migration-without-forcing-logout); the mechanics are all in that post.
What’s relevant here is what the project actually was. Nobody wants to
own an auth migration, because auth migrations are where careers get
quietly ended by a wrong assumption about token expiry. I owned it
because it was the fastest way to understand exactly how identity flowed
through a fintech company handling nine figures of customer assets. The
same instinct showed up smaller-scale in the internal tooling I built
for identity repair and account support — unglamorous by definition,
since its whole job is fixing other people’s mistakes — which cut manual
support burden by 65%.

At InsideTrack, the unglamorous version was consolidation — [taking the
microservice footprint from ten services down to six](https://sublimecoding.com/blog/ruby-to-elixir-migration-ten-to-six-services) during a
Ruby-to-Elixir migration. Consolidation produces almost no visible
feature output for months. It also requires understanding every one of
those ten services well enough to know which ones could actually be
absorbed and which were only redundant on paper. That’s a map of the
whole platform, and most engineers on the team never had reason to build
it, because their job was one service.

At AAMP Global, it was ingestion pipelines processing telemetry from
40K+ telematics devices across commercial fleets, running on Kubernetes,
while migrating legacy Ruby services alongside new Elixir feature work.
Infrastructure plumbing that has to be correct at volume and gets
noticed only when it’s wrong.

And at Qualaroo, the unwanted work wasn’t even technical. It was
managing twelve offshore engineers across multiple countries and
timezones while rebuilding growthhackers.com in Rails. Coordinating a
distributed team and absorbing the timezone math nobody local wants to
own is thankless, and it gets handed to whoever says yes first. Saying
yes meant learning how the whole delivery pipeline worked, end to
end.

## What I look for now, on the other side of the hire

I’ve written about [why
the outcome-owner is the hire you want, not the specialist with the
cleanest scope](https://sublimecoding.com/blog/professional-owns-the-outcome), and I won’t re-run that argument here. This is
narrower. The question worth asking a staff-level candidate is what
unglamorous thing they picked up voluntarily in their last role, then
listening for whether the answer maps to a system boundary or to a
single task. [The
staff-engineer loop I’d run](https://sublimecoding.com/blog/how-id-hire-a-staff-engineer-at-an-ai-startup) doesn’t name this signal explicitly, but
it’s what the judgment round is fishing for. Either a candidate can
describe how the unwanted thing connected to three other systems, or
they can’t, and the difference shows up immediately in conversation.

It’s also what I look for a candidate to have *avoided*.
Someone who’s only ever taken interesting work isn’t disqualified by
that alone, but it’s a gap worth probing, because it usually means
they’ve never built the cross-system map the unwanted work forces on
you. That gap shows up later as slow ramp time on an unfamiliar
codebase, at the point when ramp is expensive instead of free.

## The discriminator: does it touch the whole system, or just sit there being annoying

Not all unwanted work is equal, and this is the part that’s easy to
get wrong. There’s a trap version of “take the unglamorous work” that
just means volunteering for tedium: cleaning up a stale spreadsheet,
renaming a batch of variables in one file, sitting through a meeting
nobody wants to attend because the agenda is boring. That work is
genuinely low-value. It’s tedious and isolated, and doing a lot of it
teaches you how to tolerate tedium, which is not a skill worth building
a career around.

The test isn’t whether the work is tedious. It’s whether the work is
diffuse.

The filter I actually apply: before you take on unwanted work, ask
how many other systems it forces you to understand to do it well. A
vulnerability backlog forces you through every service’s dependency
graph, because you can’t triage a CVE without knowing what imports the
vulnerable package and what breaks if you patch it. An identity
migration forces you through every auth-dependent flow in the product. A
cloud migration forces you through every cost center and every network
boundary the company has. Those are diffuse. The work is unwanted
precisely because it’s cross-cutting, and the cross-cutting is the
payoff. A spreadsheet cleanup is unwanted because it’s boring, full
stop, and boring-but-isolated work stays boring however long you do it.
**Pick the unwanted work that widens your map. Skip the unwanted
work that’s just a chore in a box.**

If you’re not sure which one is in front of you, ask what you’d have
to learn to do it *well*, not just to do it *at all*. If
the honest answer is “nothing new,” it’s a chore. If the honest answer
is a list of three or four systems you don’t currently understand, it’s
the kind worth taking, even though nobody’s asking you to spend the time
there.

## The honest cost

I’m not going to dress this up as a growth hack. Building a security
program from scratch, running infrastructure migrations, managing
offshore coordination — none of it produces a demo, a launch tweet, or a
line item that reads well on a promotion packet next to “shipped feature
X, drove Y% growth.” It’s low-visibility by construction. Nobody invites
the person who wrote the incident-response runbook to speak at a
conference about it. The evidence-collection months at Lavender didn’t
make anyone’s highlight reel, including mine, and I wasn’t expecting
them to.

Years of this mean years where the visible artifact of your quarter
is a checklist, a migration nobody notices because it went right, or a
policy document instead of a feature people can click on. If you’re
optimizing purely for the fastest path to a promo packet full of
shippable wins with clean before-and-after metrics, this is not that
path, and I’d rather say so plainly. I made this trade repeatedly and
would make it again. Eyes open, every time.

## Where it actually leads

What it compounds into is narrower and less dramatic than a victory
lap: the ability to walk into a system you’ve never seen and get
oriented fast, because you’ve done exactly that involuntarily, over and
over, as the price of taking the job nobody else wanted. I’ve written
about doing it deliberately in [how I triage an unfamiliar
codebase in ninety minutes](https://sublimecoding.com/blog/triage-a-new-codebase-90-minutes). That skill isn’t innate. It’s the
residue of a decade of being handed the audit, the migration, and the
offshore coordination. You get fast at reading an unfamiliar system by
being forced through enough of them, at the times nobody else wanted to
go first, until the pattern-matching stops feeling like effort.

## What to do this week

- **Name the diffuse work nobody’s covering.** Look for
the thing that’s unowned because it cuts across three or more systems,
not the thing that’s unowned because it’s boring in one place.

- **Take it before it’s assigned to you.** Volunteering
changes what you learn from it. You scope it, so you decide how wide the
map gets.

- **Skip the isolated chores.** If the payoff caps at
“this one thing got cleaner,” let someone else have it.

- **Write down what it taught you about the system, not just
what it fixed.** The map is the asset. The completed task is the
receipt that you built it.

- **Say the cost out loud before you commit to a multi-quarter
version of this.** Know what you’re trading in visibility, and
decide on purpose, before you’re two years into low-visibility work you
never chose.

If you’re a founder or an engineering leader trying to figure out who
on your team is quietly building this kind of map — or trying to build
it yourself before your next raise or audit — that’s a conversation I
have often. [Let’s talk](https://sublimecoding.com/consulting).

---

## What Four Months of Embedded AI Engineering Ships

URL: https://sublimecoding.com/blog/four-months-embedded-ai-engineering
Published: 2026-08-04
Tags: AI startups, founders, AI

>
**TL;DR:** This engagement exists for four situations:
one senior engineer is quietly carrying every AI feature, an LLM feature
is stuck at demo quality, agent infrastructure nobody owns is running in
production, or the model bill has no name attached to it. It’s
composited from real work rather than a named client — the shape below
comes from building production AI systems and agentic workflows in Go
for 100K+ users at Lavender, and building EnergyConnect from zero as
founding engineer. Month one ships a real merge into the path that bills
money. Months two and three are where agent-assisted throughput
compounds, and running this way compressed Lavender’s delivery timelines
by an estimated 40–55%. Month four is the handoff: a
`CLAUDE.md` the next engineer actually reads, a runbook, a
test suite on the paths that touch money, and a named internal owner. It
does not include SOC 2 evidence collection or audit response — [that’s
a different engagement](https://sublimecoding.com/blog/what-a-fractional-security-engagement-actually-looks-like). The whole thing is designed to end.

## The call worth having is narrower than “we need an AI strategy”

The version of this conversation that’s worth having is narrower and
uglier than an AI strategy. It’s usually one of four things.

An LLM feature looks great in the demo and falls apart the first time
a real user does something unscripted. Nobody built the eval harness or
the retry logic behind the happy path, because the demo was the whole
spec.

One senior engineer is the only person who understands how the agent
pipeline works. Every roadmap conversation quietly becomes a question
about that person’s calendar, and every incident becomes a page to
someone who can’t be paged twice a week forever.

Agent infrastructure — orchestration, tool-calling, prompt versioning
— got stood up fast during a sprint and now nobody owns it, including
the person who wrote it. They moved on to the next feature.

The model bill arrives every month with a number on it, and nobody in
the company can explain which feature, which customer segment, or which
retry loop produced that number.

If none of that lands — if you already have a platform team and
someone who owns model spend end to end — this isn’t the engagement you
need, and I’d rather say so now than four months in.

## Where this comes from

This is composited from real work, not a named client engagement. The
shape below comes from doing it in-house, which is where I’d rather
source it from anyway: production AI systems and agentic workflows in Go
serving 100K+ users at Lavender, and building EnergyConnect from zero as
founding engineer starting this April — TypeScript and Node against
Prisma and Postgres, Redis-backed job queues, GCP Cloud Run, with Claude
Code and Codex open all day.

I scope these as fixed-period rotations, six to sixteen weeks, [the band laid out on the consulting page](https://sublimecoding.com/consulting). What
follows is the long end of it: sixteen weeks, call it four months.

## Month one is the first merge

Month one does not produce a slide deck, an architecture diagram, or
a list of recommendations. It produces one merged thing in the path that
actually bills money or serves a real user request, and everything else
that month is subordinate to getting that shipped.

At Lavender the equivalent was production AI systems and agentic
workflows in Go serving 100K+ users, and the work that shipped was
always a concrete change to an inference or retry path, never a document
about one. At EnergyConnect, starting from genuinely zero, month one
meant standing up the real architecture — job queues, portal auth, the
Cloud Run deploy path — and getting a working slice of the product
running end to end on the real deploy path.

Here’s the constraint you only find by shipping: multi-cloud reality
doesn’t show up in a diagram, it shows up in the pager. Lavender ran
across GCP, AWS, and Azure at 99.99% uptime, and a number like that only
becomes real to you once you’re the one being paged for the outage that
threatens it. I’ve written up [what a real multi-cloud
migration cost and saved](https://sublimecoding.com/blog/cut-350k-cloud-spend-six-months). You find the real constraint by being
inside the system with a change ready to ship. The first days usually
look close to [triaging
an unfamiliar codebase in ninety minutes](https://sublimecoding.com/blog/triage-a-new-codebase-90-minutes) — ninety minutes isn’t
enough to understand everything, but it’s enough to find the one thing
worth shipping first.

A demo proves the model works. A merged PR into the path that bills
money proves the system works.

## The eval harness is the thing that was missing

The stuck-at-demo-quality problem almost always has the same root:
there is no way to tell whether a change made the feature better or
worse. The team ships a prompt tweak, someone eyeballs four outputs, it
looks fine, it goes out. Two weeks later a customer finds the case where
it doesn’t, and nobody can say when it broke, because there was never a
baseline to break.

So the first structural thing I build is usually an eval harness, and
it’s much less glamorous than the phrase suggests. A set of real inputs
— pulled from production traffic, not invented — with the outputs a
human actually accepts. A scoring function per case, which for most
features is not a model grading a model but something blunt and
checkable: did it return valid JSON against the schema, did it pick the
right tool, did it cite a document that exists, did it stay under the
latency budget. Then a way to run the whole set on every prompt or model
change and diff the result against the last known-good run.

That’s it. It’s a test suite for a nondeterministic component, and
the reason teams skip it isn’t that it’s hard — it’s that it produces no
demo. It is, in the sense of the [career built on work
nobody wanted](https://sublimecoding.com/blog/career-built-on-work-nobody-wanted), exactly the kind of unglamorous work that turns out
to be load-bearing. Once it exists, the prompt tweaks stop being coin
flips, and the senior engineer who was the only person allowed to touch
the pipeline stops being the only person allowed to touch the
pipeline.

## Months two and three are where agent-assisted throughput compounds

The compounding comes from a deliberate split between what an agent
does and what stays mine. Scaffolding, test coverage on paths I’ve
already designed, first-draft migrations, repetitive refactors across a
codebase I now understand well enough to review fast — that’s agent
work, reviewed line by line, typed by the agent. Architecture decisions,
anything touching auth or money, and the judgment calls about what
*not* to build stay human. I’ve written at length about what that
split looks like day to day in [why
agent-assisted engineering is a genuinely new workflow](https://sublimecoding.com/blog/ai-assisted-engineering-is-a-new-workflow), and I won’t
re-run that argument here.

The honest number: running this way, delivery timelines for AI
features at Lavender compressed by an estimated 40–55%. I say
“estimated” because that’s what it is — an estimate, not a controlled
measurement. The clean-sounding version of that number doesn’t survive a
second question, so here’s the messy one. The commit volume this
produces is real too, and I put actual figures on a comparable stretch
of my own agent-assisted output in [4,154 commits in six
months](https://sublimecoding.com/blog/4154-commits-six-months-ai-agents). Because estimating an agent-assisted timeline is a genuinely
different exercise than estimating a manual one, I wrote [how I estimate client work
when agents do most of the typing](https://sublimecoding.com/blog/estimating-client-work-ai-agents) separately — that’s the piece to
read if the scoping math is the part you don’t trust yet.

The compounding isn’t the agent typing faster — it’s a senior
engineer who used to be the bottleneck spending months two and three on
the ten decisions that actually needed a human.

The hours an agent reclaims from scaffolding and boilerplate go
straight to the architecture calls, the security review, and the “should
we build this at all” conversations that kept getting deferred because
the one senior engineer never had the bandwidth to have them
properly.

## Putting a name on every line of the model bill

The unexplained model bill is the trigger founders are most
embarrassed to say out loud, and it’s the most mechanical one to fix.
The bill is unexplainable because the calls are anonymous: the code
calls the provider, the provider bills the account, and nothing in
between records who asked or why.

The fix is attribution at the call site. Every model call carries a
small set of dimensions — which feature, which customer or tenant, which
environment, whether it was a retry — emitted alongside token counts and
latency into whatever telemetry the company already runs. No new vendor,
no dashboard project. I’ve argued the general case for this in [why an
AI product needs a telemetry layer before it needs a better model](https://sublimecoding.com/blog/your-ai-product-needs-telemetry-before-better-model),
and the Phoenix-specific version is in [instrumenting LLM calls with
telemetry](https://sublimecoding.com/blog/phoenix-llm-telemetry).

What that buys within a couple of weeks is the ability to answer
questions the finance conversation actually turns on. Which feature is
60% of spend. Whether the expensive customer is also the profitable one.
How much of the bill is retries — which, on a pipeline with no eval
harness and generous retry logic, is routinely the largest single line
item and the easiest to cut. It also does something less obvious: once
spend is attributable per feature, “should we build this at all” becomes
a question with a number attached, which is the conversation the
overloaded senior engineer never had time to force.

## Month four is the handoff, and the handoff is the deliverable

“We hand off documentation” is the easiest promise to make and the
easiest to quietly break. Here’s what actually goes into it:

- **A `CLAUDE.md` the next engineer reads on day
one** — the file the coding agent itself loads on every session:
the architecture decisions and why they were made, the parts of the
system that are deliberately unfinished, the footguns specific to this
codebase that cost real time to find.

- **A runbook for the paths that page someone** — the
inference path, the agent retry logic, the job queue — written while I
was still the one being paged for them.

- **A test suite on the paths that bill money or touch customer
data.** If a regression there costs a customer money, it’s
tested. Everything else is a documented judgment call.

- **A named internal owner**, identified and ramped
before month four ends — a specific person who has taken ownership and
demonstrated it by shipping something in the system themselves before I
leave.

If any one of those four is missing when I roll off, the engagement
failed regardless of what shipped in the first three months. **The
code was never the deliverable. The system continuing to run correctly
without me is.**

## What this engagement does not include

SOC 2 evidence collection, audit response, security questionnaire
work, or any of the compliance paperwork a Series A due diligence
process or an enterprise procurement team will eventually demand. That’s
real work, and I do it. It runs on a completely different rhythm — I
laid out that side of the practice in [what
a fractional security engagement actually looks like](https://sublimecoding.com/blog/what-a-fractional-security-engagement-actually-looks-like). I’d rather
point you there than blur the two into one vague offering that’s
mediocre at both. If you genuinely need both tracks running, you need
both. You don’t need one person doing security theater in month two of
an engineering rotation because the acronyms happen to overlap on a
services page.

## The engagement is built to end

The rotation is built to terminate. Success is a named internal owner
who doesn’t need me on the call anymore, a `CLAUDE.md` and a
runbook that outlive the rotation, and a codebase where the next hard
decision doesn’t require reconstructing context that was only ever in my
head. If the engagement is working, the founder’s dependency on me
shrinks every week. That’s the opposite incentive from an open-ended
retainer that quietly becomes permanent, and it’s why I scope this in
weeks with an end date attached.

What that looks like from the founder’s side at the end of month
four: the feature that was stuck at demo quality is in front of real
users, with an eval that catches regressions before customers do. The
senior engineer who was carrying everything alone has a second person
who genuinely understands the system, and neither of them is a single
point of failure anymore. The model bill has a name attached to every
line item, and someone on the team can explain that number in a board
meeting without needing me in the room.

## What to do if you recognized the trigger

I take a small number of these at a time, on purpose — the model
depends on being inside the codebase reviewing real diffs. If you
recognized your company at the top of this piece, the engagement tiers
and how scoping works are laid out on [the
consulting page](https://sublimecoding.com/consulting).

---

## Estimating Client Work When Agents Do the Building

URL: https://sublimecoding.com/blog/estimating-client-work-ai-agents
Published: 2026-08-03
Tags: engineering leadership, agents, AI

>
**TL;DR:** Story points and velocity math were built for
a world where effort was roughly continuous — a medium ticket took
roughly medium effort, every time. Agents broke that assumption. The
same feature can be twenty minutes of agent work or three days of
agent-debugging, and you can’t always tell which one you’re in until
you’re in it. My fix isn’t a smarter estimate — it’s a different
contract. I quote ranges instead of points, bill milestones instead of
hours, and I’m honest with clients about where the variance actually
lives: not in the typing, which compressed, but in the review,
integration, and verification that didn’t. What you’re pricing now is
judgment and blast-radius control, not code output — and the sooner you
say that out loud to a client, the fewer awkward invoices you have to
explain later.

## The SOW doesn’t know what a “medium feature” means anymore

A client asks how long the integration will take and what it’ll cost.
I used to have an honest answer built from years of pattern-matching: a
CRUD resource with auth and three states is a day and a half, a
webhook-driven sync job is three days, a multi-tenant billing change is
two weeks and I want a second engineer’s eyes on it. That
pattern-matching still works for *scope*. It stopped working for
*time*, because time is no longer a stable function of scope.

I’ve written before about running an agent-heavy workflow on my own
project — [4,154
commits in six months](https://sublimecoding.com/blog/4154-commits-six-months-ai-agents), more than my previous four years combined.
The velocity is real. It’s also wildly uneven in a way that a
client-facing estimate has to account for or it becomes a lie by
omission. A feature that looks identical on the ticket — same acceptance
criteria, same rough size — can take an agent twenty minutes to produce
a correct, tested implementation, or it can take three days of the agent
confidently generating something that’s subtly wrong against a part of
the codebase it didn’t have full context on, me catching it, redirecting
it, and doing that loop four more times. Same-sized ticket. An
order-of-magnitude difference in time. That’s not an estimating error I
can train away with more experience — it’s the actual shape of the work
now.

This isn’t just my anecdote. METR ran a randomized controlled trial
with experienced open-source developers in 2025 and found that when they
used current AI coding tools on real tasks in codebases they knew well,
they took **19% longer** to finish — despite forecasting a
24% speedup going in, and despite *still believing*, after the
fact, that AI had made them faster ([METR,
2025](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/)). That gap between felt velocity and measured velocity is
exactly the trap a fixed-price estimate falls into if you quote off
vibes. The variance is real, it’s directionally unpredictable at the
ticket level, and both the agent’s output and your own sense of how fast
you’re moving will mislead you if you let them.

There’s an active discourse in the agile press about the mechanics of
this at the team-process level — Scrum.org has argued that [story
points don’t map cleanly onto autonomous agents](https://www.scrum.org/resources/blog/ai-augmented-scrum-framework-when-half-your-team-autonomous-agents) because agents don’t
experience effort, fatigue, or uncertainty the way the point system was
built to measure, and Agile Insider has proposed [treating
agent throughput as a separate parallel capacity budget](https://medium.com/agileinsider/agentic-ai-is-rewriting-the-sprint-lifecycle-2ecd15bee4c4) rather than
folding it into team velocity. That’s a real and useful conversation if
you’re running internal sprint ceremonies. It’s not this post. I’m not
trying to fix your standup. I’m trying to answer a narrower, more
expensive question: how do you write a Statement of Work, a milestone
schedule, and an invoice for a client when you genuinely don’t know in
advance which end of that variance a given feature will land on.

## Ranges instead of point estimates

The first mechanical change is refusing to give a client a single
number for anything above trivial scope, and saying why out loud instead
of hedging silently.

I quote a range — low estimate assumes the agent nails it on the
first or second pass and I’m mostly reviewing and integrating; high
estimate assumes at least one full detour where the agent’s confident
output turns out to be wrong against something it didn’t have context
on, and I have to catch it, redirect, and re-verify. For most
feature-sized work that range runs somewhere between 2x and 4x, not the
20-30% pad I used to build into hand-written-code estimates. That’s a
bigger spread than clients are used to seeing, and the instinct is to
round it down to look confident. Don’t. A confident wrong number costs
you the relationship the first time you blow through it; an honest range
costs you nothing but a slightly less impressive-looking proposal.

The range isn’t hedging. It’s the most honest number I can give you,
because the honest answer is a distribution, not a point.

What I don’t do is quote the range and then go silent about which end
we’re tracking toward. Once work starts, I tell the client within the
first day or two of a milestone whether we’re running toward the low end
or the high end, and roughly why — “the agent handled the core flow
cleanly, we’re tracking low” or “we hit a data-shape the agent didn’t
expect and I’m doing a manual pass on three files before I trust them,
we’re tracking toward the high end.” That’s a five-minute Slack message.
It converts a range that could look like padding into a range that’s
clearly grounded in what’s actually happening in the work.

## Bill milestones, not hours

The second change is where the money actually attaches. I’ve mostly
moved off pure hourly time-and-materials billing for agent-heavy
engagements, for a reason that has nothing to do with client trust and
everything to do with the fact that hours stopped meaning what they used
to mean. An hour of my time on a ticket that goes smoothly might be
twenty minutes of prompting and review. An hour on a ticket that goes
badly is genuinely an hour of debugging. Billing both the same way
either overcharges the client on the easy ticket or undercharges me on
the hard one, and neither error is visible to the client in real time —
they just see a line item.

Milestone-based billing sidesteps that. I define a milestone as a
demonstrable, working outcome — “the integration flow live in staging,
all three of its paths tested” — not a time box, and price it against
the range I quoted, not against logged hours. The client pays for the
outcome; I absorb the variance inside the range I already flagged; and
if a milestone runs consistently toward the high end across a project,
that’s the signal to renegotiate scope or price going forward, in the
open, rather than a signal I quietly eat by working unpaid hours or a
signal the client discovers as an inflated final invoice.
Time-and-materials isn’t gone from my toolkit — it’s still the right
call for genuinely exploratory work where even the scope is uncertain,
which I’ll get to below — but for defined feature work, milestones are
the honest unit now because they’re the unit the client actually cares
about.

## The part that didn’t compress

Here’s the part of this that a lot of agent-era pricing conversations
skip, because it’s less exciting than the productivity story: not
everything got faster, and pricing has to reflect that split
honestly.

The 2026 AI Engineering Report, cited in Faros AI’s research, found
that under high AI adoption, median time to first PR review is up
156.6%, average time spent in review is up 199.6%, average PR size is up
51.3%, and bugs found per PR are up 54% ([Faros
AI, 2026](https://www.faros.ai/blog/ai-code-quality-senior-engineer-review-burden)). I’ve cited that stat before writing about [what a performance
review has to measure now that commit volume is agent-inflated](https://sublimecoding.com/blog/performance-reviews-when-agents-type), and
it applies just as directly here: the generation step compressed hard,
and the review, integration, and verification steps didn’t compress at
all — they got *more expensive*, because there’s more volume to
review, the code is stylistically clean enough that bugs hide behind
good formatting, and the reviewer has to actually understand the logic
every time instead of pattern-matching on ugly code as a tell.

That’s the part of my estimate that still looks like the old
estimate. Reviewing a generated diff for correctness against business
logic the agent didn’t fully have, tracing an integration point through
a system the agent only saw one file of, deciding whether a change is
safe to ship to production without breaking three other paths that share
the same code — none of that got faster, because none of that was ever
bottlenecked on typing speed. It was always bottlenecked on judgment,
and judgment doesn’t compress just because the draft arrives faster. If
I quote a client a number that assumes the whole feature moves at
generation speed, I’ve quietly promised them a review-and-verification
pass that doesn’t exist, and the gap shows up as either a missed
deadline or a corner cut on the pass that catches the expensive bug.

## Fixed-bid vs. time-and-materials, reconsidered

The old rule of thumb was: fixed-bid when scope is well understood
and the risk is mine to manage, time-and-materials when scope is
genuinely unknown and the client should share that risk. That rule still
holds, but the inputs to “well understood” changed.

Scope being well understood used to mean I’d built something like it
before and could estimate the *hand-written* effort with
confidence. Now it has to mean something narrower: I’ve built something
like it before *with an agent*, in a codebase with similar shape,
and I have a real sense of where that specific kind of feature tends to
land in the variance — closer to the twenty-minute end or closer to the
three-day end. That’s a much smaller set of things I’ll fixed-bid
confidently than it used to be, because the agent-era track record on
any given feature type is younger than my hand-written track record
was.

For genuinely novel integration work, unfamiliar third-party APIs, or
anything touching a part of the client’s system I haven’t worked in —
the places where an agent is most likely to confidently produce
something wrong because it’s inferring conventions instead of following
ones it’s seen — I lean time-and-materials with the milestone framing
layered on top: bill against demonstrated progress, not raw hours, but
don’t pretend I can fix-bid a number I genuinely don’t trust yet.
Fixed-bidding scope I can’t confidently place in the variance range
isn’t confidence, it’s a bet I’m making with the client’s budget instead
of my own, and I don’t like that trade even when I’d probably win
it.

## What the estimate is actually pricing

The honest reframe, the one I try to say plainly to a client rather
than leave implicit in the invoice, is that the thing being priced
changed. It was never really pricing typing speed — that was always a
rough proxy — but it’s an especially bad proxy now, and pretending
otherwise is the fastest way to either underprice yourself or
overpromise a client.

What a client is actually paying for in an agent-heavy engagement is
judgment about what to build and what not to, the review discipline to
catch the plausible-looking wrong answer before it ships, and
blast-radius control — the discipline of knowing which change is safe to
let run wide and which one needs a narrower, more careful path because
the failure mode is expensive. I’ve written about that same blast-radius
framing from the inside of my own project: shipping faster means
shipping more surface area, and more surface area means more that can
break at 2am, which is exactly the argument I make in [why AI won’t shrink your team](https://sublimecoding.com/blog/ai-wont-shrink-your-team)
— the velocity doesn’t remove the need for the people who catch what it
breaks, it raises the stakes on them. When I quote a client, that’s the
labor the number is actually covering, whether or not I say so. Saying
so is the difference between a client who understands what they’re
buying and one who thinks they’re buying code by the pound.

It’s also, not coincidentally, close to the same lens an investor’s
technical reviewer applies during diligence — reading commit history for
who’s actually exercising judgment rather than who’s generating volume,
a pattern I’ve laid out in [surviving
technical due diligence as an AI founder](https://sublimecoding.com/blog/surviving-technical-due-diligence-ai-founder). Clients doing diligence on
a contractor and investors doing diligence on a founding team are asking
a version of the same question: is the judgment real, or is the output
just volume dressed up as progress.

## Say the quiet part to the client

The uncomfortable option, and the one I think a lot of consultants
are quietly taking right now, is pocketing the variance. Quote the old
hourly rate, let the easy tickets run fast and profitable, eat the hard
tickets’ extra hours without mentioning it, and let the average work out
in your favor over enough projects. It’s not fraud, exactly. It’s also
not the operating system I run the rest of my practice on, and clients
notice the difference between a consultant who explains the shape of the
risk up front and one who lets them find out the hard way.

I’ve written elsewhere about running teams — and agents — on [trust as the actual operating
system](https://sublimecoding.com/blog/trust-is-the-operating-system) rather than the process artifacts stacked on top of it. The
client relationship runs on the same substrate. Telling a client “this
range is wide because the work is genuinely bimodal now, here’s why, and
here’s how I’ll keep you posted on which end we’re tracking toward”
costs you a slightly less polished-looking proposal in the first
meeting. It buys you a client who doesn’t feel ambushed by the invoice
on the milestone that ran long, and who trusts the next range you quote
them because the last one held up. That trade is not close.

## If you’re scoping an agent-heavy engagement

If you’re staffing or scoping work where agents are doing a
meaningful share of the building — whether you’re the one hiring the
consultant or the one writing the SOW — this is exactly the kind of
thing worth walking through before the contract is signed rather than
after the first milestone slips. [Let’s talk about
how the engagement should actually be structured](https://sublimecoding.com/consulting).

---

## What I Check When I Inherit a Vibe-Coded App

URL: https://sublimecoding.com/blog/inheriting-a-vibe-coded-app
Published: 2026-08-07
Tags: founders, engineering leadership, AI

>
**TL;DR:** A new service category has formed with three
different names and no coordination between the vendors selling it —
evidence that “take over an app someone else built with an AI agent and
no longer understands” is now a real, priced engagement, not a
hypothetical. This is the intake playbook I run when I inherit one: what
I open first (dependency manifest, secrets handling, the honest state of
the tests, the deploy path, the data model), the specific failure
signatures abandoned AI-built systems share, and the decision most
people get backwards — kill-and-rewrite versus stabilize-in-place. I
price the discovery audit as a fixed fee, not hourly, because the thing
you’re actually selling in week one is a verdict, not time.

## The scenario, and why it’s different from a normal handoff

A normal codebase handoff is uncomfortable but tractable. There’s a
departing engineer you can still Slack for a week, a README that’s stale
but not fictional, a test suite that at least ran once. The
AI-build-rescue scenario removes the one thing every handoff playbook
quietly assumes: a person who can answer “why is it built this way” from
memory.

Three versions of this show up, and they read differently at intake
even though the audit is nearly identical:

- **The departed contractor.** Someone was hired to move
fast with an agent, delivered something that demoed well, and left when
the contract ended. The commit history is dense but the commit messages
are agent-generated boilerplate — “fix bug,” “update component,”
repeated hundreds of times, none of it explaining a decision.

- **The employee dabbler.** Someone inside the company
who isn’t an engineer — ops, a PM, a founder’s cousin — built a
genuinely useful internal tool with an agent because it was faster than
filing a ticket, and now half the company depends on it and nobody who
can read the code owns it.

- **The founder who hit their ceiling.** The most common
version I expect to keep seeing: a non-technical or lightly-technical
founder built a real product with Claude Code or Cursor, got it to
paying customers, and hit the point where the next feature requires
understanding the system as a whole instead of describing the next
change in isolation. This is the inverse of the seat I’ve written about
for [an AI
founder walking into technical due diligence](https://sublimecoding.com/blog/surviving-technical-due-diligence-ai-founder) — same underlying
system, but here nobody’s grading it for an investor. Somebody just has
to be able to change it safely, starting now.

In all three, the constant is the same: no docs, no tests you can
trust, and no living person to ask “why.” That’s precisely the gap I’ve
written about from the new-hire side in [onboarding an agent-heavy
codebase](https://sublimecoding.com/blog/onboarding-agent-heavy-codebase) — except here there’s no one left to ask, ever, and the
whole system has to be understood from evidence alone. That’s the job
description for the first week.

## The market told on itself

I went looking for who’s already selling “fix the app an AI built and
the builder walked away from,” expecting a niche. I found three
unrelated vendors, three different names, live right now. Coretechs
Consulting — a Kensington, MD dev shop running since 1986 — sells [AI Code Rescue](https://ai-code-rescue.com/). Nine by Six, a
web agency, sells [AI
Build Rescue](https://ninebysix.com/ai-build-rescue) for projects that stalled 80% of the way to launch on
Cursor, v0, Lovable, or Bolt. Sam Carlton, a solo consultant, offers [AI App Rescue](https://samcarlton.com/ai-app-rescue/) —
workflow mapping and a 30-day plan.

None of these three cite each other, and none reads like it copied
the others’ positioning — they landed on the same offer independently,
which is stronger evidence than any one of them alone that there’s now
real inventory of abandoned AI-built systems to work through. And the
pricing shape one of them publishes — Nine by Six lists its audit from
$499 before any fix work — is close to right, for reasons I’ll get to.
The pattern behind why this category exists is the one I’ve been
documenting from the inside for a while: [agents optimize for
tests going green, not for a system a stranger can maintain a year
later](https://sublimecoding.com/blog/why-ai-agents-write-unmaintainable-code), and eventually somebody other than the original builder has
to open the result.

## What I open first

The order matters, because each step either rules out a bigger
problem cheaply or tells you where the expensive one is hiding. I run
these roughly in this sequence on any inherited system, agent-built or
not — but an agent-built one changes what I expect to find at each
stop.

**The dependency manifest, before anything else.**
`package.json`, `mix.exs`,
`requirements.txt` — whatever the language uses. An agent
working fast and unsupervised tends to add a new library for a problem
the codebase already solved three files over, because it only knows
what’s in its context window and has no map of what already exists in
the repo — the same reinvented-helper pattern that shows up constantly
in agent sessions. A manifest with four different HTTP clients or three
different date libraries isn’t a style complaint; it’s a fast read on
how much of the system was built with no one holding a consistent mental
model of it.

**Auth and secrets handling, second.** Where do
credentials live, how does a request get authorized, and is there a
consistent boundary or does every route roll its own check. This is a
five-minute grep, and it’s the single highest-signal five minutes in the
whole audit — I’ll get into why in the failure-mode section below.

**Test coverage as it actually is, not as it’s
described.** Run the suite. Read what it actually asserts, not
the count of files that pass. A “we have tests” claim in an inherited
AI-built system needs verification more than almost any other claim,
because generating a test that passes is exactly the failure mode agents
default to when the intent behind a feature was never made explicit.
This is a different read than the same check in a diligence engagement,
too — there you’re grading the claim’s credibility for someone else’s
decision; here you’re deciding whether you can safely change a line
without breaking a customer who’s already relying on it.

**The deploy path.** How does code get from a laptop to
production, who has the credentials, and is there a rollback that’s ever
actually been exercised. Vibe-coded systems frequently have a deploy
path that only ever worked because one person’s local machine had the
right environment variables set — which is fine until that person is the
one who left.

**The data model, last, because it’s the slowest to audit and
the hardest to fix later.** Is there a schema at all, or is shape
enforced only by whatever the frontend happens to send? Are there
migrations, or did the table get altered by hand against production more
than once? This is the one place where “we’ll fix it later” gets
exponentially more expensive with every week of new data written against
an unstable shape.

## The failure signatures, and which ones are structural

Not every gap in an inherited system is equally alarming — the
calibration question is the same one I use in due diligence: is this
expected at this system’s stage, or is it a genuine structural risk. A
few signatures show up often enough — in my own agent-driven work, and
in the scan data above — that they’re worth checking by name.

**Plaintext or hardcoded secrets.** This is the one I
check first because it’s the cheapest to find and the most binary to
grade — either the key is in the repo or it isn’t. It’s also not a
hypothetical risk with AI-generated code: GitGuardian’s 2026 State of
Secrets Sprawl report found Claude Code-assisted commits leaking secrets
at a 3.2% rate against a 1.5% baseline across all public GitHub commits
— roughly double, though the report is careful to attribute the gap to
developers overriding or ignoring guardrails under time pressure, not a
tool failure ([GitGuardian,
2026](https://blog.gitguardian.com/the-state-of-secrets-sprawl-2026/)). Separately, Escape.tech scanned roughly 1,400 applications
built on vibe-coding platforms and reported 2,038 highly critical
vulnerabilities and 400+ leaked secrets, including 175 instances of
exposed PII — bank account data among it ([Escape.tech](https://escape.tech/state-of-security-of-vibe-coded-apps)).
If you’re inheriting a system built the way most of these were, assume
there’s an exposed credential somewhere until you’ve grepped and
disproven it — my checklist for the actual remediation is in [managing secrets in an
agent-driven workflow](https://sublimecoding.com/blog/ai-agent-secrets-management).

**Tests that assert nothing.** Not “no tests” — that’s
an honest, visible gap. This is a green suite that passed because it was
written to observe the code’s current behavior and lock it in, not to
check the behavior against what the feature was supposed to do. It’s the
specific pattern I’ve written up in detail from my own review sessions
in [the
tautological-test failure mode](https://sublimecoding.com/blog/things-ai-coding-agents-get-wrong), and I’m not going to re-list the
catalogue here — the short version for intake purposes is: read the
assertions, not the pass count, and ask of each one whether it would
fail if the behavior were wrong.

**Dead endpoints and stale routes.** Agents rarely
delete. Ask one to replace a flow and the old route often survives,
unreferenced by the current frontend but still live, still deployed, and
still an attack surface nobody’s thinking about because nobody remembers
it exists.

**Mock or seed data left in a production code path.** A
fixture built to make a demo look good during development that never got
swapped for the real integration — a hardcoded response, a stubbed
payment confirmation, a “success” branch that never actually calls the
service it claims to call. This one is dangerous specifically because it
looks like working code in a walkthrough and is invisible until the
exact moment someone relies on it being real.

## Kill-and-replace vs. stabilize: the actual decision

The instinct on seeing a system with several of the above is to burn
it down and start clean. That’s usually the wrong call, and it’s worth
being explicit about why, because the wrong call here is expensive in a
specific way: a rewrite throws away the part of the system that’s
actually validated — the fact that real users are using it and it mostly
works.

The framework I use is closer to triage than to a rewrite-or-don’t
binary:

- **Stabilize in place** when the core data model is
sound and the failure signatures are concentrated in a knowable set of
files — the auth boundary, one integration, the deploy script. You can
fix a bounded set of specific things faster than you can rebuild and
re-validate an entire system, and you keep the thing that’s genuinely
hard to fake: production traffic that already exercises the happy paths
correctly.

- **Kill and rebuild the specific module, not the app**
when one component is rotten in a way that’s cheaper to replace than to
understand — commonly the auth layer, if it was clearly bolted on
route-by-route with no consistent boundary, or the data layer, if the
schema was never designed and just accreted from whatever the frontend
sent over time.

- **Full rewrite** is the answer far less often than
instinct suggests, and it’s only right when the data model itself can’t
be trusted — when you genuinely don’t know what’s real in the database
because nothing enforced its shape — or when the system has no
meaningful production usage yet, so there’s nothing validated to
preserve by keeping it.

The question in front of you isn’t whether the code is good. It’s
whether the business the code supports is worth more standing than it is
rebuilt.

That reframe is the actual point of the audit. A technically
embarrassing codebase serving real paying customers with an intact data
model is usually worth stabilizing. A clean-looking one with no data
integrity and no users yet is often cheaper to replace than to fully
understand.

## Pricing the takeover: fixed-fee audit, then a retainer

Only one of the three above publishes a price for the audit itself,
which I think is the piece they’re all circling and worth naming
explicitly, because it’s a different pricing shape than most of my
consulting work. I’ve written separately about [pricing ongoing feature
work when an agent is doing the building](https://sublimecoding.com/blog/estimating-client-work-ai-agents) — that post is about
milestone billing for defined, forward-looking scope, where the client
and I both roughly know what we’re building next. A takeover audit is
the opposite problem: neither of us knows what’s actually in the system
yet, which makes hourly billing the worst option for both sides — it
pays me to find more problems and pays the client for uncertainty they
can’t budget against.

So I price the intake audit as a flat fee, scoped to a fixed output:
a written findings report, the failure-signature checklist above scored
against the actual system, and the kill-vs-stabilize recommendation with
a rough cost estimate for whichever path it points to. That’s the
distinction from fixed-bidding remediation, which I won’t do
sight-unseen: I’m fixing the price of the report, not of whatever the
report turns up. That’s a bounded deliverable regardless of what I find
inside, which is exactly the property a nervous founder or a company
that just discovered its internal tool has no owner needs from the first
engagement — a fixed number they can say yes to before committing to
anything bigger.

What comes after the audit is a retainer, not a per-ticket quote,
because stabilizing an inherited system isn’t a fixed scope of work —
it’s an ongoing judgment call about what to fix now, what to fix next,
and what’s fine to leave alone, applied by someone who now actually
understands the system because they did the audit. That’s a different
shape of engagement than shipping the next feature on a system everyone
already understands, and pricing it like the same thing undersells the
actual value: the audit is buying certainty about what you own, and the
retainer is buying the judgment to keep making it better instead of
worse.

## If you’re staring at one of these right now

If you’re a founder who built past the point your agent can carry
alone, an operator who just found out the internal tool the whole team
depends on has no owner, or you’re the one who has to make the call on
whether to hand a client’s inherited system to a rescue vendor or bring
someone in-house to own it — that first audit is the highest-leverage
hour you can spend, and it’s the one most people skip because it feels
like paying to find bad news. [Let’s talk about
what that first audit should actually cover for your system](https://sublimecoding.com/consulting).

---

## Every Insane Codebase Was Once Rational

URL: https://sublimecoding.com/blog/insane-codebases-were-once-rational
Published: 2026-08-08
Tags: engineering, engineering leadership

>
**TL;DR:** A new grad running his first project asked me
how to know whether his architecture decisions are right. They aren’t
right or wrong — they’re trade-offs, and certainty was never available
at any level of seniority. The reframe that helps: every codebase you’ve
inherited and cursed was a sane response to constraints that are now
invisible to you. Which means your decisions get judged the same way, by
someone who won’t have your constraints either. So decide for the
horizon you can actually see — three to six months — and stop paying
interest on a future you’re guessing at. The exception matters and I’ll
name it: one-way doors get the long view, and there are fewer of them
than you think.

## The question underneath the question

A new grad reached out a few weeks ago. About a year in, he’s running
an entire project — front end, back end, plus a younger developer
reporting to him. He asked whether I’d mentor him, and when I asked what
specifically he was stuck on, the answer was the one I expected:

He’s not sure he’s making the right architecture decisions.

Then, a few days later, the more honest version: he assumes there’s
some looming factor he hasn’t considered — something a more experienced
person would have seen coming — and that assumption is what stalls him.
Lately more of his week goes to fires than to building. He’s worried
he’s already built an insane codebase.

I want to answer the second version, because the first one has no
answer.

## There is no right decision, only trade-offs you can defend

You can never be certain you’re making the right architecture
decision. Not at one year, not at thirteen. Most things in software
aren’t correct-or-incorrect, they’re trade-offs with different bills
arriving at different times. Postgres or a queue. One service or three.
Fat model or thin. Every one of those choices buys something and
mortgages something else, and which side of the trade you want depends
entirely on facts about your company that will change.

That’s not a comforting non-answer. It’s the actual mechanism, and
once you see it as a mechanism you can work with it instead of waiting
for a certainty that isn’t coming.

## The reframe: every insane codebase was once rational

Here’s the sentence I keep coming back to, and the one I gave
him.

Every insane codebase is a perfectly sane response to constraints
that existed when it was written.

Think about the worst codebase you’ve worked in. The one with the god
object, the three overlapping auth paths, the config system nobody can
explain. It feels like it was built by people who weren’t paying
attention.

It almost never was. It was built by people making reasonable calls
against constraints you can’t see from where you’re standing:

- A deadline tied to a funding round, where shipping in six weeks was
the difference between a company and no company.

- A team of two who couldn’t operate five services, so everything went
in one, correctly.

- A library that was the obvious choice in 2019 and got abandoned in
2022.

- A customer contract that required a specific behavior, signed before
anyone knew what it would cost to build.

- A person who understood the whole thing and left, taking the
reasoning with them and leaving only the artifact.

Strip the constraints out and the code looks insane. Put them back
and most of it looks like what you’d have done. The insanity isn’t in
the decisions. It’s in the gap between the decisions and the context
that’s no longer attached to them. I go through this exercise in detail
whenever I take over someone else’s work — [the questions I ask when I
inherit a vibe-coded app](https://sublimecoding.com/blog/inheriting-a-vibe-coded-app) are mostly archaeology, trying to recover
the constraints before I judge the code.

Now turn it around, because this is the part that’s actually useful
to a nervous first-time lead. **Your decisions will be judged the
same way.** Someone will open your repo in three years without
your deadline, your headcount, your customer, or your 2026 tooling, and
they will think you weren’t paying attention. That’s not a failure you
can avoid by being smarter. It’s the default outcome of time
passing.

Which means optimizing to look good to that future person is
optimizing for an audience whose constraints you cannot know. It’s not
achievable. Aim somewhere else.

## Aim at three to six months

The horizon you can actually see is about a quarter or two. That’s
the window where you have real information: the customers you have now,
the team you have now, the roadmap that’s been committed to, the tools
that exist today. Inside that window, your predictions are grounded.
Outside it, you’re doing fiction with extra steps.

So make the decision that’s right for the next three to six months,
and let the version after that be made by someone — probably you — who
has information you don’t have yet.

This is agile’s actual point, before it got turned into a meeting
schedule. Not “move fast.” Not “two-week sprints.” The original
manifesto put it as [responding to
change over following a plan](https://agilemanifesto.org/) — you change direction as information
arrives, which only works if you haven’t already spent the budget
pre-building for a direction you guessed.

That horizon has gotten shorter, not longer, and I say this as
someone who mostly builds with agents now. The tooling under your
project is changing faster than the project. Model capabilities, the
agent harnesses around them, and the pricing of the whole thing — which
is currently [an
arbitrage with an unknown shelf life](https://sublimecoding.com/blog/ai-coding-agent-costs-real-numbers). An architecture chosen in
January to work around a limitation that no longer exists in July is not
a wise long-term bet. It’s a fossil.

## What “optimize for the near term” actually means in code

This is where the advice usually goes vague, so let me make it
concrete. Optimizing for the visible horizon is not “write it sloppy.”
It’s a specific set of preferences:

**Prefer decisions you can reverse over decisions that are
correct.** A choice you can undo in a day doesn’t need to be
right. It needs to be undoable. Spend your certainty budget on the ones
you can’t unwind.

**Prefer deletable code over extensible code.**
Extensibility is a bet on a specific future shape; deletability is a bet
that you’ll be wrong about the shape, which is the safer bet. A module
with two callers and no config surface can be ripped out on a Tuesday —
a plugin system with one plugin cannot. This is the same reason the
abstraction you add for a second use case that doesn’t exist yet is
usually wrong about it. Generalizing from two real examples is easy.
Guessing one is not.

**Write down why, not just what.** This is the one that
pays back the most, and almost nobody does it. Most of the pain of
inheriting a codebase is missing constraints, not missing docs. A
three-line comment or ADR saying “single service because two people and
a March deadline; split when we’re past four engineers” converts a
future insane decision into a legible one. That’s the entire
difference.

**Let the pain point tell you when to change.** Don’t
restructure on a schedule or because a blog post said to. Restructure
when something concrete hurts — deploys are slow, this module breaks
every time that one changes, onboarding takes two weeks. Then fix that.
Then stop.

## The exception: one-way doors get the long view

I’d be selling you something if I stopped there, because “decide for
six months” is genuinely bad advice for a small set of decisions. Some
doors only open one way, and the whole reason the three-to-six-month
rule works is that you’ve correctly sorted out the ones it doesn’t apply
to.

The one-way doors, roughly:

- **Data model and storage semantics.** Not which
database — you can move databases. What you promise about your data:
uniqueness, ordering, what a record means, what you kept and what you
threw away. Discarded data is unrecoverable, and a wrong meaning
propagates into every report and integration built on it.

- **Your public interface.** API shapes, webhook
payloads, URL structure, anything a customer or partner has written code
against. Once someone else depends on it, changing it is their project,
not yours.

- **Authentication and identity.** Migrating identity
providers with live users is a genuinely hard project — [I’ve
done one across 225K+ users without forcing a mass logout](https://sublimecoding.com/blog/aws-cognito-to-auth0-migration-without-forcing-logout), and the
reason it was hard is that every edge case in every auth flow had to be
handled, not just the happy path.

- **Anything touching money or compliance-relevant
records.** Ledgers, audit trails, retention. Getting these wrong
is a legal problem, not a refactor.

For these, take the long view, think hard, and be slower than feels
comfortable. For everything else — service boundaries, folder structure,
which queue, which state library, how you organize your tests — decide
fast, on near-term evidence, and stay ready to change your mind. The
pattern I see most often is the inversion: weeks of agonizing over
reversible layout decisions, and a permanent data-semantics call made in
an afternoon because it didn’t feel like a big deal.

## The looming factor usually isn’t there

Back to what he actually said, because it’s the part that matters
most and it isn’t technical: he assumes there’s some factor he hasn’t
considered, and the assumption itself is what’s slowing him down.

Sometimes there is one. Usually there isn’t. What’s actually there is
the ordinary gap between the decision and the outcome — the thing
everyone experiences and almost nobody says out loud, because senior
engineers narrate their past decisions with the confidence of hindsight
and it reads like they knew.

They didn’t know either. They found out, then told the story
backwards.

You never know as much as you will in hindsight. That’s not a
seniority problem you can study your way out of; it’s the shape of the
work. The senior engineer’s real advantage isn’t foresight, it’s a
bigger library of failure modes they’ve personally hit — which is
exactly what you’re building right now, in the fires you’re embarrassed
about putting out. [The path
to senior always ran through that pile](https://sublimecoding.com/blog/stop-making-senior-engineers), and I’ve written elsewhere
about how [the work
nobody wants is what builds the map fastest](https://sublimecoding.com/blog/career-built-on-work-nobody-wanted). Those fires aren’t
evidence you’re doing it wrong. They’re the mechanism.

The right posture isn’t confidence and it isn’t paralysis. It’s
making the call, writing down why you made it, and staying willing to be
wrong out loud in three months. Which, incidentally, is also the thing
that makes people trust you as a lead — [certainty theater is a much
worse look than a documented trade-off](https://sublimecoding.com/blog/trust-is-the-operating-system).

## What to do this week

- **Sort your open decisions into two-way and one-way
doors.** Two-way: decide today, note the reasoning, move.
One-way: slow down, get a second opinion, and be sure you’re right about
what it means, not just what it is.

- **Find the oldest decision you’re still paying for and ask
what constraint produced it.** If the constraint is gone, you’re
allowed to change the decision. If you can’t find the constraint, that’s
your first thing to write down.

- **Add a `why` line to your next non-obvious commit
or PR.** Constraint, horizon, trigger to revisit. Three lines.
It’s the highest-leverage documentation you’ll ever write.

- **Stop pre-building for the second use case.** Find one
abstraction you added for a case that never arrived, and delete it.

- **Name your next architecture decision’s expiry date out
loud.** “Single service until we’re past four engineers.” A
decision with a stated horizon stops feeling like a permanent verdict on
your judgment.

If you’re leading a small team and trying to figure out which
decisions deserve the long view — or you’ve inherited something that
looks insane and you need to know what’s salvageable — [let’s talk](https://sublimecoding.com/consulting).

---

## Anthropic's Extreme Scenario Has a Precondition

URL: https://sublimecoding.com/blog/anthropics-extreme-scenario-has-a-precondition
Published: 2026-09-10
Tags: AI, founders, AI startups

>
**TL;DR:** Anthropic’s Economics team published a
scenario explorer this week putting an “Extreme” 2030 outcome (US GDP up
32.4%) on the table, and named its engine: “likely driven by recursively
self-improving AI systems.” The same week, Anthropic’s alignment science
lead said publicly that Anthropic does not yet have a plan to align
superintelligence and is “not clearly on track to.” The upside case and
the unsolved problem share one precondition. I’m skeptical of the
specific number attached to the risk this week. It’s unfalsifiable, and
it flatters the speaker either way it lands. What doesn’t change: a
pre-Series-A founder’s security posture, this week, because of any of
it.

## Two documents, same company, same week

In September 2026, [Anthropic’s
Economics team published v1.0 of an Econ Scenario Explorer](https://www.anthropic.com/institute/econ-scenarios), built
off their technical report *Economic Scenarios for Transformative
AI* (Korinek, Jones, Sacher, Cotter, McCrory, 2026). It models three
2030 outcomes for the US economy at 2025 price levels: Modest (+1.6%,
$34.1T), Substantial (+8.3%, $36.3T), and Extreme (+32.4%, $44.4T).

Only one of those three names its own engine. The page’s verbatim
description of the Extreme case: “in the extreme scenario, AI drives a
completely transformed, unprecedented economy, likely driven by
recursively self-improving AI systems and a faster rate of AI adoption.”
Not “faster models.” Adoption speed is in there too, but it’s paired
with recursive self-improvement, and only in this scenario. That pairing
is the thing that gets you from Substantial to Extreme.

On September 9, on X, [Jacob
Coxon, a pretraining researcher with three years across OpenAI and
Anthropic, announced he was resigning](https://x.com/hilbertspaess/status/2097476196791709843), writing that “neither company
is acting responsibly,” that they “are racing straight to self-improving
superintelligence and gambling with our lives,” that “the people
building AI earnestly believe that it could kill us all by the end of
the decade,” and that “no other human activity poses this level of
danger.” He gave no percentage. [TechCrunch’s
coverage](https://techcrunch.com/2026/09/09/gambling-with-our-lives-anthropic-researcher-quits-warns-against-self-improving-ai/) and [TechSpot’s](https://www.techspot.com/news/113788-anthropic-researcher-resigns-after-warning-ai-race-could.html)
both quote him making no numerical claim, and Anthropic “did not
immediately return a request for comment” per TechCrunch.

Evan Hubinger — Anthropic’s alignment science lead, still at the
company — replied on X the same day. His words, not Coxon’s: [“Jacob is
correct here – we really do earnestly believe AI could kill all humans!
I personally think it is >10% within the next decade.”](https://x.com/EvanHub/status/2097497037956891126) And: “I
believe Anthropic is trying its best, but we do not yet have a plan to
solve alignment for superintelligence and are not clearly on track
to.”

That’s the joint. The Economics team’s highest-growth scenario names
recursive self-improvement as its likely engine. Anthropic’s own
alignment lead says the company building that engine doesn’t have a plan
to align what comes out of it. Read the two documents in either order
and you land in the same place: the upside case has a precondition, and
the precondition is exactly the part nobody claims to have solved.
That’s the structure. The number Hubinger attached to it is a separate
matter, and I don’t buy it.

## What the econ page is, and isn’t

It’s worth being precise about what the scenario explorer actually
is, because it’s easy to read more certainty into it than it contains.
It’s a task-bundle model: a structured way of asking what happens to
output, wages, and labor’s share of income at three different automation
intensities. The page itself carries the caveat: “Like every model, it
is a stark simplification of a complex reality.” It assigns no
probability to any of the three scenarios. The word “probability”
doesn’t appear on the page. (A reader poll further down does show
“around 10% of respondents have views in line with the extreme scenario”
— that’s a self-selected visitor survey, not a model output, and
conflating the two would be its own kind of sloppy reading.)

What the page does give you, concretely, if you take it as a planning
input rather than a prophecy: in the Substantial scenario, wages for
knowledge workers are essentially flat; in the Extreme scenario, they
fall by more than 10% by 2030. Labor’s share of the economy runs 59.4%
in Modest, 56.1% in Substantial (capital gaining 3.9 points), down to
45.2% in Extreme (capital gaining 14.8 points). Those are directional
numbers about where value accrues if automation intensity keeps climbing
— genuinely useful for thinking about pricing, hiring mix, and where
margin comes from three years out.

None of that requires believing the Extreme scenario is likely, or
even plausible. It requires believing automation intensity is a spectrum
you’re somewhere on, and that “somewhere on it” is worth planning
around. That’s a different claim than “AI could kill all humans, >10%
within the decade” — and the page’s own writers keep those two claims
apart, even in the same week Hubinger didn’t.

## Why I don’t buy the number

I don’t have a p(doom) of my own, and I’m not going to manufacture
one for this post. What I have is a problem with the specific number
that got attached to the risk this week, and it’s a structural problem,
not a disagreement about the underlying danger.

“>10% within the next decade” is unfalsifiable on any timeline
that matters to a reader. Nobody collects on that bet in 2036 in a way
that updates anyone’s prior — there’s no observation between now and
then that confirms or denies it, only the outcome itself, which by
construction happens at most once. And it’s self-serving in both
directions at once: if you’re building the thing you’re warning about, a
stated double-digit chance of catastrophe reads as candor and caution to
one audience, and as proof the thing you’re building is important enough
to bet a decade of civilization on to another. The number does work for
the speaker whether the reader hears it as an alarm or as a pitch. [I’ve made a version
of this argument before](https://sublimecoding.com/blog/jose-valim-anthropic-incentive-problem), about Anthropic specifically not having
separated its security case from its commercial interest — this is the
week the two cases got published side by side, on the record, by the
company itself, rather than inferred from the outside.

The number does work for the speaker whether the reader hears it as
an alarm or as a pitch.

None of this is a claim that the underlying risk is zero, or that
alignment is solved, or that Coxon or Hubinger are wrong to be worried.
It’s narrower than that: a specific percentage, offered without a
falsification path, attached to a decade-out event, from someone whose
employer’s valuation benefits from the stakes reading as maximal — that
number is not evidence I’d put weight on either way, and I’d say the
same about a competitor’s engineer publishing the inverse claim with
equal confidence.

## What changes in a client’s security posture this week?

Nothing. That’s the honest, slightly unsatisfying answer, and it’s
worth saying plainly rather than dressing it up.

I do [fractional CTO and vCISO work for
pre-Series-A AI startups](https://sublimecoding.com/consulting), and the controls I’d tell a founder to
have in place this week — model access scoping, data retention limits,
who can push to prod, what gets logged, what a vendor’s own security
posture looks like before you build on top of it — don’t move because a
researcher resigned or because an alignment lead put a number on a
decade-out risk in a reply thread. Those controls were never priced off
p(doom). They were priced off the actual attack surface a seed-stage
company has right now: too much standing access, no incident response
plan, vendor dependencies nobody’s actually read the terms on. That
surface doesn’t get bigger or smaller because Anthropic published two
documents on the same day.

Where the econ page earns a place in a founder’s actual planning is
the labor-share and wage numbers, used as a spectrum rather than a
prophecy: if automation intensity in your market keeps climbing, what
does that do to your hiring mix, your pricing, your margin structure
over the next three years? That’s a real question with a real range of
answers, and it’s the kind of question I’d rather spend a planning
session on than a decade-out casualty estimate nobody can falsify. If
you’re modeling team shape rather than headcount, [I’ve argued the effect runs the
other way: AI surfaces the backlog and teams staff up to absorb it](https://sublimecoding.com/blog/ai-wont-shrink-your-team).
That’s the tension worth sitting with. The econ page’s Extreme case
moves 14.8 points of income from labor to capital, and a growing team is
not what that looks like.

## The future is not predetermined

That’s the econ page’s own line, not mine: “The future is not
predetermined. Ultimately, what the economy looks like in 2030 depends
on many factors, like what AI can do, and how companies and workers
choose to adopt it.” Read it the way Anthropic’s economists apparently
meant it, as a hedge on their own model, and it’s a fair one; three
scenarios, no probabilities attached, a disclaimer about stark
simplification. Read it the way an operator should read it, and it’s not
a hedge at all. It’s the whole argument. The two documents this company
published this week don’t converge on a forecast. They converge on the
fact that the outcome is still being decided by what gets built and how
it gets adopted — which is the one part of this story a founder actually
has a hand in, this week and every week after it.

What that means for your own stack and controls is a narrower
question than what it means for the industry, and it’s the one actually
worth answering this week. If you want it answered against your actual
stack rather than the news cycle, [that’s the work
I do](https://sublimecoding.com/consulting).

---

## I Put Omarchy on an M1 MacBook Pro

URL: https://sublimecoding.com/blog/omarchy-asahi-m1-macbook
Published: 2026-09-07
Tags: developer workflows, productivity, engineering

>
**TL;DR:** I’m a few days into running [Omarchy](https://omarchy.org/) — a Hyprland-based, opinionated
Linux setup — on an M1 MacBook Pro, dual-booting macOS. There’s a
purpose-built path for this: [omarchy-mac](https://github.com/omacom/omarchy-mac) puts
Omarchy on Apple Silicon via Asahi Alarm in about one command.
Installing it is the easy part, and it’s documented well. What isn’t
documented is the layer underneath — where the install still breaks, and
what daily life costs once macOS isn’t there anymore. The surprise:
thirteen coding agents come pre-wired, the OS hands segfaults to
whichever one you’ve set as default, and configuring the machine by
asking an agent is the designed path rather than a workaround. I don’t
know yet if I’m keeping this. Field report, not a verdict.

## Why this exists

macOS is still on the disk right now, dual-booting. A few days ago I
put [Omarchy](https://omarchy.org/) on this machine using [omarchy-mac](https://github.com/omacom/omarchy-mac), which
pairs Asahi Alarm with Omarchy and does most of the work in a single
command.

The install is the well-trodden part. omarchy-mac has been around
since mid-2025, it’s actively maintained, and the README walks you
through it properly. What I couldn’t find much of was the next layer
down: what still breaks on the way through, whether a working dev setup
actually survives the move, and what you give up day to day once macOS
isn’t underneath anymore. That’s the gap this post is trying to fill,
and it’s why I’m publishing now instead of waiting three months to have
a settled opinion.

I want to be upfront about what this post is not. It’s not a
recommendation. It’s not a “why I switched” essay — I haven’t switched.
macOS is still on the disk. It’s a status report from someone a few days
into an experiment who doesn’t know how it ends.

## The install path

Two steps. First the Asahi Alarm bootstrap, run from the macOS
terminal — it partitions the disk and puts Arch Linux ARM on the machine
alongside macOS. Then, booted into that fresh Arch, the omarchy-mac
setup command, which installs Omarchy itself.

[Asahi](https://asahilinux.org/) is the project that ports
Linux to Apple Silicon in the first place; Asahi Alarm is its Arch Linux
ARM flavor. omarchy-mac is the layer that takes that base and makes it
an Omarchy machine. None of this is exotic territory anymore.

It still broke.

## Where it broke: aquamarine

Omarchy’s installer failed partway through, trying to install
`aquamarine` — the rendering backend library that [Hyprland](https://hypr.land/) depends on (maintained by the
Hyprland team, source at [hyprwm/aquamarine](https://github.com/hyprwm/aquamarine)). The
version the installer tried to pull was too new, and it wouldn’t
install. I didn’t save the error.

The fix was to downgrade to an older `aquamarine` build
out of the local pacman cache rather than take whatever the installer
wanted to pull fresh. That got the install moving again. Worth being
clear about what this wasn’t: not an architecture problem. omarchy-mac
is built for M1/M2 specifically. This is the ordinary rolling-release
failure mode — a dependency moves faster than the thing consuming it,
and you land on the wrong side of a version for a few days. Arch does
this. It’s the price of current packages, and it’s as true on x86 as it
is here.

## What’s actually good

Once it was up, the parts that are working are working well:

**Hyprland’s tiling and the keyboard-driven flow.**
Window placement, workspace switching, everything stays on the keyboard,
and that’s the thing I notice most hour to hour. Most of the bindings
survive the MacBook keyboard unchanged; the ones that don’t [get their own page](https://sublimecoding.com/blog/omarchy-mac-keyboard-shortcuts).

**Speed.** This is the same M1 hardware I was running
macOS on, and it feels fast. Not a benchmark claim — just how it feels
day to day on identical silicon.

**Opinionated defaults.** Omarchy makes a lot of choices
for you — theming, keybindings, the tool selection — and I haven’t had
to configure much of anything. Someone already decided, and mostly I
agree with the decisions.

## The agent stack isn’t ported. It’s built in.

I’d budgeted a day for getting my agent setup working again. I didn’t
need it, and the reason turned out to be the most interesting thing I’ve
found so far.

Omarchy’s own manual opens its AI chapter with this: *“Omarchy
treats AI coding agents as first-class citizens, but it doesn’t pick a
favorite for you.”* That is not marketing copy bolted onto a Linux
distribution. It describes how the system is wired.

Thirteen agent CLIs come [pre-wired
as lazy-loaded launchers](https://github.com/omacom/omarchy/blob/quattro/manual/17-ai.md) — `claude`, `codex`,
`opencode`, `grok`, `copilot`,
`crush`, `pi`, `omp`, `agy`,
`ori`, `hermes`, `muse`,
`cursor-agent`. They’re mise-managed stubs in
`~/.local/bin/`, so nothing actually downloads until the
first time you run one. You pick a default with
`omarchy default agent <name>`, and a fresh install
prompts you to choose. `Super + Shift + Ctrl + A` opens it in
a dedicated terminal.
`omarchy agent prompt "Review this project"` launches it
straight into a task — unattended, in its don’t-stop-to-ask mode.

My Elixir/Erlang toolchain came across without special handling too,
but that part I expected. Toolchains are portable. The rest of this
isn’t a toolchain, it’s a set of assumptions.

Two features make the assumption concrete. The top bar grows an
**agents panel** the first time it sees AI coding usage on
the machine — your plan, how much of the five-hour and weekly limits
you’ve burned, tokens by day and by model. Usage tracking for coding
agents, in the OS status bar, next to the clock.

The second is better. Omarchy watches systemd-coredump, and when
something segfaults you get a “Process crashed” notification. Click it
and **the crash is handed to your default agent** along
with a diagnose-crash skill that walks it through reading the core dump
and deciding whether the thing is worth reporting upstream. The
operating system’s answer to a segfault is to hand it to your coding
agent.

There’s also a skill for configuring the machine itself — Hyprland,
the bar, themes — symlinked into `~/.claude/skills`,
`~/.codex/skills`, `~/.pi/agent/skills` and the
generic `~/.agents/skills`, so most harnesses find it without
being told. In practice this has been the fastest way to fix things on
this machine, including things unrelated to writing code. Worth
repeating the project’s own warning though: it labels that skill
experimental, suggests running plan mode first, and points at
`omarchy reinstall configs` for when the agent makes a mess.
That matches my experience — it’s very good and you should still watch
it.

Most of what I do is a terminal, a browser, and an [agent loop](https://sublimecoding.com/blog/claude-code-plugin-stack). I assumed that
stack would be indifferent to the OS underneath it. On this one it isn’t
indifferent — agentic work is the assumption the desktop is designed
around, which is a different thing from an OS that merely stays out of
your way.

## Has the Apple Silicon hardware actually held up?

So far, yes, in the narrow sense that nothing has broken — but I want
to be precise about what that does and doesn’t mean. I haven’t hit any
hardware gaps in the time I’ve been on this. I’m not claiming external
display, sleep, or the webcam work — I haven’t specifically tested or
stressed those. I’m also not claiming any of them are broken. Nothing
has bitten yet, and it’s only been a few days.

That distinction matters more than it sounds like it should. A few
days of normal terminal-and-browser use doesn’t exercise the parts of a
laptop that tend to be the hard problems on a reverse-engineered
hardware port — sleep/wake cycling, external display timing, camera and
audio pipelines. Asahi’s own project documents an ongoing, per-chip [hardware
feature support matrix](https://asahilinux.org/docs/platform/feature-support/overview/) for exactly this reason; I haven’t gone
looking for where my specific gaps are yet, because I haven’t needed to.
Take my “nothing’s broken” as “no news,” not as a clean bill of
health.

## The container caveat

This is the sharpest technical finding so far, and it’s worth real
space because it’s the one that actually shapes what I’d tell someone
else considering this.

Container images build fine locally on this arm64 machine. They don’t
work on GCP, because of the architecture mismatch — arm64 built locally,
x86_64 expected on the other end.

`buildx` plus QEMU does get you cross-arch builds that
work. But those cross-arch builds run 3-5x slower than the same builds
on macOS — and those builds were already fairly slow to begin with.
Getting QEMU installed and working in the first place was itself a chore
— not as straightforward as you’d hope.

If your workflow involves building containers locally that need to
run on GCP’s standard x86_64 infrastructure, I don’t think this current
setup serves you well. You’d be better off on a Dell XPS or some other
native bare-metal x86 install where the build target matches the build
machine and you’re not paying a QEMU emulation tax on every image. Worth
noting that “just buy another machine” is a [more expensive answer than it was a
year ago](https://sublimecoding.com/blog/ai-tax-hardware-costs).

## What the walled garden actually costs

It’s hard to leave the Mac walled garden. Omarchy is pulling me
toward leaving it, but that pull isn’t free — there’s real, daily
friction from missing creature comforts I didn’t think about until they
were gone.

iMessage has no usable path on Linux. iCloud Drive, Photos, and
Keychain sync don’t follow you off macOS — they just stop. I use both
Proton Mail and Proton Drive, and neither has an Arch Linux install,
which forces me back onto their web clients — not the ideal experience
for either. Backblaze is the same story: I haven’t found a Linux
equivalent that fits how I was using it on macOS.

None of this is a dealbreaker on its own. It’s a daily tax — small
friction, repeated every day, that adds up in a way a single dramatic
failure wouldn’t. A crashed install is a bad afternoon and then it’s
over. A missing sync feature is a bad moment every single day,
indefinitely, and those compound differently than a one-time failure
does.

## Where it stands

I’m still dual-booting. macOS is still on the disk, and I’m not
deciding anything yet. The tiling, the speed, the opinionated defaults,
and the fact that my actual dev and agent workflow just works are real
points in Omarchy’s favor. The container-to-GCP story and the daily
walled-garden tax are real points against it, at least for how I
currently work.

What would actually decide this for me: whether the container
workflow gets less painful, and whether the macOS creature comforts I’m
missing stop mattering as much as they do right now. I don’t have a
timeline on either of those, and I’m not going to pretend I do. For now,
this is where it stands — an open experiment, not a verdict.

If you’re weighing a build-target decision like this one — where the
machine, the toolchain, and the deploy target all have to agree — that’s
a conversation I have with clients regularly. [Reach out](https://sublimecoding.com/consulting) if you want to think it through
together.

---

## The Agent Evidence Pack for a Security Review

URL: https://sublimecoding.com/blog/agent-evidence-pack-security-review
Published: 2026-08-25
Tags: security, agents, founders

>
**TL;DR:** When an enterprise reviewer asks about your
agent, they want two artifacts, not a conversation: a tool allow-list
that shows what it can reach and who approved it, and a tool-call log
that shows what it actually did. Most teams have a paragraph describing
both and neither one as a real document. This is a field-by-field spec
for both, plus the one thing that makes them survivable past week one —
generating them from the code that already registers your tools, instead
of hand-maintaining a spreadsheet that’s wrong by the second sprint.

## What two documents actually satisfy an agent security review?

An allow-list that names every tool your agent can call, its
credential scope, and who approved it — plus a tool-call log that
records every invocation with enough detail to reconstruct what happened
without you in the room. Everything else a reviewer asks — autonomy
limits, data handling, incident response — gets answered faster once
these two exist, because most of those questions are really “show me the
allow-list” or “show me the log” phrased differently. I’ve written
separately about the questionnaire itself, the specific lines InfoSec
teams ask and what a credible early-stage answer sounds like, in [why auto-filled
security questionnaires kill AI deals](https://sublimecoding.com/blog/enterprise-security-reviews-ai-startups) — that post covers the
conversation. This one covers the two documents you hand over when the
conversation is done.

## The tool allow-list: what actually goes in each row

A tool allow-list is a table with one row per tool: what that tool
can reach, under which credential, at what risk class, whether it needs
confirmation, and — the column most teams skip — who approved it and
when. What it isn’t is a sentence on your security page saying “we
restrict what our agent can access.” Below is the field set and why each
one is there.

Field
Why it’s on the row
What it proves

Tool name
Uniquely identifies the capability, not the code that implements
it
The list maps to what the agent can *do*, not to file
names

What it can reach
Named system, table, or endpoint — not “internal APIs”
The blast radius is bounded and specific

Credential and scope
The actual token or service account, and its permission set
The tool can’t do more than the row claims

Risk class
Read-only, reversible-write, irreversible-write, or spend
Reviewers can triage by consequence, not by tool count

Confirmation required
Yes/no — does a human gate this before it runs
Autonomy limits are a lookup, not a claim

Approved by, and when
A name and a date, not a team or a Slack thread
This is evidence a decision was made, not documentation of a
default

Last reviewed
Date of the most recent re-check against current usage
The list reflects the agent as it runs today, not as it shipped

That “approved by, and when” column is the whole difference between
evidence and documentation. A list that says what the agent can do is a
description of your intentions. A list that says a named person signed
off on this exact scope on this exact date is a record a reviewer can
act on — they can call that person, or at minimum they know someone was
accountable for the decision instead of it accreting by default because
a tool was easy to wire up.

A list that describes what the agent can do is documentation. A list
that names who approved it and when is evidence.

## The tool-call log: the field set for one record

A tool-call log record needs enough fields that someone who wasn’t in
the room can reconstruct exactly what the agent did, to whom, and
whether a human signed off — not a debug log line that happens to
mention a tool name. Here’s the field set for a single record and why
each field earns its place:

- **Timestamp** — when the call happened, not when the
batch job that emitted the log ran.

- **Actor and actor type** — which agent, session, or
automated job triggered the call, and whether the actor itself was a
human-initiated request or a scheduled process. This is the field that
answers “was a human even in this loop at all” without you having to
ask.

- **Tenant** — which customer’s context the call ran
under. In a multi-tenant product this is the single field that turns an
incident investigation from a scan of everything into a filter.

- **Tool invoked** — the exact tool name, matching the
allow-list row it came from. If the log and the allow-list use different
names for the same tool, neither one is trustworthy.

- **Arguments** — the actual parameters passed, not a
summary. This is what lets you answer “did it write to the record it was
supposed to, or the one next to it.”

- **Outcome** — success, failure, or partial, plus the
return value or error. Without this you know the agent tried something;
you don’t know what happened.

- **Human confirmation** — whether a human approved this
specific call, and who, if the tool’s allow-list row requires
confirmation. This field is what makes the allow-list’s “confirmation
required” column checkable against reality instead of aspirational.

- **Provider request ID** — where the call passed through
a third-party LLM or API provider, their own request identifier, so you
can correlate your log with theirs during an incident instead of
guessing which of their calls was yours.

“The agent called the refund tool at 3:14pm” is a log line. A
reviewer can’t do anything with it — they can’t tell whether that was a
$12 refund a human approved or a $12,000 refund nobody looked at. The
field set is the difference. It is also what an incident actually
consumes — [the postmortem of an
agent failure](https://sublimecoding.com/blog/ai-agent-postmortem) turns on precisely this: the permissions delta between
what the agent had and what it needed, and the gate that didn’t
fire.

That last field has a second life on the storage side. The same
record, written to a table an auditor samples, is a different problem —
schema, retention clocks, whether you keep the raw prompt at all — and I
take that up separately next week. This post stops at the field set,
because a reviewer in a security review is asking what you can show
them, not what your migration looks like.

## How do you produce both without buying a governance platform?

You generate the allow-list from the code that registers your tools,
and you get the log for free from the instrumentation point most agent
frameworks already give you — a dedicated platform buys you a UI on top
of data you should already be capturing, not the data itself. If your
agent framework has a central place where tools get registered — a list
of function definitions, an MCP server config, a router that dispatches
tool calls — that registration is your allow-list’s source of truth. [Building an MCP server in
Phoenix](https://sublimecoding.com/blog/build-mcp-server-phoenix-hermes) is the worked version of this: the tools you deliberately
expose there *are* the allow-list, in code. A short script that
walks it and emits the table above, including pulling the credential
scope from whatever secrets manager backs each tool, keeps the document
honest because it can’t say anything the code doesn’t actually do.

The log is usually less work than teams expect, because the dispatch
point that calls a tool is a single chokepoint in the code — every tool
call already passes through it. Adding structured logging at that one
function, with the field set above, gets you every record without
touching the individual tools. If you’re already emitting application
logs, the fix is making tool calls a distinct, queryable event type
inside them rather than indistinguishable from every other log line —
retained for a defined window, filterable by tenant and by request ID,
so pulling the trail for one incident is a query, not a grep
session.

## The drift problem is the real reason evidence packs fail

A hand-maintained allow-list is wrong the first sprint nobody
remembers to update it, because someone adds a tool and updates the
spreadsheet never — the fix isn’t better discipline, it’s making the
document impossible to drift from the code. This is the actual failure
mode I’d flag first if you handed me your current security artifacts:
not that the allow-list is missing, but that it was accurate the day
someone wrote it and hasn’t been checked since. A reviewer who asks
“when was this last reviewed” and gets a date from three tool-additions
ago has found the same gap a hand-maintained document always produces —
the list describes an agent that no longer exists.

Generating the list from source closes that gap structurally instead
of procedurally. It can’t lag the code, because it *is* the code,
read and rendered. The log has the same property for free — it can’t be
stale, because it’s a record of what already happened, not a description
someone has to remember to update. The only manual step left is the
approval column, and that’s exactly the column that should stay manual:
a person deciding to grant a tool a scope is a real decision, and
automating it away would remove the one field that makes the whole
document evidence instead of a config dump.

## Where this fits with what you should have already built

An evidence pack documents a control layer; it doesn’t substitute for
building one. I’ve made the case for the layer itself in [running
security at an AI-native company](https://sublimecoding.com/blog/running-security-at-an-ai-native-company-2026): per-action authorization,
capability-scoped credentials, action-level audit logging, rate-limiting
as a circuit breaker, and confirmation prompts on risky actions. If you
haven’t built those yet, start there. The allow-list and the log are how
you prove you did.

The threat model underneath them is [OWASP’s
LLM06:2025, Excessive Agency](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/), which names three root causes:
excessive functionality, excessive permissions, and excessive autonomy.
Per-action authorization and capability scoping are the direct answers
to the middle two.

## Build it before the request lands

The teams that hand over a clean allow-list and a queryable log in
the same call they got asked for them didn’t write either document that
week — they built the agent so both fell out of the architecture, the
allow-list from the tool registry and the log from the dispatch point
every call already passes through. That’s the same posture a reviewer is
actually grading in every other part of a security review: not whether
you have zero gaps, but whether what you show them is real. If you’re
heading into a review and want someone to read your agent the way the
buyer’s security team will before they do, [that’s
the kind of engagement I run](https://sublimecoding.com/consulting).

---

## Most Security Controls Assume a Team You Don't Have

URL: https://sublimecoding.com/blog/security-controls-assume-a-team
Published: 2026-08-18
Tags: security, AI startups, founders

>
**TL;DR:** Security checklists are written either by
vendors selling tooling or by people whose job is to avoid blame, and
neither author is ever penalized for recommending a control a
five-person startup can’t staff. A control you can’t run is worse than
no control, because it teaches the team that security artifacts are
theater — a lesson that generalizes to the controls that actually
matter. This is the pre-hoc companion to [what I check when I inherit a
vibe-coded app](https://sublimecoding.com/blog/inheriting-a-vibe-coded-app) (post-hoc: what’s already broken) and [what enterprise
reviewers ask you](https://sublimecoding.com/blog/enterprise-security-reviews-ai-startups) (buyer-driven: reacting to someone else’s
checklist). This one is proactive, regardless of whether anyone’s asking
yet — which four controls to defer and at what headcount they stop being
theater, and the three-part test for telling a real control from a paper
one.

## The asymmetry nobody names

Open any “AI security checklist for startups” published in the last
year and count the items. They only ever go up. Nobody publishes a
checklist that says “drop these five, they’re not worth it at your size”
— because the two kinds of people who write these documents have no
incentive to say that.

The first kind sells tooling. A checklist item that maps to a product
category is marketing with a bow on it; a checklist item that says “skip
this until you have 30 engineers” sells nothing. The second kind is
avoiding blame — a security lead, a consultant, an auditor writing for
an audience of one imagined future incident where someone asks “was this
on the list.” Adding an item costs the author nothing. Leaving one off
costs them everything if it’s ever the item that mattered. So the list
only grows, checklist over checklist, because the asymmetry is
structural, not personal — nobody in that chain is rewarded for saying
“no.”

Aikido’s [“State
of AI in Security & Development 2026”](https://www.aikido.dev/reports/2026-state-of-ai-in-security-development) is a fair example, and I
want to be clear it’s a genuinely good document — the underlying survey
(450 CISOs, AppSec engineers, and developers across the US and EU) found
69% had identified a security vulnerability introduced by AI-generated
code, and 20% had suffered a serious incident linked to it. Those are
real, useful numbers. But the checklist built around them assumes an
AppSec function, a platform team, and a training function to hand items
to. [Fencer’s
8-item founder checklist](https://www.fencer.dev/blog/how-to-secure-your-vibe-coded-app) makes the same assumption in smaller print,
written for a founder who has — or is assumed to have — someone to hand
each item to.

None of this is an argument against the underlying advice. It’s an
argument that the advice was never priced against your staffing, because
the people writing it don’t pay the bill when you follow it and nobody’s
there to run it.

## What a paper control actually costs

The instinct is to read “adding a control I can’t staff is worse than
not having it” as hyperbole. It isn’t, and the mechanism is
specific.

Attention is the scarce resource at a five-to-fifteen-person company,
not risk tolerance. Every control you adopt is a claim on someone’s
attention forever — not once, at setup, but every week it exists,
because a control nobody’s watching drifts out of date, and a stale
artifact is worse than an honest absence. A prompt-provenance log nobody
reads is a line item in a runbook that says “check the log” during an
incident, written by someone who’s about to discover the log stopped
being written to three months ago. Nobody caught it, because nobody’s
job was to catch it.

That’s the mechanism I’ve written about from the trust side in [psychological safety
as a security control](https://sublimecoding.com/blog/psychological-safety-security-control) — a control’s value depends entirely on
whether the humans around it treat it as real, and a team learns what’s
real by watching what gets enforced versus what gets checked off and
ignored. A dead control doesn’t just fail to help; it actively trains
the team that the *category* — security paperwork — is
decorative. That lesson doesn’t stay contained to the one dead control.
It generalizes to the rotation policy that does matter, the incident
review that does matter, because the team has already learned that this
kind of document doesn’t get read.

A control that exists only on paper doesn’t just fail silently — it
teaches the team that security artifacts are theater, and that lesson
generalizes to the controls that do matter.

## The four to defer — and where each one turns real

Deferring isn’t skipping. Each of these becomes worth adopting at a
specific point, and the point is almost always “a specific person’s job
now includes this,” not “we bought the tool.” The control most often deferred past its point is visibility into what your team already signed up for — [shadow AI, at a startup that cannot buy a CASB](https://sublimecoding.com/blog/shadow-ai-startup-casb).

**Prompt-provenance inventories.** Logging which model,
which prompt version, and which agent produced a given piece of code or
a given customer-facing output is good practice — once somebody’s job
includes reading it back during an incident. Below that, it’s a table
nobody queries, growing forever, that gives false comfort in a
postmortem where the actual question — “what changed and who approved
it” — still gets answered by grepping git history because that’s the log
that was actually maintained. The threshold: adopt it when you’ve named
the person who opens it when something goes wrong, not before.

**Separate workforce and customer production
environments.** Splitting the environment your team touches from
the one customers’ data lives in is a real control against a real
failure mode — an internal tool with broad access becoming the path into
production data. It also requires a platform team to build and run two
environments instead of one, with the deploy pipelines, access
boundaries, and on-call coverage that implies. At five engineers, that’s
a second infrastructure project competing for the same three people who
are also shipping the product. The threshold: when you have someone
whose job is platform, not “whoever’s free.”

**A formal secure-prompting training program.** Teaching
engineers what not to paste into a prompt, how to handle a customer’s
data in an AI-assisted workflow, what “never trust frontend input”
actually means in practice — all real. Below roughly 30 people, it’s not
a curriculum, it’s a conversation, and pretending otherwise means
someone builds slides nobody revisits while the actual transmission
mechanism — a senior engineer correcting a junior one in a PR comment —
keeps doing the real work uncredited. The threshold: when onboarding
stops being one person walking the next hire through it in an
afternoon.

**An internal catalog of pre-vetted security building
blocks.** A library of approved auth patterns, sanctioned secrets
handling, blessed API client configs that engineers pull from instead of
reinventing — genuinely valuable, and genuinely a platform-team output.
Below that, “the catalog” is one file that goes stale the week after
someone writes it, because no one owns keeping it current against what
the codebase actually does now. The threshold: when a platform team
exists to own artifacts as a job, not a project.

None of these four are wrong ideas. They’re right ideas with an
unstated staffing precondition, and the checklist that recommends them
never states it because the person writing the checklist doesn’t have to
run it.

## Do assurance levels matter under 20 engineers?

Mostly no — the five-level maturity ladders common in this space
collapse under 20 engineers to one real question: **does this code
path touch auth, money, or customer data?** If yes, it gets real
review regardless of what tier the framework says you’re supposed to be
at. If no, the five-level model is measuring a distinction that doesn’t
exist yet at your size — you don’t have five people to staff five
maturity levels, you have one review discipline, applied selectively to
the paths where being wrong is expensive.

This isn’t an argument against assurance frameworks in general —
they’re built for organizations large enough to have separate teams
operating at separate maturity levels simultaneously, which is exactly
the condition a pre-Series-A company doesn’t have. Importing the ladder
without the org chart that makes it meaningful just produces a scorecard
nobody’s accountable for filling in accurately.

## The short list that isn’t optional

A handful of controls don’t get the “defer until you’re bigger”
treatment, because the failure mode they prevent doesn’t wait for you to
staff up — Base44’s authentication bypass, [found by
Wiz Research’s Gal Nagli](https://www.wiz.io/blog/critical-vulnerability-base44) and disclosed in July 2025, patched within
roughly a day with no evidence of exploitation, existed because a
guessable application ID was the only thing standing between an
unauthenticated request and a full SSO bypass into private enterprise
apps. That’s a fundamentals problem, and fundamentals don’t scale down.
If you’re leaning on a frontend check as your actual security boundary,
read what Lovable’s own docs say about it: the frontend “runs in the
user’s browser, is always public, and must never be trusted.” I’m not
going to re-argue the specific line items here — [secrets management for
agent-driven workflows](https://sublimecoding.com/blog/ai-agent-secrets-management) covers the credential-handling piece, and [enterprise security
reviews for AI startups](https://sublimecoding.com/blog/enterprise-security-reviews-ai-startups) covers what a serious buyer is actually
going to check. Both stay non-negotiable at any headcount. If you’re
finding gaps like these after the fact instead of heading them off, [what I check when I inherit a
vibe-coded app](https://sublimecoding.com/blog/inheriting-a-vibe-coded-app) covers the secrets-and-git-hygiene audit for a system
already in that state.

If you’re trying to work out which of the two lists — defer, or
non-negotiable — a specific control you’re staring at belongs on, [the triage tool](https://sublimecoding.com/tools/security-controls) runs the same
reasoning against your actual headcount and stack instead of a generic
checklist’s assumptions.

## The test

You don’t need a framework to tell a real control from a paper one.
You need three names, and if any of the three is blank, the control is
paper:

>
A control is real if you can name three things: the person who runs
it, the artifact it produces, and what happens when it fails. If any of
the three is blank, it is paper.

“Everyone’s responsible for security” fails the first name — everyone
means no one, and the control has no owner the day it actually matters.
A dashboard nobody’s job includes checking fails the second — an
artifact that exists but that no process consumes isn’t a control, it’s
a screensaver. A rotation policy with no defined response when a key
doesn’t get rotated on schedule fails the third — a control with no
failure mode is enforced in name only.

Run every item on whatever checklist you’re currently working through
against those three questions before you adopt it. The ones that pass
are worth the attention they’ll cost. The ones that don’t are worth
naming out loud as deferred, with the headcount or hire that flips them
real — which is a more honest artifact than a document that quietly went
stale six months ago.

## Where this fits with hiring the person who’d actually run these

Most of what turns a paper control real is a person whose job
description includes it, which is the same tradeoff I’ve written about
in [security
engineer or vCISO as the first security hire](https://sublimecoding.com/blog/security-engineer-or-vciso-first-hire) — the sequencing
question of when a fractional or full-time security role stops being
premature and starts being the thing that converts your deferred list
into a real one. And the deeper version of “who actually enforces a
control” is the trust question, not the tooling question — covered in [trust as the operating
system](https://sublimecoding.com/blog/trust-is-the-operating-system).

If you’re staring at a vendor checklist and trying to figure out
which four to defer and which are non-negotiable regardless of your
headcount, that’s exactly the conversation worth having before you adopt
any of it wholesale. [Let’s work out which controls
are real for where you actually are](https://sublimecoding.com/consulting).

---

## What Elixir Gives a Coding Harness for Free

URL: https://sublimecoding.com/blog/elixir-coding-harness
Published: 2026-08-15
Tags: Elixir, agents, AI tools

>
**TL;DR:** José Valim [posted](https://x.com/josevalim/status/2088186994849468659)
that people are sleeping on Elixir as the substrate for a coding harness
— the client you actually drive an AI agent through, not the backend
that runs a fleet of them. He’s right about the runtime: hot-code
reload, the actor model, and built-in distribution really do turn three
hard harness problems into defaults instead of designs. He’s quieter
about the tax: the sandbox boundary still has to be built by hand, the
plugin ecosystem lives in TypeScript, and the model SDKs land in Python
first. The runtime argument wins. The ecosystem argument is the bill
that comes due every week you maintain the thing.

## The claim, and why it’s not just BEAM boosterism again

I’ve made the general case before that [the BEAM is the runtime AI agents
want](https://sublimecoding.com/blog/elixir-beam-ai-agents) — that post was about running fleets of agents in production:
supervision trees restarting crashed workers, process isolation
containing failures, per-agent processes cheap enough to spawn thousands
of. Valim’s tweet is a narrower and more interesting claim. It’s not
about the thing agents run *on*; it’s about the tool a developer
runs *through* — the Claude Code / [OpenCode](https://opencode.ai) / [Pi](https://pi.dev/) class of coding harness that reads a repo,
edits files, and executes tools in a loop while a human watches.

[He named
three specific mechanics](https://x.com/josevalim/status/2088186994849468659), not a general vibe:

>
People are sleeping on Elixir for a coding harness:

- Hot-code swapping allows you to build an extensible plugin system
similar to Pi, which reloads live without dropping state

- Designing a client-server architecture, similar to OpenCode, is
basically a byproduct of the actor model (plus you get both IO/CPU
concurrency)

- The built-in distribution means you can easily isolate the brains
(model + session) from the hands (sandbox + tools). For example, you can
[run] the agentic session on your machine which coordinates agents
executing inside Docker or a remote node. Or even have one agent session
coordinate multiple nodes (this is basically how Livebook works
anyway)

Those can definitely be built from scratch in other languages, but in
Elixir the building blocks are basically part of the runtime.

Each one is a real, specific harness problem, and that closing line
is the actual claim. Not that Elixir *can* do these things — that
you don’t build them.

Valim isn’t a neutral narrator here, and it’s fair to say so without
treating it as a knock: he and Dashbit build [Tidewave](https://tidewave.ai), an Elixir/Phoenix-native
agent-tooling product, so he has real hands-on evidence for what the
runtime buys. The disclosure runs the other way too, so take it as the
preface to everything below: I ship this site on Phoenix, and I
recommend Elixir to clients because it’s the right default for most of
what gets built today. None of that is in tension with the costs I’m
about to name. A language worth recommending is one whose limits you can
state out loud.

## Hot-reload solves the one thing every other harness fakes

A coding harness’s state *is* the conversation: the message
history, the open-file context, the tool-call log the model is reasoning
over. Every plugin system built on Node or Python has to solve the same
problem when a plugin changes — reload the process and lose that state,
or bolt on serialization so the session survives the restart. Most don’t
bother; you restart the harness to pick up a new tool definition, and
the in-flight session is gone.

Elixir sidesteps this because code reloading was never a hack bolted
onto the language — it’s how the BEAM ships production upgrades without
dropping connections, a capability that predates the AI harness use case
by decades. A `GenServer` holding a coding session’s state
doesn’t need to die to pick up a new module version; the process stays
alive, the state stays in memory, and the code executing against it
changes underneath it. For a plugin system, that’s the difference
between “install a tool, keep working” and “install a tool, lose your
context.”

## Client/server and concurrency are what you get, not what you design

OpenCode’s shape — a headless server plus swappable clients (TUI,
editor extension, web UI) — is an architecture other stacks have to
deliberately design: pick a wire protocol, decide what state lives
server-side versus client-side, build the reconnection logic. In Elixir
it’s closer to the default shape of a `GenServer` plus
however many client processes talk to it. You’re not inventing a
client/server split so much as declining to collapse the one the actor
model hands you.

The concurrency half of the claim is worth separating from the
buzzword. A harness session is doing at least three things
simultaneously: streaming tokens from the model (IO-bound), watching the
filesystem for changes the agent should notice (IO-bound), and running
the build or test suite the agent just triggered (CPU-bound). The BEAM’s
scheduler handles both classes on the same set of lightweight processes
without the harness author reaching for a thread pool and a separate
async-IO runtime and gluing the two together, which is closer to normal
in Python and Node harnesses today.

## Can you actually isolate the brains from the hands?

Yes — and this is the part of Valim’s argument that’s least
hypothetical, because Livebook already ships it. Livebook’s [attached-node
runtime](https://hexdocs.pm/livebook/use_cases.html) lets a notebook process on your machine connect to and
evaluate code on a completely separate Elixir node — the same primitive
Valim is describing for a coding agent, just applied to REPL cells
instead of tool calls. A minimal version of “session here, execution
there” is genuinely a few lines:

`[](#cb1-1)Node.connect(:"sandbox@10.0.0.4")
[](#cb1-2)Node.spawn(:"sandbox@10.0.0.4", Tool, :run, [cmd])
```

The session process, model context, and conversation history stay on
your laptop; the Tool.run/1` call executes wherever
`sandbox@10.0.0.4` happens to be — a Docker container, a
remote node, a machine that isn’t yours at all. One session coordinating
several such nodes is the same primitive applied twice.

## Where’s the tax?

The tax is everywhere the runtime’s elegance stops and the ecosystem
around it starts.

**The distribution primitive gets you coordination, not a
sandbox.** Valim names Docker himself — “agents executing inside
Docker or a remote node” — and that passing mention is carrying the
entire weight of the cost. `Node.connect/1` and
`Node.spawn/3` move code execution to another machine; they
say nothing about what that code is allowed to do once it’s there. A
BEAM process is cheap and isolated for *failure* — a crash
doesn’t take down its siblings — but that isolation was never designed
against a hostile payload. A rogue tool call is a
`System.cmd`, a NIF, or an open port, and none of those
respect a process boundary. So the sandbox is still yours to build, out
of Docker or a microVM or OS-level controls, and it’s a different
discipline from the coordination layer sitting above it. For scale: even
the far narrower problem of keeping parallel agents from colliding on
ports, databases, and asset watchers took [a whole post of
Phoenix-specific plumbing](https://sublimecoding.com/blog/parallel-claude-code-agents-phoenix), and none of that plumbing was a security
boundary. Coordination is the easy 80%. The boundary is the 20% that
actually protects you.

**Plugin authors live in TypeScript, not Elixir.** A
hot-reloadable plugin system is only as valuable as the population
willing to write plugins for it, and the people writing coding-agent
extensions today write JS/TS almost universally — it’s the ecosystem
every major harness (VS Code, Claude Code, Cursor) already targets. An
Elixir harness either accepts a much smaller plugin pool, or embeds a JS
runtime to host third-party plugins — at which point you’ve given back a
chunk of the hot-reload story you built the system to get, because now
two runtimes need to agree on what “reload without dropping state”
means.

**Model-provider SDKs land in Python and TypeScript
first.** I’ve made the general version of this point — [the model layer was never the
BEAM’s](https://sublimecoding.com/blog/elixir-beam-ai-agents) — but the harness version is narrower and bites more often,
because it isn’t about local inference, it’s about protocol churn.
Structured-output formats, new tool-call shapes, streaming-protocol
changes — these ship against the two languages providers actually
maintain official SDKs for. An Elixir harness maintainer is frequently
doing the HTTP-level translation work a Python maintainer gets for free
from `pip install`. I’ve written about [building the agent loop
itself in about fifty lines of Elixir](https://sublimecoding.com/blog/build-ai-agent-loop-50-lines-elixir), and the loop genuinely is
that small — it’s the provider integration surface around the loop, not
the loop, that recurs as a maintenance cost every time a provider ships
a new feature.

## So who should actually take the trade?

Distribution moves the execution. It doesn’t decide what the
execution is allowed to do.

If you’re already running production Elixir — an Oban-backed job
runtime, a Phoenix app, [the stack you’d pick for
an AI startup backend anyway](https://sublimecoding.com/blog/why-elixir-for-ai-startup-backend) — building an *internal* coding
harness on the same stack is a genuinely good trade. You inherit
hot-reload, distribution, and the concurrency model for the coordination
layer, you’re not selling a plugin ecosystem to strangers, and you can
lean on patterns like the [durable Oban-based agent
runtime](https://sublimecoding.com/blog/oban-durable-ai-agent-runtime) for the parts that need to survive a restart. Your team
already pays the Elixir-hiring and Elixir-SDK tax for everything else
you ship; the harness doesn’t add a new cost, it just uses the one you
already carry.

If you’re building a general-purpose harness for the whole market —
competing with Claude Code, Cursor, or OpenCode for developers who’ve
never touched Elixir — the ecosystem tax compounds against you on every
axis that matters for adoption: who can write a plugin, who can read the
source and contribute, how fast you track provider API changes. The
runtime elegance doesn’t show up in the sales pitch; the plugin count
and the time-to-support-the-newest-model do. That’s not an argument
against Elixir — it’s an argument for knowing which of the two harnesses
you’re actually building before you pick the stack.

If you’re weighing that call for a team already on Elixir — internal
harness versus buying one, and where the sandbox boundary actually has
to sit — that’s the kind of decision I work through with founders as a
[fractional CTO](https://sublimecoding.com/consulting).

---

## The Office Mandate Is a Measurement Failure

URL: https://sublimecoding.com/blog/office-mandate-measurement-failure
Published: 2026-08-09
Tags: engineering leadership, agents, productivity, AI

>
**TL;DR:** Nobody mandates presence when they can see
output. The four-day mandate is a symptom of a measurement system that
died, and the proxies it was built on — commits, pull requests, hours
visible at a desk — were already bad before agents started producing
most of the volume. Now they’re worse than useless, because the person
doing the best work may be the one with the fewest keystrokes. The
replacement isn’t a better activity metric. It’s the goal: did we ship
what we set out to ship, and was the goal ambitious enough to be worth
hitting. And every honest case for being in the same room — onboarding a
junior, standing up a new project, responding to an incident — is a
scheduled event with a start and an end date. Fund those out of the rent
you stop paying.

## The mandate is a symptom

Here’s the thing that gives it away: no one issues a mandate about
the work.

They issue it about the location. Four days a week, badge in, be
seen. Then everyone drives in, sits down, puts on noise-cancelling
headphones, and joins the same video calls they would have joined from a
kitchen table. The mandate produced attendance. It did not produce a
single additional decision, shipped feature, or resolved incident.

That’s not an accident of implementation. It’s what the policy is
actually for. A company that can look at an engineer and say “here’s
what you delivered this quarter, here’s what it was worth” has no reason
to care where the chair was. A company that can’t say that has exactly
one signal left, and it’s a body in a building. The mandate isn’t a
collaboration strategy. It’s an admission, written in real estate.

I want to be clear about my standing here before I go further: I’ve
never issued a return-to-office mandate and I’ve never worked under one.
Every company I’ve worked for has been remote, and several of them were
very good at it. Some of those companies later failed or cut headcount —
and in none of those cases was distributed work the reason. The market
moved, or the model didn’t work, or the funding stopped. Nobody’s
post-mortem said “we should have been in a room.” That’s an observation
from where I’ve stood, not a study, and you should weigh it that
way.

What I can speak to directly is the measurement problem, because it’s
the thing I’ve had to solve on every engagement I run.

## Agents destroyed the proxies

The old proxies were never good. Lines of code, commit count, pull
requests merged, hours visible — every engineering leader who’s thought
about it for ten minutes knows these measure typing, not value. We kept
using them anyway, because they were cheap and they correlated with
effort badly but nonzero.

Agents broke the correlation.

Start with the volume problem. In six months of building with agents,
I put [4,154 commits
into a codebase that grew to about 1.5 million lines](https://sublimecoding.com/blog/4154-commits-six-months-ai-agents) — and I said in
that post, and I’ll repeat here, that the line count includes generated
code, vendor code, scaffolding, and configuration. It isn’t 1.5 million
lines of artisan craft. It’s *surface area*. That’s the honest
framing, and it’s exactly why the metric is now dangerous: the number
went up by an order of magnitude and my hands did not get faster. If
you’re ranking engineers by output volume in 2026, you’re ranking their
tooling.

Then there’s the shape of the day, which is stranger and matters
more. Running agents well is mostly waiting and steering. You set one
going, it works, it comes back with a question, you answer it, you check
whether it drifted, you send it back. Between those moments there’s real
downtime — enough that you can hold several in flight at once, and
enough that a lot of the supervision doesn’t need a desk at all. Boris
Cherny, who created Claude Code, [runs
multiple sessions in parallel across terminal worktrees and cloud
sessions](https://newsletter.pragmaticengineer.com/p/building-claude-code-with-boris-cherny) and starts a batch of them from his phone in the morning,
checking in through the day. He’s reported [shipping
dozens of pull requests a day this way and not hand-writing code at all
in 2026](https://www.aol.com/articles/claude-codes-creator-says-setup-091701000.html).

Whatever you think of that as a way to work, notice what it does to
the mandate’s premise. One of the most productive engineers in this
industry does a meaningful share of his highest-leverage work from a
phone, between other things. There is no desk-hours metric that captures
him. There’s no badge reader that would tell you he was working. A
supervisor watching the floor would conclude he’d checked out.

If your measurement system can’t distinguish the most productive
engineer in the building from someone who’s checked out, the system is
broken — and putting everyone in the building doesn’t fix it.

This is the part the office argument never survives. The mandate
assumes presence is evidence. Agents made presence and evidence fully
independent variables.

## What I actually measure

On distributed and embedded engagements, the thing I ship is
goals.

That’s the unit. Not velocity, not story points, not a dashboard of
activity. We decide what we’re setting out to accomplish in a defined
window, we say out loud what “done” looks like, and at the end we answer
two questions:

**Did we accomplish the thing we set out to do?**
Binary, or close to it. It shipped, or it shipped partially and here’s
what’s left, or it didn’t and here’s what we learned. This is
unambiguous in a way no activity metric ever is, and it’s the only
question a client has ever actually cared about.

**Was the goal ambitious enough?** This is the one
people skip, and it’s where the real signal lives. A team that hits
every goal on time might be executing beautifully or might be setting
targets it can clear without stretching. A team that misses might be
sandbagged by something real or might be reaching correctly and learning
fast. You cannot interpret the first question without the second. Two
quarters of clean hits with no near-misses is not a sign of health —
it’s a sign the goals are too easy, which is a management failure, not a
team failure.

Underneath that, the smaller numbers still have a use, but only in
aggregate and only as texture. Commit counts, client acceptance, escaped
defects, cycle time — in aggregate, across a team, over a quarter, those
paint a picture. What they don’t do is judge a person. A low number
doesn’t mean unproductive. A high number doesn’t mean productive. I’ve
seen the most valuable contribution in a month be a conversation that
killed a feature nobody should have built, which shows up in every
metric as a person who did nothing that month. [Proving the return on the work](https://sublimecoding.com/blog/prove-the-return) is a
different discipline than counting the activity, and it’s the one worth
building.

This is also why the honest version of estimating agent-heavy work
has gotten harder rather than easier — I wrote about [what happens to client
estimates when agents do the building](https://sublimecoding.com/blog/estimating-client-work-ai-agents), and the short version is that
the throughput went up while the predictability didn’t. Measure the
goal. The activity underneath it stopped meaning what it used to
mean.

There’s a related failure mode worth naming: if you can’t measure
output and you know it, [the performance review
becomes theater](https://sublimecoding.com/blog/performance-reviews-when-agents-type) — a ritual for producing a rating rather than a
mechanism for finding out what happened. The office mandate is that same
instinct, expressed as a lease.

## Presence is an event budget, not a lease

Now the part where I’ll argue against the strongest version of the
other side, because there is one and it deserves better than a
dismissal.

Some things genuinely go better in a room. I believe that. What I
don’t believe is that any of them require a building you pay for twelve
months a year.

**Onboarding juniors.** Real. Getting someone new up to
speed is faster with a whiteboard, a shared table, and the ability to
interrupt. It’s also *bounded*. That’s a week, maybe two, on
location, at the start. It is not a permanent seating arrangement, and
treating it as justification for one is a category error — you’re using
a two-week need to buy a two-year lease. This matters more now, not
less, because [onboarding
into an agent-heavy codebase](https://sublimecoding.com/blog/onboarding-agent-heavy-codebase) is a different job than onboarding used
to be, and the supervision that makes it work is about reading someone’s
reasoning, not watching them type.

**Incident response.** Partly real, and usually stated
wrong. What incident response needs is *response distance* —
someone who can be hands-on inside an acceptable window. That’s a
locality requirement, and for some systems it’s a legitimate one to
write into a job description. It is not a desk requirement. Sitting in
the office all day Tuesday does not make you faster on Thursday night’s
page. And notice the arithmetic nobody does: if the person is in the
office, they still have to get to the office. Thirty minutes,
forty-five, an hour, whatever the commute is — the mandate doesn’t
eliminate travel time, it just charges it to the employee every single
day instead of to the incident.

**Kickoff and planning.** The most real of the three.
The opening phase of a project, the architecture argument, the
whiteboard session where the shape of the thing gets decided — those are
better in person, and I’d fight for them. They’re also, again, events. A
few days. Occasionally a week.

Add those up and you get something like: a couple of onboarding weeks
per new hire, a kickoff per project, a few team gatherings a year.
That’s a travel budget and an offsite budget. Put a number on it, then
put it next to what an office costs — rent, power, water, internet,
furniture, cleaning, parking, the facilities person, the whole daily
carry, paid every month whether the room is used or not. The travel line
is a rounding error against the lease line.

And here’s the part that should bother anyone who actually cares
about culture: the travel version is *better at the thing the office
claims to do*. A week working through hard problems together
somewhere, with real meals and real evenings, produces more bonding than
a year of people sitting near each other with headphones on. If the goal
is a team that trusts each other, spend the money on that directly
instead of buying a building and hoping proximity generates it as a side
effect. [Trust is the
actual operating system](https://sublimecoding.com/blog/trust-is-the-operating-system), and you can’t lease it.

## What’s left

Strip out the real-estate sunk cost and the middle-management layer
whose primary function is confirming that people are at their desks, and
what remains of the four-day mandate is a company saying, in the most
expensive way available, that it doesn’t know what its people
produce.

That was an embarrassing thing to admit in 2019. In 2026 it’s a
strategic problem, because the gap between your best engineers and your
average ones is now mediated by how well they direct agents — and that
difference is completely invisible to attendance. You will promote the
wrong people. You already might be.

The fix is not a policy about buildings. It’s writing down what
you’re trying to accomplish, being honest about whether the target was
ambitious enough, and getting comfortable judging results you didn’t
watch happen. That’s harder than a badge reader. It’s also the only
thing that works when the typing is done by something that doesn’t
commute.

If you’re leading a team where agents are doing a growing share of
the building and your existing measurement system is quietly falling
apart underneath it, that’s the problem I help engineering leaders and
founders fix. [Let’s talk](https://sublimecoding.com/consulting).

---

## The Schlep Is the Only Moat Agents Can't Cross

URL: https://sublimecoding.com/blog/schlep-is-the-moat-agents-cant-cross
Published: 2026-08-05
Tags: AI, agents, founders, AI startups

>
**TL;DR:** In 2012 Paul Graham argued that founders
unconsciously avoid ideas made of tedious work, and that this is why so
few people built Stripe. Fourteen years later the avoidance costs more.
Coding agents collapsed the price of the part everyone was running
toward — the demo, the CRUD, the auth scaffold, the marketing site — to
something close to zero. That half of your product is now copyable by
anyone with a prompt and a weekend. The schlep half is not, because a
schlep isn’t hard code. It’s work that requires a counterparty, an
accountable human, and usually a signature: bank onboarding, a
customer’s security review, dirty-data migration, the support escalation
that ends in a refund. Graham’s question was “what problem do I wish
someone else would solve for me?” The 2026 version is narrower: which
half of my idea survives a competitor pointing an agent at it?

## What Graham noticed in 2012

[Schlep Blindness](https://paulgraham.com/schlep.html) is
a short essay from January 2012 with one idea in it. “Schlep was
originally a Yiddish word but has passed into general use in the US. It
means a tedious, unpleasant task.” Founders avoid ideas built out of
schleps, and — this is the part that makes the essay good — they don’t
know they’re doing it. “Your unconscious won’t even let you see ideas
that involve painful schleps.”

His example was payments. “For over a decade, every hacker who’d ever
had to process payments online knew how painful the experience was.
Thousands of people must have known about this problem. And yet when
they started startups, they decided to build recipe sites, or
aggregators for local events.”

The Collison brothers took the schlep instead. Banks, underwriting,
fraud, chargebacks, PCI scope, a different regulator in every country
they expanded into. Graham’s summary line is the one worth keeping: “A
company is defined by the schleps it will undertake.”

In 2012 that was a claim about opportunity cost. If you skipped the
schlep, you skipped a big business and built a small one. Fine — plenty
of people are happy running small ones.

## The cost of skipping changed, and only in one direction

Here’s what’s different now. Take any product idea and cut it in
half.

The first half is the part you can picture: the UI, the data model,
the CRUD endpoints, auth, billing wiring, a landing page, onboarding
email. In 2012 that half was most of the work. It was six months of two
people’s lives and it was a real barrier to a competitor.

That half is now a weekend. I don’t think this is controversial
anymore for anyone who has actually shipped with agents in the loop —
it’s the observation underneath [why estimating client work
changed](https://sublimecoding.com/blog/estimating-client-work-ai-agents). The typing stopped being the constraint.

The second half is the schlep: getting a bank to onboard you, passing
a Fortune 500 customer’s security review, migrating fifteen years of
someone’s malformed CSVs, staffing the phone number on the invoice,
being the name on the incident report.

Agents made the first half nearly free. They did approximately
nothing to the second half. So the ratio between them inverted, and the
strategic consequence is blunt: **the half you were unconsciously
running toward is now the half that provides no defense at
all.**

If your product is entirely first-half, you don’t have a company, you
have a head start measured in weeks. Somebody with a similar idea and a
similar agent budget arrives shortly. This is the actual mechanism
behind the “it’s just a wrapper” complaint, which is usually stated as
an insult when it’s really a structural observation. A wrapper is a
product made only of the cheap half.

## Why an agent can’t eat a schlep

The tempting counter is that agents will eventually eat the schlep
too. Some of it, yes — I’ll get to that. But most schleps aren’t hard
because the code is hard. They’re hard because of three properties that
have nothing to do with code generation.

**A schlep has a counterparty who moves at their own
speed.** A bank’s onboarding team, an auditor, a customer’s
procurement department, a state regulator. You cannot prompt your way to
the other side of that table. When a prospect’s security team sends you
a 200-question vendor questionnaire, the bottleneck was never writing
the answers — an agent drafts those fine. The bottleneck is that [the questionnaire
is a gate someone on the other side controls](https://sublimecoding.com/blog/enterprise-security-reviews-ai-startups), and clearing it takes
weeks of their calendar, not yours.

**A schlep requires an accountable human.** SOC 2
doesn’t ask whether the access review happened, it asks who ran it and
produces their signature on the evidence. An agent can generate the
policy document, and honestly it writes a better first draft than most
people do. It cannot be the party who attests. This is the whole reason
[SOC 2 works
as a revenue tool instead of a security artifact](https://sublimecoding.com/blog/soc-2-is-a-revenue-tool-not-a-security-tool) — the buyer is
purchasing someone to blame, and a model can’t be blamed.

**A schlep is where the mess is, and the mess is
undocumented.** Every migration I’ve watched go sideways went
sideways in the same place: not the transform logic, but the eleven
years of exceptions nobody wrote down. The customer who was manually
invoiced because of a 2019 dispute. The three states where the tax rule
is stored in a spreadsheet on someone’s laptop. An agent is excellent at
the transform and blind to the exceptions, because the exceptions aren’t
in the repo. They’re in a person’s head, and getting them out is a
series of uncomfortable meetings.

Put together: the schlep is the part of the business that runs on
trust, accountability, and other people’s institutions. That’s not a
temporary gap in model capability. It’s the layer where somebody has to
own the outcome, which is a [claim I’ve made about
professionals generally](https://sublimecoding.com/blog/professional-owns-the-outcome) and which gets more load-bearing the cheaper
generation gets.

Jason Fried made the demand-side version of this argument [back
in March](https://world.hey.com/jason/the-bespoke-software-revolution-i-m-not-buying-it-4bfad9ec): most people don’t want to build their own software no
matter how cheap building gets — “a powerful excavator doesn’t turn a
homeowner into a contractor.” That matches every client conversation
I’ve had. Nobody asks me to help them build a system. They ask me to
make a problem go away, and the system is an implementation detail
they’d rather never think about again. Cheap generation didn’t mint a
nation of builders. It lowered the price of hiring someone who already
likes owning the outcome.

## Where this argument is weakest

I’d rather say this myself than have you shout it at the screen:
agents genuinely have shrunk some schleps, and I don’t want to pretend
otherwise.

Integration glue used to be a schlep and mostly isn’t. Writing a
client for a badly documented vendor API, parsing their inconsistent
responses, mapping their enums to yours — that was weeks of grinding
work and is now an afternoon. Document extraction is another. Turning
ten thousand PDFs of purchase orders into structured records was a real
business four years ago; now it’s a pipeline you stand up in a sprint.
Backfill scripts, one-off data repairs, test fixtures for legacy code,
first-line support triage — all cheaper.

So the honest version of my claim isn’t “agents can’t do tedious
work.” They’re very good at tedious work. The claim is narrower:
**agents eat the tedium; they don’t eat the
accountability.** When a schlep is only tedious, expect it to get
cheap and stop treating it as a moat. When a schlep is tedious
*and* has a counterparty, a signature, or a liability attached,
the tedium falls away and the hard part stands there untouched — now
more visible, because everything around it got easier.

That’s a useful filter, actually. Look at any schlep in your business
and ask what’s left after you delete the typing. If nothing’s left, it
was never a moat. If what’s left is a relationship, an attestation, or a
person who’s on the hook, that’s the thing you own.

## The 2026 version of Graham’s question

Graham’s closing move was a reframe: “Instead of asking ‘what problem
should I solve?’ ask ‘what problem do I wish someone else would solve
for me?’” That question still works — it’s still the best trick I know
for getting around your own blindness, because your annoyance is a
signal your conscious mind can’t suppress.

But it’s no longer sufficient, because it selects for
*painful* problems and says nothing about *defensible*
ones. In 2012 those overlapped almost completely. Painful implied nobody
had done it, which implied room. Today, “painful” includes a large
category of problems that are painful right up until someone spends a
weekend with an agent.

So run a second question after his: **which half of this idea
survives a competent stranger pointing an agent at it?**

Be concrete. Write down the two halves. If the surviving column has
entries — we’re the ones with the bank relationship, we hold the
certification the buyer requires, we have the eleven years of exception
rules encoded from customers who told us in person, we answer the phone
at 2am and eat the refund — you have a business. If the surviving column
is empty, you have a feature, and you should either go find a schlep to
attach to it or accept that you’re building something with a short shelf
life.

The failure mode I see most often in AI startups is a team that has
done this exercise implicitly, found the surviving column empty, and
responded by making the cheap half more impressive. Better UI. More
models. A slicker demo. It’s the schlep blindness Graham described, with
a new escape hatch: instead of building a recipe site, you build a very
polished version of the copyable half and mistake the polish for a
position. [Proving the return](https://sublimecoding.com/blog/prove-the-return) is
what breaks that spell, because a buyer who can measure your value
doesn’t care how the sausage was generated.

## The schlep is the company now

Graham’s line was that a company is defined by the schleps it will
undertake. In 2012 that was mostly a statement about ambition. In 2026
it’s closer to a definition, because the schleps are what’s left after
you subtract everything a competitor can regenerate.

Which means the calculus for founders inverted. The schleppy idea
used to be the brave choice — bigger prize, worse odds, more suffering.
Now it’s the conservative one. It’s the version of the plan where,
eighteen months in, you still have something a stranger with a good
agent can’t stand up next to you in a week.

For what it’s worth, the individual-career version of this is the
same shape and I’ve written it separately: [the career I built on
work nobody wanted](https://sublimecoding.com/blog/career-built-on-work-nobody-wanted) is one long argument that the unglamorous
assignment teaches you the system, and that the same asymmetry applies
to a person’s skills that applies to a company’s moat. Nobody competes
with you for it, which is exactly why it’s worth having.

Pick the schlep. It’s the only part of the plan that’s still yours in
a year.

---

## AIOS v2: The Scope Is the Filesystem

URL: https://sublimecoding.com/blog/aios-v2-scope-is-the-filesystem
Published: 2026-07-31
Tags: AI tools, agents, productivity

>
**TL;DR:** Two weeks after I [open-sourced AIOS](https://sublimecoding.com/blog/aios-ai-operating-system-markdown), I
rebuilt its directory structure in place. v1 organized notes by
*type* — Atlas, Calendar, Efforts, the LYT ACE pattern — which is
fine with one scope and quietly breaks at multi-scope scale, because
“whose note is this?” gets answered by re-deriving membership from globs
in four different places, and those four places can disagree. v2 makes
scope the first path segment: one top-level directory per scope,
everything inside it shaped the same way. The prefix *is* the
invariant now — guard checks, ingest, leak audits all key on it instead
of trusting a config file to agree with the folder tree. If you’re
starting a vault today, start scope-first even with exactly one scope;
the overhead is negligible and it makes a second scope free later
instead of a redesign.

## What v1 got right, and where it stopped working

The [launch post](https://sublimecoding.com/blog/aios-ai-operating-system-markdown)
described AIOS as an editorial memory system: raw session signal
compounds into durable project notes, a Knowledge Map indexes them, an
agent reads the index first instead of searching everything. That
architecture hasn’t changed — it’s still the [three-layer wiki
design](https://sublimecoding.com/blog/build-an-llm-wiki-for-your-agent) I wrote about before AIOS existed. What changed is the
directory tree underneath it, and it’s worth being honest about why,
because the failure mode is one anyone building a filesystem-as-memory
system for an agent is going to hit.

v1 organized by note *type*, borrowed from Nick Milo’s [LYT ACE pattern](https://www.linkingyourthinking.com/) — Atlas
(timeless), Calendar (time-based), Efforts (projects), Sources (curated
inputs):

`+/ inbox
AIOS/ core + scope content
Atlas/ timeless notes
Calendar/ time-based notes
Efforts/ projects
Sources/ curated inputs
x/ templates, archive
```

That’s a reasonable shape for a single person with a single context,
which is most of what I described in the launch post — and the bootstrap
interview still tells most people one scope is the right answer for
them. But I run more than one scope myself, and at that scale the
type-first tree stops answering the question that actually matters to a
machine reading it: not “what kind of note is this,” but *whose*
note is this.

## The question v1 couldn’t answer from the path alone

In v1, a scope’s identity was smeared across four unrelated places:
AIOS/me-<scope>.md` for identity,
`Sources/<scope>/` for curated inputs,
`AIOS/Projects/<scope>/` for project brains, and a glob
entry in `AIOS/Systems/layers.tsv` telling everything else
which paths counted as that scope. None of those four locations shared a
parent directory. Nothing about the path told you the scope — you had to
already know the manifest, or grep for it.

That produced three specific failures, documented in [`docs/structure-evolution.md`](https://github.com/sublimecoder/aios/blob/main/docs/structure-evolution.md):

- **Every scope-aware mechanism re-derived membership from
globs.** The write guard, the ingest script, the leak-audit
subagent — each one re-read `repo-layers.tsv` and matched
paths against it independently. The config and the actual folder layout
could disagree, and nothing caught that until something broke.

- **Silent failures when a glob pointed at a directory that
didn’t exist.** A stale or mistyped entry in the manifest didn’t
error — it just matched nothing, and a scope’s notes quietly stopped
being classified as that scope’s notes.

- **Adding a scope meant four coordinated edits.** A new
`me-<scope>.md`, a new
`Sources/<scope>/` directory, a new
`AIOS/Projects/<scope>/` directory, and a new glob row
— miss one and the scope worked in three of the four subsystems and not
the fourth, with no signal telling you which.

None of this showed up with one scope, which is exactly why it
shipped that way and why the launch post’s advice to default to one
scope still holds. It showed up the moment a second scope needed the
same guarantees as the first, and the fix wasn’t a patch to the glob
logic — it was admitting the type-first tree was answering the wrong
question first.

## v2: the scope is the filesystem

The rebuild, landed as [`restructure v2: scope-first tree`](https://github.com/sublimecoder/aios),
makes scope the first path segment instead of an attribute you look
up:

`+/ inbox
AIOS/ scope-neutral OS core
<scope>/ one dir per scope, e.g. main/
 me.md
 notes/ timeless, hand-written
 content/ drafts, works-in-progress
 projects/ AI-maintained project brains
 sources/ immutable curated inputs
archive/ formerly x/
```

Calendar/` didn’t survive as a top-level bucket either —
it moved to `AIOS/History/{days,reviews}`, because time-based
notes are the agent’s memory surface, not a peer of Atlas and Efforts.
That’s a smaller call than the scope restructure, but it’s the same
instinct: stop filing things by what they superficially resemble and
file them by what actually reads them.

The type-based split didn’t disappear — `notes/`,
`content/`, `projects/`, `sources/`
still carry that distinction. It just moved inside the scope boundary
instead of living alongside it. A path’s first segment now names its
scope, full stop, and everything downstream of that segment is free to
keep whatever internal shape makes sense.

A path’s first segment names its scope — the prefix everything keys
on, instead of a config file everything has to trust.

## What changes when the prefix is the invariant

This is the part that’s worth the rebuild, and it’s mechanical rather
than philosophical:

- **Guard checks become prefix validation.** The write
guard used to match a path against a glob pulled from a config row. Now
it checks whether the path starts with a known scope directory. There’s
no config to drift out of sync with the folder tree, because the folder
tree *is* the config.

- **“Sources are immutable” becomes a shape rule, not a
per-scope rule.** In v1, immutability was enforced per glob, once
per scope, and a new scope needed that rule written for it explicitly.
In v2 the rule is `*/sources/*` — it applies to every scope
automatically, including one that doesn’t exist yet.

- **Adding a scope is a `mkdir` and one manifest
row.** Not four coordinated edits across unrelated directories —
one new top-level folder (or a copy of the shipped `main/`
scaffold) and one glob entry pointing at it.

- **`AIOS/` stays deliberately
scope-neutral.** The whole prefix invariant only holds if the OS
core never itself looks like a scope. That’s a constraint on future
changes to `AIOS/`, not just a description of the current
tree.

The commit history behind this is left readable on purpose — v1’s LYT
ACE layout is still there to diff against in git log, not squashed away,
because the shape of what broke is as useful to see as the shape of the
fix.

## The quieter half: cleanup ported back from the vault

Alongside the restructure, a batch of fixes that had already proven
themselves in my own running vault got ported back into the public
template — smaller changes individually, but worth naming because
they’re the kind of thing that only surfaces after a system runs
unattended for a while:

- **`aios-digest.sh` now fingerprints its diffstat
output**, so identical digest blocks collapse into one instead of
piling up duplicate entries. It also floors untracked-only churn and,
critically, commits *only* the digest paths it wrote — so staged
human work sitting in the vault never gets swept into an unattended
commit alongside it.

- **`aios-nightly-ingest.sh` gained a heartbeat log
and a `.lastrun` marker**, plus an atomic
`mkdir`-based lock with a stale-lock reaper, so a scheduled
3am run can’t race a human firing `/aios-ingest` by hand
mid-session.

- **The skill set consolidated from nine to three**:
`propagate` now covers what used to be separate cascade and
harmonize skills, `chronicle` absorbed three overlapping
skills, and `vault-map` absorbed the builder and janitor
skills that used to duplicate its work. Three new janitors replaced ad
hoc cleanup: `log-rollup` (monthly log rotation,
count-verified so nothing silently vanishes), `km-rotate`
(Knowledge Map lines rotate back to summaries without dropping facts),
and `open-flags` (⚠️/OPEN markers get dispositioned on every
ingest with a 30-day escalation instead of aging out unnoticed).

- **A new `publish-verify` agent** runs a
post-publish token grep and traces first-person claims back against the
vault — an unsourced claim at publish time is treated as a fabrication
risk, not a style nitpick.

None of that required the restructure. It shipped in the same pass
because it was already vault-proven and the template was the honest
place for it to land.

## The general lesson, if you’re building something similar

If a filesystem’s primary reader is a machine and it has to answer
“does this belong to X” for more than one X, don’t organize by content
type and bolt scope on as metadata. Organize by scope first and let
content type be a convention *inside* that boundary. Type-first
taxonomies like LYT ACE are built for a single human browsing their own
notes, where “whose is it” is never in question because the answer is
always “mine.” The moment a second identity enters the picture, that
assumption is the whole design, and everything downstream — access
guards, leak audits, “what can I show this agent” — either keys on a
path prefix or keys on a config file hoping it still matches the folder
tree. Prefixes don’t drift. Config files do.

## Try it

Everything above is in [github.com/sublimecoder/aios](https://github.com/sublimecoder/aios),
including the [full
structure-evolution writeup](https://github.com/sublimecoder/aios/blob/main/docs/structure-evolution.md) with the v1 tree, the v2 tree, and the
reasoning in more detail than fits here. If you’re setting one up fresh,
start scope-first even with one scope — `main/` ships
pre-filled as exactly that scaffold, so there’s no extra step for the
common case, and a second scope later is a directory copy instead of a
rewrite.

If you’re past the point of running this for yourself and thinking
about how a team standardizes shared context across an AI-native
engineering org — where the scope boundaries actually need to sit, and
where a real productivity system stops and tool sprawl starts — that’s
the kind of design question I work through with clients in [fractional engineering leadership
engagements](https://sublimecoding.com/consulting).

---

## The One File Where Your AI Model Names Live

URL: https://sublimecoding.com/blog/model-routing-file-ai-coding-agents
Published: 2026-07-29
Tags: AI tools, agents, productivity

>
**TL;DR:** If you run agents across more than one repo,
your model choices are probably hardcoded in a dozen places — a
`CLAUDE.md` here, a shell alias there, a hardcoded model
string in a subagent config. That’s fine until a model gets deprecated,
repriced, or quietly downgraded, and now you’re grepping the filesystem
for every place you typed `claude-opus` by hand. The fix is
one file: a git-versioned markdown table that maps task type to model,
imported into every session on the machine. Change a model once, every
repo inherits it. It’s not clever. It’s just the one place model names
are allowed to live.

## Model names rot the moment they’re hardcoded

Every agent config has a model name in it somewhere. A
`CLAUDE.md` that says “use Opus for planning.” A subagent
definition with `model: sonnet` baked into the frontmatter. A
shell script that calls a specific model ID because that’s what worked
when you wrote it eight months ago. None of this is wrong on day one.
It’s wrong on day two hundred, when the vendor ships a new model,
deprecates an old one, or moves the price on the one you’d standardized
on — and you discover you never actually standardized on anything. You
scattered the same decision across every repo you touch, and now every
one of those repos needs its own fix.

The failure mode isn’t dramatic. It’s just drag. One repo still
points at a model that’s been quietly superseded. Another repo’s
subagent is running a heavyweight model on work that never needed it,
because nobody went back and right-sized it after the first draft. A
third repo has a stale model ID that still technically works but isn’t
what you’d pick today. None of this breaks anything outright — it just
means every model decision you’ve ever made is still live,
uncoordinated, and nobody’s the one place you’d go to see the whole
picture or change it in one motion.

The tell that you have this problem: try to answer “which model
handles code review across all my projects” without grepping. If you
can’t, the decision isn’t a decision — it’s a pile of copies.

## The one-file pattern

The fix is boring on purpose. One markdown file, versioned in git,
that is the single place model names are allowed to live. Every repo and
session on the machine imports it rather than declaring its own. Here’s
the core of the file I run — trimmed to the table itself, row labels
lightly compressed:

`[](#cb1-1)# Effort table — task type → model
[](#cb1-2)
[](#cb1-3)| Task type | Effort | Model |
[](#cb1-4)|---------------------------------|--------|-----------|
[](#cb1-5)| Orchestration, judgment, review | high | Opus 4.8 |
[](#cb1-6)| Drafting / bulk edits, subagent | medium | Sonnet 5 |
[](#cb1-7)| Scans, greps, per-item grunt | low | Haiku 4.5 |
```

Four columns in the real file, three rows. The fourth column —
trimmed here for width — is the literal model ID string
(claude-opus-4-8`, `claude-sonnet-5`,
`claude-haiku-4-5-20251001`), kept right beside the
human-readable name so a rename in a vendor’s marketing doesn’t force
you to go hunting for the ID that actually ships. Task type is the thing
you’re actually deciding — not “which model is smartest” but “what kind
of work is this.” Effort is the reasoning-effort dial
(low/medium/high/max) that goes with the model, because the tier and the
dial are the same decision made twice if you split them into separate
files.

The three rows map to three kinds of work I actually do, every day,
across every project: orchestration and judgment calls that are
wrong-once-and-it’s-expensive go to the top tier; mid-weight drafting
and bulk edits that a subagent handles go to the middle tier; mechanical
grunt work — scans, greps, per-item busywork where the failure mode is
cheap and obvious — goes to the bottom tier. That’s it. No fourth row
for “special cases,” because special cases are exactly the kind of local
knowledge that belongs in a repo’s own override, not in the shared
table.

## How it’s wired: one import, one override

The file lives in a git-versioned personal vault, not inside any
single project. My global Claude Code config
(`~/.claude/CLAUDE.md`, loaded at the start of every session
everywhere on the machine) `@import`s it. That’s the entire
mechanism — no build step, no registry, no service. Every repo I open
inherits the same three rows the moment the session starts, because the
import happens before the session sees any project-specific
instructions.

The override path matters as much as the import. A repo is allowed to
override one row in its own `CLAUDE.md` — a project that does
almost nothing but mechanical bulk edits might permanently pin the
middle tier down a notch, or a security-sensitive repo might promote
review work up a tier regardless of what the shared table says. The
override is local and explicit, which is the point: the shared file is
the default, not a mandate, and a repo that needs to diverge says so in
its own config where anyone reading that repo can see why.

The rule that keeps the whole thing from rotting the same way the
scattered version did is one sentence baked into the file itself: change
a model or a tier here, and only here — every consumer inherits it. That
sentence is doing real work. It’s the difference between a table that
stays canonical and a table that becomes one more copy the moment
someone “just quickly” hardcodes an ID somewhere convenient.

It’s the same earned-entry discipline I use for [CLAUDE.md itself](https://sublimecoding.com/blog/claude-md-after-50-commits): don’t add
a row speculatively, add it because a real duplication just cost you
something, and prune anything that stops being true. The table is
smaller than most CLAUDE.md files for exactly that reason — three rows
have survived because three rows are all the ones that have actually
earned their place so far.

## What a swap actually looks like

This is the part that justifies the whole exercise: what happens the
day a model gets deprecated, repriced, or a better option ships.

Before the file existed, that day meant an audit. Grep every repo for
the old model string, one at a time, replace it, hope you didn’t miss a
subagent config buried three directories deep, hope the replacement
didn’t silently change the reasoning-effort dial along with the model
name because those two facts lived in different places. After the file
exists, that day is a one-line diff to a single row, plus a line in the
changelog section at the bottom of the file recording what changed and
when. Every repo that imports the table picks up the new model on its
next session, with zero repo-level edits, because none of them ever
hardcoded the name — they all deferred to the one file that did.

I applied the identical move one level up, three days after I built
this table. I run a separate sibling file — same mechanism, same import,
same edit-once-everywhere property — that isn’t about which model to
call but about how to reason once you’re in a session: read the actual
intent before solving, re-derive facts instead of recalling them, mark
certainty levels honestly, attack your own conclusion before shipping
it. That file has nothing to do with model names; it’s the operating
discipline a session should follow regardless of which model is running
it. The point of building both the same way is the same point twice:
neither the model choice nor the reasoning habit should be something you
re-litigate per repo. Get it right once, in one file, and let everything
else defer to it.

Neither the model choice nor the reasoning habit should be something
you re-litigate per repo.

That’s the mechanics side of a bet I’ve made in more detail
elsewhere: your accumulated agent knowledge should [outlive any single agent](https://sublimecoding.com/blog/ai-memory-outlives-any-agent),
because nobody knows which vendor is ahead a year from now. This post is
the routing half of that bet — not the argument for why portability
matters, but the specific, boring artifact that makes model choice
portable in practice.

## Enterprises are converging on the same shape

I don’t want to oversell the parallel, but it’s worth naming plainly:
this is the same architecture the largest software company on earth just
described in public. Satya Nadella [posted on
X](https://x.com/satyanadella/status/2080329851127669104) that the move is “using the right model for each task,” with
harness, memory, context, and skills deliberately externalized outside
of the model itself. I read the memo itself in [the postscript to my
token-capital piece](https://sublimecoding.com/blog/nadella-token-capital-small-teams) — the point that belongs here is narrower.
Microsoft’s version of this file is Foundry plus first-party evals plus
traffic-routing infrastructure sized for a company with a model family
of its own to route toward. Yours is one markdown table in a git repo.
The shape is identical; the budget is the only thing that isn’t.

## What this file does not do

Be precise about the boundary, because it’s easy to oversell a
pattern this simple. The effort table routes. It does not evaluate. It
tells you which model handles which class of work; it has no opinion on
whether that assignment is actually correct, and it never will, because
a static markdown table can’t watch outcomes.

The judgment about whether Haiku-tier is actually good enough for
your mechanical scans, or whether your orchestration work genuinely
needs the top tier every time, still has to come from somewhere that
watches real output against real requirements. On this site, that’s the
test suite — `mix test` and a domain-specific health check
that gates the SEO and content invariants I actually care about, and
doesn’t care in the slightest which model produced the diff it’s
checking. That’s the eval. The effort table just decides who gets to
attempt the work; the test suite decides whether the attempt was good
enough. Route with the table, judge with the suite, and don’t let the
first one pretend to do the second one’s job.

## What to do this week

You don’t need the vault infrastructure to start — [the vault itself is a
separate, later decision](https://sublimecoding.com/blog/aios-ai-operating-system-markdown) if you get there. The table works as a
standalone file on day one.

- **Write the three-row table today.** Task type, effort
dial, model, ID. Don’t overthink the tiers — orchestration/judgment,
mid-weight drafting, mechanical grunt work covers almost
everything.

- **Put it in git, not in a repo.** A personal or team
vault, versioned, separate from any single project — so it survives that
project getting archived.

- **Import it globally, override locally.** One line in
your global agent config to pull it into every session; let individual
repos override a row only when they have a real reason to.

- **Add the one editing rule.** “Change a model or tier
here, and only here.” Write it into the file itself so the discipline
doesn’t depend on anyone remembering it.

- **Keep a changelog at the bottom.** Every swap gets a
dated line. Six months from now you’ll want to know when and why a model
moved tiers, and the file is the only place that history should
live.

- **Let your test suite stay the judge.** Don’t ask the
table to do more than route. If a model assignment is wrong, that shows
up in your review gate or your test failures, not in the table.

If you’re building this out for a team rather than just yourself —
where the override question turns into a real governance question about
who gets to touch the shared file — that’s the kind of setup work I help
AI startups and engineering teams get right the first time. [Let’s talk](https://sublimecoding.com/consulting).

---

## What Together AI's $800M Round Says About Elixir

URL: https://sublimecoding.com/blog/together-ai-elixir
Published: 2026-07-23
Tags: Elixir, AI startups, engineering

>
**TL;DR:** Together AI closed an [$800M
Series C at an $8.3B valuation](https://www.businesswire.com/news/home/20260701243402/en/Together-AI-Raises-$800-Million-at-$8.3-Billion-Valuation-to-Make-Frontier-AI-Accessible-to-All) on July 1, 2026 — led by Aramco
Ventures, with NVIDIA and General Catalyst participating, and [annualized
bookings past $1.15B](https://techcrunch.com/2026/07/01/neocloud-together-ai-raises-800m-leaps-to-8-3b-valuation/) last quarter. Here’s the receipt that doesn’t
make the funding headlines: Together AI’s own [engineering
job postings](https://job-boards.greenhouse.io/togetherai/jobs/5169470007) describe Elixir/Phoenix services on the BEAM running
authentication, SSO, OAuth, organizations, projects, API keys, and
role-based access control — the layer every other product call in the
company routes through. Elixir’s raw usage share is still low single
digits. The signal isn’t the share. It’s where a company at this
valuation chose to put its critical path, and that it’s hiring publicly
to defend the choice.

**Update, September 2, 2026:** the Greenhouse job posting this post relied on has since been removed, and no archived copy exists. Together AI’s current job board lists no Elixir roles, and its public GitHub org shows none. The Elixir also appears to have arrived through the December 2024 acquisition of CodeSandbox, scoped to that team’s auth and org-management services, which this post did not disclose. The funding figures below still hold; the present-tense claim that Together AI is hiring Elixir engineers into its critical path does not. A row-by-row re-check of this and fourteen other “companies using Elixir” claims publishes here on September 24, 2026: [Who Actually Runs Elixir in Production, Fact-Checked](https://sublimecoding.com/blog/elixir-in-production-fact-checked).

## The round

On July 1, 2026, [Together
AI announced](https://www.businesswire.com/news/home/20260701243402/en/Together-AI-Raises-$800-Million-at-$8.3-Billion-Valuation-to-Make-Frontier-AI-Accessible-to-All) an $800 million Series C at an $8.3 billion post-money
valuation — more than double the $3.3 billion mark from its Series B in
early 2025. Aramco Ventures led the round; [TechCrunch’s
coverage](https://techcrunch.com/2026/07/01/neocloud-together-ai-raises-800m-leaps-to-8-3b-valuation/) lists Vista Equity Partners, General Catalyst, Emergence
Capital, and NVIDIA among the participants, notes the prior $3.3 billion
Series B mark, and reports annualized bookings past $1.15 billion in the
company’s most recent quarter — on the back of enterprises moving
inference workloads off closed frontier models and onto the open models
Together AI serves at a fraction of the cost.

None of that is an Elixir story on its face. Together AI is a
GPU-cloud-and-inference company; its headline product is compute, not a
web framework. But funding announcements and cap tables don’t tell you
what’s actually running in production, and the interesting part of this
story isn’t in the press release — it’s in the job board.

[image: Infographic: Elixir isn’t a niche bet anymore — Together AI’s $800M Series C at an $8.3B valuation, $1.15B+ annualized bookings, and the BEAM running authentication, organizations, API keys, access control, and collaboration]

Infographic: Elixir isn’t a niche bet
anymore — Together AI’s $800M Series C at an $8.3B valuation, $1.15B+
annualized bookings, and the BEAM running authentication, organizations,
API keys, access control, and collaboration

## The receipt: what’s actually on the BEAM

Together AI’s Greenhouse listing for a [Software
Engineer role on its Identity & Collaboration team](https://job-boards.greenhouse.io/togetherai/jobs/5169470007) — part of the
Product Foundations engineering group — spells out the stack in plain
language: “Build and ship features across the stack — TypeScript/Next.js
on the frontend and Elixir/Phoenix services on the backend.” The scope
of that backend isn’t a side project. It’s “authentication flows
(including SSO and OAuth), organizations, projects, API keys, and
role-based access controls,” and engineers on the team are expected to
“learn the Elixir/Erlang VM (BEAM) and how we run it in production.”

Read that list again and notice what it isn’t. It isn’t a marketing
site, a docs portal, or an internal admin tool — the places a startup
tends to park a language nobody wants to bet the company on. It’s
identity and access control: the layer that authenticates every request,
scopes every API key, and enforces every permission check across the
product. If that layer goes down or gets the concurrency model wrong,
nothing else in the platform works correctly either. Together AI put
that on Elixir and Phoenix, running on the BEAM, and is hiring engineers
into it by name.

I’ve made the architecture case for exactly this kind of choice
before — [why Elixir
fits the AI startup backend](https://sublimecoding.com/blog/why-elixir-for-ai-startup-backend) covers the concurrency and supervision
reasoning in full, and I won’t re-run it here. What’s worth sitting with
instead is what it means when a company at an $8.3 billion valuation,
closing a $800 million round, is the one making that choice
publicly.

## Why raw adoption share still says “niche” — and why that’s the wrong number to read

Here’s the concession, stated plainly instead of buried: by raw
usage, Elixir is not mainstream. In the [2025 Stack
Overflow Developer Survey](https://survey.stackoverflow.co/2025/technology), Elixir sits at 2.7% of all respondents —
low single digits, well behind JavaScript, Python, and the other
top-line languages. If you’re grading “has Elixir gone mainstream” as a
flat yes/no against that number, the honest answer is no, and anyone
telling you otherwise is selling something.

But raw adoption share answers the wrong question for this story. It
tells you how many developers, in aggregate, across every kind of
company and every kind of workload, happen to write Elixir day to day.
It says nothing about *where* the growth is concentrating, or
*who* is choosing it for the parts of their system that can’t
afford to be wrong. Those are different questions, and the second one is
the one funded AI infrastructure companies are answering with their
hiring budgets right now.

A company at an $8.3B valuation putting Elixir on its identity and
access-control critical path — and hiring for it publicly — is a
different kind of signal than a language-share percentage.

Adoption share is a lagging, backward-looking average across the
entire industry, most of which isn’t building anything like what
Together AI is building. It smooths a real signal into noise. The signal
that actually matters for anyone deciding what to build a new
AI-infrastructure backend in isn’t “what percentage of all developers
use this,” it’s “what are the best-capitalized, fastest-growing
companies in the category choosing for the part of the system every
request has to pass through.” Together AI answering that with Elixir, in
public, on a job board, the same month it closes a nine-figure round, is
a stronger data point for that narrower question than the aggregate
percentage will ever be.

It’s not an isolated one, either. Elixir’s flagship web framework,
Phoenix, was the [most
admired web framework in Stack Overflow’s 2023 survey](https://survey.stackoverflow.co/2023/#section-admired-and-desired-web-frameworks-and-technologies) — more
developers who had shipped on it wanted to keep using it than for React,
Node.js, or Next.js — a satisfaction signal from the people who’ve
actually run it, which is a very different thing from a popularity count
of people who haven’t. Small usage share and high admiration from the
people who use it in production is a specific pattern: a technology that
isn’t losing engineers who try it, it’s losing at the top of the funnel,
before anyone’s actually run it. Together AI’s hire is what it looks
like when a well-funded team gets past that top of the funnel and puts
it on the critical path anyway.

## One data point — but the right kind

This isn’t a “companies using Elixir” listicle, and I’d be doing you
a disservice if I dressed it up as one — a list of logos proves nothing
on its own. What’s worth tracking instead is the pattern this round is
one data point in: funded AI-infrastructure companies reaching for the
BEAM specifically for the stateful, concurrent, always-on layer —
identity, access control, collaboration, the connective tissue every
other service depends on — while building the rest of their stack in
whatever language fits the rest of their stack. It’s the same reasoning
behind [why the BEAM keeps turning
up under AI agent workloads](https://sublimecoding.com/blog/elixir-beam-ai-agents): the parts of a system that have to hold
state, stay up, and coordinate concurrent work reliably are exactly the
parts where the BEAM’s guarantees stop being a nice-to-have. That’s not
“Elixir is eating the world.” It’s “Elixir is winning a specific,
high-stakes layer of infrastructure at companies with the capital and
the technical judgment to choose deliberately,” and that’s a narrower,
more useful claim to make than the flat mainstream-adoption one, because
it’s the claim the evidence actually supports.

If you’re already building on Elixir and want the durable-execution
side of this argument rather than the identity-layer side, I’ve covered
that separately: [running
AI agent jobs on Oban instead of a bespoke queue](https://sublimecoding.com/blog/oban-durable-ai-agent-runtime) is the same BEAM
guarantees applied to a different critical path.

If you’re weighing the same choice for an AI startup’s backend — what
belongs on the BEAM, what doesn’t, and where the concurrency model
actually earns its keep instead of just sounding good in a pitch —
that’s the conversation I have with founders and CTOs in [fractional CTO and engineering leadership
engagements](https://sublimecoding.com/consulting). Bring the actual system you’re building; the right
answer depends on what’s in it, not on a job posting from a company that
isn’t yours.

---

## Shadow AI at a Startup That Can't Buy a CASB

URL: https://sublimecoding.com/blog/shadow-ai-startup-casb
Published: 2026-07-22
Tags: security, founders, vCISO

>
**TL;DR:** Every “shadow AI” article on page one is
written by a vendor selling you a CASB — a piece of enterprise software
that watches every cloud app your employees touch and enforces a policy
about what data can go where. It’s the right tool at 500 employees and
the wrong conversation at 12. You don’t need software; you need three
free things done in the next two weeks: a card-statement-and-SSO-log
audit to see what’s actually connected, a no-blame survey to find what
isn’t, and a one-page policy with a plain data-classification rule. The
part that determines whether any of it works isn’t the audit — it’s
whether the survey is genuinely no-blame. Punish the disclosure once and
the tools don’t go away, they just go invisible, and invisible is the
actual risk, not usage.

## What “shadow AI” actually means, and why the standard fix doesn’t fit you

Shadow AI is a specific, boring thing: your employees are pasting
company information into AI tools you never approved, evaluated, or even
know exist. Someone drops a customer’s support ticket into ChatGPT to
draft a better reply. Someone uploads a contract to Claude to summarize
it. Someone connects a Notion doc to a Chrome extension that “makes your
writing better” and never reads what permissions it asked for. None of
it is malicious. All of it is your company’s data now sitting on
infrastructure you didn’t choose, under a data-handling policy you never
read, and — depending on the tool’s terms — sometimes feeding a model
you have no relationship with.

The scale of this is not a niche worry. Verizon’s 2026 Data Breach
Investigations Report found that the share of workers using AI tools on
corporate devices tripled in twelve months, from 15% to 45%, and that
shadow AI is now the third most common non-malicious insider action
showing up in data-loss-prevention logs — a fourfold year-over-year
increase ([Verizon,
2026 DBIR](https://www.verizon.com/business/resources/T1ae/reports/2026-dbir-data-breach-investigations-report.pdf)). A separate 2026 survey of 1,250 office professionals at
large organizations found two-thirds had used an AI tool at work despite
believing it wasn’t permitted under company policy, and 88% had shared
work-related information with a public AI system — 34% customer
information, 31% sensitive business documents ([PagerDuty,
2026 Shadow AI Workplace Survey](https://www.pagerduty.com/newsroom/shadow-ai-workplace-survey-2026/)). Those numbers are from companies
with a compliance department. At a 12-person startup, nobody’s even
counting.

Every piece of enterprise coverage on this topic converges on the
same fix: buy a CASB. A **cloud access security broker** is
software that sits between your employees and every cloud app they
touch, discovers what’s connected (often via OAuth grants and network
traffic), and enforces a policy — block this category, allow that one
read-only, flag anything touching a data type you’ve labeled sensitive.
Palo Alto, Netskope, Zscaler, and a wave of AI-specific entrants now
sell exactly this for the “shadow AI” problem specifically. It’s a real
product solving a real problem, and it is comically oversized for a 5-15
person company. A CASB assumes a procurement process, a security team to
tune its rules, a device fleet uniform enough to enforce an agent on,
and a budget line that starts in five figures a year before you’ve hired
anyone to run it. You have none of that, and you’re not supposed to yet
— buying one now is the same mistake as buying a full-time CISO before
you have a security program for them to run, and I’ve written about [why that hire
ordering is backwards at your stage](https://sublimecoding.com/blog/security-engineer-or-vciso-first-hire).

The risk was never that your team uses AI. The risk is that you don’t
know which AI, with what data, and you found out from a customer instead
of from your own team.

So skip the vendor conversation entirely. Here’s the version that
costs nothing and takes about two weeks of part-time attention, which is
the version I actually run with clients at this stage.

## Step 1: the audit you already have the data for

Before you ask anyone anything, find out what’s already connected.
You don’t need a discovery tool — you need two things you already pay
for.

**Your corporate card statement.** Pull the last three
months from Brex, Ramp, or whatever you use, and search for AI vendor
names: OpenAI, Anthropic, Perplexity, Otter.ai, Jasper, Gamma,
Grammarly, Fireflies, Clay, and the dozen coding and writing assistants
that bill monthly. Every one of these is an employee who decided the
company needed a tool badly enough to expense it, and every one of them
is a data flow you didn’t design.

**Your SSO or Google Workspace admin console.** If
you’re on Okta, Google Workspace, or Microsoft Entra, there’s an admin
view of every third-party app your employees granted access to via “Sign
in with Google” or “Sign in with Microsoft” — usually under something
like connected apps, OAuth grants, or the security/API access report.
This surfaces the free tools that never touch a card: the ChatGPT
account someone connected to their calendar, the note-taking AI that
reads their email, the browser extension with access to every page they
visit. This report already exists in tools you’re paying for regardless;
you’re just looking at a screen you’ve never opened.

Between the two, you’ll have a real list in under an hour — not a
complete one, but a real one. That’s the baseline. Everything from here
is closing the gap between what this audit found and what’s actually
happening.

## Step 2: the no-blame survey, and why the “no-blame” part is the whole mechanism

The card-and-SSO audit misses the AI use that never touches company
infrastructure at all: the free ChatGPT account logged in with a
personal Gmail on a personal laptop, opened in an incognito tab
specifically because someone suspected it wasn’t sanctioned. You cannot
audit your way to that list. You have to ask for it, and the only way
anyone tells you the truth is if the asking is genuinely, verifiably
safe.

Send one short form, from a founder, with a plain ask: *“What AI
tools do you actually use for work, day to day, for what? No one is in
trouble for anything on this list — we’re trying to understand our real
exposure, not build a case against anyone.”* Anonymous or not
doesn’t matter much at 12 people; what matters is that the promise is
real and someone tests it by naming something borderline.

This isn’t a soft HR nicety bolted onto a security process — it’s the
same mechanism that governs whether anyone reports a phishing click, and
I’ve written about that mechanism in more depth: [a scared team is your
biggest attack surface](https://sublimecoding.com/blog/psychological-safety-security-control). Punish the first honest answer to a survey
like this — even mildly, even just a raised eyebrow in Slack — and you
haven’t reduced the AI usage. You’ve taught your team to keep doing
exactly what they were doing, quieter. The tool doesn’t leave; your
visibility into it does. Given the choice between a team that uses an
unapproved tool and tells you, and a team that uses the same tool and
hides it, the second is strictly worse — you’ve traded a known,
manageable risk for an invisible one, for free, by reacting badly
once.

## Step 3: one page, one rule, no procurement

You don’t need a policy document. You need a data-classification rule
simple enough that someone can hold it in their head mid-task, because
that’s the only place a policy actually works — nobody re-reads the wiki
before pasting a ticket into an AI tool. A workable version fits on one
page and looks roughly like this:

Data type
Approved AI tools (paid, business-tier account, DPA in place)
Free/personal-tier AI tools
Never paste this anywhere

Public info, marketing copy, general writing
Yes
Yes
—

Internal docs, code, non-customer business data
Yes
No
—

Customer PII, contracts, financial data
Case-by-case, named tools only
No
—

Credentials, security details, anything under NDA
—
—
Always

The specific tools you name in the “approved” column matter less than
the fact that a rule exists at all and someone can check it in ten
seconds. Most AI vendors now offer a business tier with a
data-processing agreement that contractually excludes your data from
training — that’s the tier worth naming as approved for
internal-but-sensitive work, and the free consumer tier is the one worth
explicitly ruling out for anything past “public.” Write it, put it
somewhere every new hire actually sees in week one, and revisit it once
a quarter as the tools your team actually uses shift — which the survey
in step 2 will keep telling you, if you run it again in six months.

None of this requires a vendor call, a trial, or a line item. It
requires a founder spending a focused afternoon on an audit, a form, and
a table — which is precisely the kind of decision-not-implementation
work that fractional security leadership exists to run for you when you
don’t have the bandwidth yourself, and it’s the same category of
hours-per-month math I’ve laid out for [vCISO cost versus a full-time
hire](https://sublimecoding.com/blog/vciso-math-for-ai-founders).

## When you actually outgrow this

The free version has a real ceiling, and pretending otherwise past
that point is its own risk. You’ve outgrown it when any of these become
true:

**You’re handling regulated data.** Health records,
payment data, anything that triggers HIPAA, PCI, or a state privacy law
changes the calculus entirely — the free playbook is a starting posture,
not a compliance program, and regulated data needs contractual and
technical controls a one-page table can’t provide. If that’s you from
day one, most of this article still applies as your baseline, but you
need it paired with the fuller security architecture I’ve described in
[how I’d
run security at an AI-native company in 2026](https://sublimecoding.com/blog/running-security-at-an-ai-native-company-2026).

**An enterprise customer’s security review asks the question
directly.** Once a prospect’s procurement team is asking “how do
you govern employee use of AI tools,” a verbal answer and a Google Doc
stop being sufficient evidence, even if the underlying practice is sound
— you need something closer to a documented, monitored program, and
that’s usually the actual trigger, not headcount.

**Your identity provider already has the feature for
free.** If you’re already on Okta or Google Workspace at a tier
with an app-discovery or OAuth-grant report, you’re most of the way to
CASB-lite functionality with zero new spend — check what your existing
IdP surfaces before buying anything new that duplicates it.

**You’ve crossed roughly 25-30 people with no security
owner.** Past that headcount, the no-blame survey and the
one-page policy stop scaling on founder attention alone, and you need
someone accountable for re-running the audit, updating the policy, and
fielding the “can I use this new tool” questions that start arriving
faster than a quarterly check-in can absorb. That’s usually the same
inflection point where the vCISO-first hiring order I’ve written about
elsewhere starts to apply.

Until one of those is true, the CASB conversation is a distraction
dressed up as diligence. The actual failure mode at your stage was never
“employees use AI.” It’s finding out which tool, with what data, from a
customer’s security questionnaire instead of from your own team — and
the fix for that costs an afternoon, not a contract.

If you want a second pair of eyes on where your actual exposure sits
— or you’re staring down the first enterprise security review that’s
going to ask this question in writing — [that’s the
conversation I have with founders every week](https://sublimecoding.com/consulting).

---

## Security Engineer or vCISO? Your First Hire, by Stage

URL: https://sublimecoding.com/blog/security-engineer-or-vciso-first-hire
Published: 2026-07-17
Tags: security, vCISO, hiring

>
**TL;DR:** “We need to hire security” is two different
sentences wearing one set of clothes. A **security
engineer** builds and fixes — appsec, infra hardening, tooling,
the code-level work. A **vCISO** decides and represents —
the program, the priorities, the customer security reviews, the audit
strategy. They are not interchangeable, and the wrong first hire is
expensive in both directions: a $220K engineer with no program to
execute, or a strategist with nobody to implement. The stage answer,
compressed: **pre-seed, neither** — it’s engineering
hygiene and founder-owned basics. **Seed to pre-Series A selling
into the enterprise, vCISO first** — the work at that stage is
decisions and paperwork, and the hours don’t justify a full-time
anything. **Series A with a real attack surface, first security
engineer** — and the vCISO graduates to advisory. **Series
B+, a full-time security lead.** And yes — I sell vCISO work, so
read the bias disclosure below before you take my word for any of
it.

## Two different jobs wearing one word

When a founder tells me “we need to hire security,” my first question
is always the same: *do you need someone to build things, or someone
to decide things?* The blank pause that usually follows is the whole
problem.

A **security engineer** is a builder. They write and
review code, harden your infrastructure, wire up your secrets
management, tune your dependency scanning, fix the auth bug, build the
audit-logging pipeline. Their output is commits, configurations, and
closed findings. They are judged the way any engineer is judged: by what
ships.

A **vCISO** — or any security *leader* — is a
decider and a representative. Their output is a program: what you’ll fix
and in what order, what you’ll accept and document, what you’ll tell the
enterprise customer’s security team on Thursday’s call, how you’ll scope
the audit so it unblocks revenue instead of consuming a quarter. They
are judged by whether security decisions stop being bottlenecks —
whether deals close, audits pass, and the engineering team knows what
“secure enough for now” means this quarter.

The market blurs these constantly, because both get filed under
“security” in the hiring plan and both cost real money. But they fail
differently, they interview differently, and — this is the part that
matters — **they’re needed at different stages.**

## The mis-hire, in both directions

Get the order wrong and you pay twice: once in salary, once in the
work not getting done.

**The engineer with no program.** Hire a senior security
engineer — call it $220K fully loaded, and that’s not the top of the
market — into a 12-person startup with no security program, and you’ve
bought a very expensive person who now has to invent their own job. Some
can. Most, reasonably, do what engineers do: they pick the technically
interesting work. You get a beautifully tuned SIEM and a hand-rolled
scanner while the thing actually threatening the company — the
enterprise security questionnaire sitting in your sales pipeline, the
SOC 2 scoping decision, the access-review policy nobody wrote — sits
untouched, because none of that is engineering work. Six months later
the founder is confused about why “hiring security” didn’t make the
security problems go away. It did — just not *their* security
problems.

**The strategist with no hands.** The inverse fails just
as hard. Bring in security leadership — fractional or otherwise — at a
company where the actual gap is *implementation*, and you get
beautifully prioritized findings that nobody fixes. The roadmap says
rotate the credentials, isolate the tenants, add the rate limiting. The
engineering team, heads down on product, treats it as a wishlist. A
vCISO without engineering capacity to direct is a report generator. I
say this as someone who writes those reports: the engagement only works
if someone on the other side can execute, even if that someone is a
product engineer with 20% of their time carved out.

The fix for both failure modes is the same: match the hire to the
*kind* of work your stage actually generates. So let’s do it by
stage.

## Pre-seed: neither

At pre-seed you do not have a security hiring problem. You have an
engineering hygiene problem, and hygiene is not a role — it’s a set of
defaults your existing engineers either practice or don’t.

What actually matters here fits on an index card: SSO or a password
manager everywhere, MFA enforced, secrets out of the repo and into a
manager, dependency updates on, cloud account with billing alerts and no
root-key usage, laptops encrypted, and one founder who owns the answer
to “what do we do if something leaks.” That founder-owned basics list is
boring precisely because it’s effective — the pre-seed incidents I’ve
seen were all in the gap between “we knew better” and “nobody owned
it.”

Spending real money on security headcount — or even a meaningful
fractional engagement — at this stage is solving next year’s problem
with this year’s runway. The one exception: if you’re building something
regulated from day one (health data, payments infrastructure, kids’
products), the compliance floor arrives before product-market fit does,
and you should read the next section as applying to you now.

## Seed to pre-Series A: the vCISO, first — and here’s my bias

Full disclosure before this section, because it’s the one that pays
my invoices: **fractional security leadership is literally what I
sell.** [My consulting practice](https://sublimecoding.com/consulting) is built
around vCISO engagements for exactly this stage of company. Discount
accordingly — and then check the reasoning, because I’ll also tell you
below when the answer *isn’t* a vCISO.

Here’s why the strategist comes first at this stage. Look at what
security work a seed-stage company selling into the enterprise actually
generates in a given month: a customer security questionnaire (or
three), a decision about whether and when to do SOC 2, a vendor review
from a prospect’s procurement team, an access-control policy that needs
to exist in writing, a prioritization call about which of the 40 scanner
findings matter, maybe an architecture review for the feature that
touches customer data. Almost none of that is engineering. All of it is
decisions and representation — and every piece of it is blocking
revenue, not blocking uptime.

That’s a leadership workload, and it’s a *part-time*
leadership workload. There are maybe 15–25 real hours of it a month. A
full-time CISO runs $200–400K and is over-leveled for that work in both
cost and scope — I’ve done the [full cost math on this
elsewhere](https://sublimecoding.com/blog/vciso-math-for-ai-founders), so I won’t repeat the tables here. The fractional version
runs $2–4K/month and covers the actual demand. If you want to know what
those hours look like in practice — what actually happens in the first
90 days — I’ve written [a
week-by-week teardown of a typical engagement](https://sublimecoding.com/blog/what-a-fractional-security-engagement-actually-looks-like).

The other reason the strategist comes first: at this stage,
**your engineers are your security engineers.** A competent
product engineer, pointed at a clearly prioritized finding with a
clearly articulated “why,” can close most of what a seed-stage company
needs closed. What they can’t do is generate the prioritization, or sit
on the customer call, or scope the audit. Buy the scarce thing; direct
the abundant thing.

And if enterprise deals are why you’re here: start with [SOC 2 as a
revenue tool](https://sublimecoding.com/blog/soc-2-is-a-revenue-tool-not-a-security-tool), because scoping that correctly is the single most
valuable decision a vCISO makes at this stage.

## When the answer is *not* a vCISO

Three cases where I’d tell you to skip the engagement I sell:

**You have a real-time attack surface from day one.** If
you’re building deep tech where the product itself is a target the
moment it ships — a wallet, an exchange, network infrastructure,
anything holding other companies’ credentials — you don’t have a
“decisions and paperwork” workload, you have an adversary. That’s an
engineering problem first. Hire the security engineer (or make your
first ten engineering hires security-strong), and get leadership
fractionally *on top of* that, not instead of it.

**You’re regulated from day one.** HIPAA, PCI-DSS at
level, banking partnerships — when the compliance floor is structural,
the work volume justifies dedicated ownership much earlier, and a
fractional leader spread across clients may not be able to carry your
audit calendar. Some can; be honest about the hours.

**You already have a security-strong staff engineer.**
Some teams get lucky: a staff-level engineer who’s carried security at a
previous company and *wants* the ownership. If that person exists
and has the appetite, give them the mandate and 20–30% of their time,
and buy them a few advisory hours a month to pressure-test the program
instead of a full engagement. That’s cheaper than me, and for the right
person it works. The failure mode to watch: the mandate quietly
evaporates the first time a product deadline hits. If it happens twice,
the experiment failed — go back to the stage table.

## Series A: the first security engineer

Somewhere past Series A the workload flips. You now have real attack
surface: more engineers shipping faster than any part-timer can review,
customer data at a volume that makes isolation failures existential,
infrastructure complex enough that hardening is a project rather than a
checklist. The security work stops being mostly decisions and starts
being mostly *engineering* — continuous, hands-on-keyboard
engineering.

That’s when the first security engineer earns their seat. And the
hire works precisely *because* the program already exists: they
walk into prioritized work with organizational context, not a blank
page. The sequencing matters more than the individual — I’ve written
about [where security
fits in the broader pre-Series-A hiring sequence](https://sublimecoding.com/blog/pre-series-a-ai-startup-hiring-plan), and the short
version is that this hire lands after your core product team is stable,
not before.

What happens to the vCISO? The engagement should *shrink* —
advisory hours, audit-cycle support, the customer calls that still need
a title. A fractional engagement that’s still running at full intensity
eighteen months after a security engineer joined is a smell: either the
engineer isn’t being given the program, or the vCISO won’t hand it over.
Graduation is the success condition, not churn.

## Series B and beyond: the full-time lead

Eventually the decisions themselves become full-time: a security
*team* to manage, quarterly board reporting, a compliance
portfolio (SOC 2 plus the customer-specific attestations that enterprise
growth accretes), incident process with real stakes, and enough spend to
need its own budget owner. That’s a full-time security lead or CISO, and
at this point the $200–400K stops being over-leveled and starts being
table stakes — the title now carries work that fills the hours.

The practical note founders miss: your first security engineer is not
automatically this person. Sometimes they grow into it; often the
builder wants to keep building. Promote for appetite, not tenure — and
if you promote, backfill the engineering, because the building didn’t
stop.

## The decision, compressed

Stage
First security money goes to
Why

Pre-seed
Neither — founder-owned hygiene
The risks are basics, and basics are defaults, not headcount

Seed → pre-Series A (enterprise sales)
vCISO / fractional leadership
The workload is decisions + representation, ~15–25 hrs/month

Regulated or adversarial from day one
Security engineer (leadership fractional, on top)
The product is a target before it has customers

Series A, real attack surface
First security engineer
The work flipped from decisions to engineering

Series B+
Full-time security lead / CISO
The decisions became full-time too

The one-line version: **buy decisions before you buy hands,
until the hands are what’s scarce.**

If you’re at the stage where the decision is live, [run your numbers through the vCISO cost
calculator](https://sublimecoding.com/tools/vciso-cost) — four questions about your stage, a monthly number out.
And if the math points at the fractional route, [this is exactly the engagement I run](https://sublimecoding.com/consulting). If it
points at the security engineer instead — good, hire them, and spend the
money you saved on making sure they walk into a program instead of a
blank page.

---

## The 17x Discount Hiding in Your AI Coding Bill

URL: https://sublimecoding.com/blog/ai-coding-agent-costs-real-numbers
Published: 2026-07-12
Tags: AI tools, engineering leadership, productivity

>
**TL;DR:** My team pays $200 per developer per month for
AI coding agents; the heaviest agentic users run second and third
accounts, so their real line item is $400–600. But the number that
should reframe your whole budget is the ratio: metered at API list
prices, the tokens each developer actually consumes would cost
**$1,800–$3,500 a month** — and the theoretical ceiling on
that same $200 plan is roughly **$8,000**. The subscription
is an arbitrage — you’re buying tokens at a 10–17x discount in practice,
up to 40x at the limit — and once you see that, three budgeting
conclusions fall out, including one about how long the discount can
last.

## The numbers are out there now — the ratio isn’t

Search for what AI coding agents cost and you’ll find two kinds of
answers. The vendor guides — [CloudZero](https://www.cloudzero.com/blog/claude-code-pricing/),
[Finout](https://www.finout.io/blog/claude-code-pricing-2026),
getDX, the rest — all orbit the same figure: about $13 per developer per
active day in enterprise deployments, $150–250 per developer per month,
90% of users under $30 a day. It’s a real number. It also traces back to
Anthropic’s own reported aggregate across enterprise customers, which
means it’s an average over a population dominated by light users — the
developer who asks for a function here and a code review there.

And lately, real numbers have started to land alongside the recycled
average. [The
Register, citing Gartner](https://www.theregister.com/ai-and-ml/2026/06/24/ai-coding-agents-could-soon-cost-more-than-the-developers-using-them/), puts serious agentic users at $2,000–5,000
per developer per month, with extreme cases near $20,000. [Practitioners
have published per-session cost breakdowns](https://hiddedesmet.com/the-real-cost-of-ai-coding-agents), and [one
team’s “$200–600 a month reality”](https://byteiota.com/ai-coding-tools-2026-the-200-600-month-reality/) matches our sticker range almost
exactly. So the raw figures are no longer a secret.

What’s still missing is the *ratio* — what a subscription seat
actually buys you against API list price, why the plan limit is a budget
governor rather than a nuisance, and what the arbitrage means for how
long the current pricing can last. That’s this post. If you’re budgeting
for a small team where some developers run *agentic* workflows —
long-lived sessions, parallel agents, the agent reading half the repo
into context before it touches anything — your usage doesn’t sit near
anyone’s average. It sits in the tail, and the tail is where your
invoice lives.

## What we actually pay

The sticker is simple: **$200 per developer per month**
for a Max-tier subscription running an agentic coding tool all day.
That’s the line item a CFO sees, and for a meaningful chunk of the team
it’s also where the story ends.

But the distribution isn’t flat, and this is the part no pricing
explainer mentions: our heaviest agentic users don’t fit inside one
subscription. Plan limits are generous for a human typing prompts; they
are not sized for a developer running multiple long agentic sessions in
parallel, every working hour. So the heavy users run a second —
sometimes a third — account, and their real cost is **$400–600 a
month**.

Notice what the plan limit actually is in this setup: it’s not a
nuisance, it’s the *budget governor*. The subscription model
converts unbounded token consumption into a bounded, predictable seat
cost, and when a developer outgrows the bound, the overage arrives as
another flat $200 — not as a surprise five-figure metered bill. Keep
that property in mind; it’s doing more work for your finance
conversation than any feature of the tool itself.

## What we actually consume

Here’s the number that reframed how I think about all of this. Price
the tokens our developers actually push through these tools at API list
rates — input tokens, output tokens, the going per-million prices — and
the consumption comes out to **$1,800 to $3,500 per developer per
month**.

Sit with the ratio for a second. The $200 seat delivers roughly nine
to seventeen times its price in list-priced tokens.

The consumption isn’t waste, either — it’s structural to how agentic
coding works. An agent doesn’t read a function; it reads the file, the
callers, the tests, and the config, and then re-reads chunks of them
every time the context compacts. Tool output — test runs, lints, diffs —
flows back in as input tokens. Parallel subagents multiply all of it.
And the expensive direction, output tokens, is precisely what a
*coding* agent produces in bulk: code, diffs, plans, retries. A
chat user’s token profile is a trickle. An agentic developer’s profile
is a firehose, and the firehose runs most of the workday. ([What that workday actually
looks like, hour by hour, I’ve written up separately](https://sublimecoding.com/blog/my-daily-agentic-ai-workflow).)

## The ceiling is higher than our bill

Our $1,800–3,500 is *observed* consumption — a working
developer’s actual day. The more revealing number is the ceiling: take
each plan’s published rate limits, assume you saturate every reset
window around the clock for a month, and price that token volume at the
vendors’ public per-million-token API rates ([Anthropic’s](https://www.anthropic.com/pricing), [OpenAI’s](https://openai.com/api/pricing/)). The approximate
theoretical maximums, as of mid-2026:

Plan
Price
Max possible spend (approx.)
Multiple

Claude Pro
$20/mo
~$400/mo
20x

Claude Max 5x
$100/mo
~$2,000/mo
20x

Claude Max 20x
$200/mo
~$8,000/mo
**40x**

ChatGPT Plus
$20/mo
~$700/mo
35x

ChatGPT Pro 5x
$100/mo
~$3,500/mo
35x

ChatGPT Pro 20x
$200/mo
~$14,000/mo
**70x**

Two things jump out of that table. First, the tiering isn’t linear:
on both vendors, the $200 plan carries **double the
multiple** of the cheaper tiers. The heaviest plans are the most
deeply subsidized, which tells you exactly who these vendors are
competing for — the saturated agentic developer, the one whose workflow
becomes load-bearing and whose team follows.

Second, our observed numbers suddenly look less extreme and more
*predictable*. A developer who saturates a Max 20x plan during
business hours — call it a third of the month’s reset windows — lands at
roughly $2,700 of API-equivalent consumption. That’s the middle of our
measured band. The heavy users running second and third accounts aren’t
anomalies; they’re just the ones whose sessions outgrow the per-window
limits before the workday ends. If your team adopts agentic workflows
seriously, this is the consumption curve you should *expect*, not
an outlier to be explained.

## The arbitrage, and the three things it tells you

Once you see the subscription as a 10–17x token discount in practice
— 40x at the ceiling — rather than a software seat, three budgeting
conclusions follow.

**Budget per seat, not per token.** The flat plan caps
your downside in a way metered API usage never will. For developer
tooling, default to subscriptions and treat raw API keys as the overflow
lane — CI jobs, automation, scripts — not the baseline. The
predictability alone is worth real money when you’re forecasting a year
of spend for a board deck.

**Don’t model an agentic initiative at API list price — you’ll
kill projects that are actually cheap.** If someone on your team
spreadsheets “adopting agents” at $1,800–3,500 per developer per month,
the project dies in the meeting. The real marginal cost is $200–600.
I’ve watched the reverse error too — teams quietly assuming token costs
make agentic development unaffordable, when the subscription pricing has
already absorbed the problem. The whole point of [measuring return honestly](https://sublimecoding.com/blog/prove-the-return) is that the
cost side has to be the *real* cost, and the real cost is the
seat.

**But never build a *product* on the subsidized
number.** Your developers consume tokens at the discounted rate;
your product’s API calls pay list price. If the firehose economics of
agentic workloads leak into your product architecture — agents
re-reading context, verbose multi-step loops — you’ll ship something
whose unit economics only worked under a subsidy you don’t get. This is
exactly why [the
telemetry layer comes before the better model](https://sublimecoding.com/blog/your-ai-product-needs-telemetry-before-better-model): you cannot manage
per-request token economics you aren’t measuring.

And a fourth, uncomfortable one: **the gap is a moment in
time.** A vendor selling $3,000 of marginal compute for $200 —
with a worst-case exposure of $8,000, or $14,000 across the street — is
making a growth-stage bet: on falling inference costs, on capacity, on
market share. Maybe inference costs fall fast enough that the discount
becomes permanent. Maybe limits quietly tighten instead — anyone who has
watched a usage policy page change knows which way that drifts. Budget
the sticker, but leave headroom in the plan for the day the multiple
compresses. If your adoption case only works at $200 a seat and
collapses at $600, it was thinner than you thought.

## A heuristic you can put in a spreadsheet

Pulling the numbers together into something a founder can actually
budget with:

- **Baseline:** $200/month per developer who uses an
agentic coding tool seriously.

- **Heavy-user multiplier:** identify your genuinely
agentic developers — on my team it’s a minority, and you already know
who yours are — and budget them at $400–600.

- **Overflow:** a metered API line for automation and CI,
small relative to seats, watched monthly.

For a concrete ten-developer team with, say, three heavy users: seven
seats at $200 plus three at $400–600 lands you at **$2,600–3,200 a
month**, call it $31–38K a year. The same consumption at API list
prices would run $18–35K *a month*. That delta is the entire
reason the budgeting conversation has gotten easy — and the reason it
deserves a re-check every couple of quarters.

## The line items that never show up on an invoice

I’d be writing the same recycled explainer I criticized if I stopped
at the subscription math, so: the token bill is the smaller half of the
real cost, and the bigger half doesn’t appear on any invoice.

Agent output has to be reviewed, and reviewed by someone senior
enough to catch the confident wrong answer — [knowing when to
trust the agent and when to step in](https://sublimecoding.com/blog/when-to-trust-an-agent-and-when-to-step-in) is itself a skill with a
learning curve you’re paying for. Some agent work gets thrown away;
that’s not failure, it’s the workflow, but it’s real hours. And the
deepest cost is that this [isn’t faster
typing — it’s a different workflow](https://sublimecoding.com/blog/ai-assisted-engineering-is-a-new-workflow), and workflow transitions cost
senior attention for a quarter before they pay rent. None of that is an
argument against the spend. It’s an argument for counting it, because
$2,600 a month in seats is trivially justified by reclaimed engineering
hours — but only if the review-and-rework side of the ledger is
honest.

## The caveats, stated plainly

This is one team’s data. Our work skews toward exactly the
long-running agentic sessions that maximize consumption; a team using AI
as fancy autocomplete will look nothing like this, and the recycled
$13/day average might actually describe them. Your heavy-user share will
differ. Your multiple will differ.

Which is the real takeaway: **measure your own
consumption.** The tooling reports token usage; pricing pages
publish list rates; the arithmetic is an afternoon. Whatever the
discount multiple turns out to be for your team, *knowing it* is
what converts AI tooling from a faith-based line item into one you can
defend in a budget review — and what tells you, early, whether the
economics still work the day the subsidy thins. The teams that measured
were also the only ones who could [prove
the return](https://sublimecoding.com/blog/prove-the-return) when someone finally asked.

The tokens are cheap right now — [even as the hardware underneath them
gets more expensive](https://sublimecoding.com/blog/ai-tax-hardware-costs). The durable advantage is knowing exactly how
cheap, and for whom.

---

## A Scared Team Is Your Biggest Attack Surface

URL: https://sublimecoding.com/blog/psychological-safety-security-control
Published: 2026-07-11
Tags: security, engineering leadership

>
**TL;DR:** Most hackers don’t break in — they talk their
way in, by scaring a low-level employee with access into handing it over
before that employee stops to think it through. That’s not a training
failure. It’s a management failure, and the manager who built it usually
built it on purpose, because a team that’s afraid of you is a team that
does what you say. The mechanism that makes that trade look cheap is the
same one social engineering exploits: fear suppresses the honest report.
You bought MFA, phishing training, and a SOC 2 badge. The cheapest
control you didn’t buy is a team that isn’t afraid to say “I clicked the
thing.” Psychological safety isn’t a culture nice-to-have sitting next
to your security stack. It’s inside it.

## The mechanism

Here’s how most social-engineering attacks that actually work get in:
not through a zero-day, but through a phone call or a message to the
person with the least power and the most access — a help-desk tech, a
support rep, a junior engineer with prod credentials nobody’s rotated.
The attacker doesn’t need to break the person’s judgment. They just need
to make the person more afraid of *not* complying right now than
of the vague, deferred consequence of complying. “Your manager needs
this reset immediately.” “IT flagged your account, act now or you’re
locked out.” Every version of that script rents the same asset: an
employee who’s learned that hesitating or escalating gets punished
faster and more reliably than getting it wrong gets caught.

That asset isn’t free. Somebody built it — and it’s almost never the
attacker. It’s whoever taught that employee, over months, that the
fastest way to avoid trouble is to comply first and think later. The
attack surface a phishing simulation measures isn’t the real one. The
real one is upstream, in whatever made “comply without asking” feel like
the safest move available.

I’ve watched a version of this pattern at almost every org I’ve been
at, under different managers. It doesn’t look like a security failure
while it’s happening. It looks like management. A manager tells the team
their jobs aren’t safe, that they “could be out on the streets any day.”
A manager stands over standup and calls the work “shit,” strong-arming
the room into moving faster. None of that reads as a security incident
in the moment. But it’s laying the exact groundwork an attacker later
gets to walk in on for free — a team trained to treat urgency and
authority as things you obey, not things you verify.

## Psychological safety is a security control

Security budgets go where security is legible: endpoint protection,
MFA enforcement, phishing simulations, the audit report someone can
point to. All of that is real and worth buying. None of it touches the
failure mode above, because that one isn’t a gap in tooling — it’s a gap
in whether an employee believes saying “I think I just clicked something
bad” gets them help or gets them blamed.

Think about what determines dwell time — how long an attacker sits
inside your systems before anyone notices. It isn’t just detection
tooling. It’s whether the first person who suspects something is wrong
says so in the next five minutes, sits on it for three days, or never
says anything because the last person who admitted a mistake got reamed
out in front of the team. A team afraid to report extends every incident
by exactly the length of that fear — detection only starts the clock
once someone tells it to.

That’s the answer to whether psychological safety improves security
reporting, and it isn’t subtle: reporting requires believing that
surfacing bad news is safer than sitting on it. Fear inverts that math.
If the honest report gets punished and the cover-up sometimes works,
people bet on the cover-up, rationally, given what you’ve taught them
about the odds.

## The unpredictable manager is the bug

The specific thing that breaks a team isn’t a strict manager — a
known, consistent bar is something people can plan around. What breaks
people is a manager whose reaction they can’t predict, where the same
mistake gets a shrug on Tuesday and a public dressing-down on Friday
depending on mood. There’s no safety to admit anything to someone like
that, so people stop admitting. They start managing the manager instead
of managing the work — which mostly means minimizing the moments where
the manager notices them.

There’s an old idea about this, phrased as a ruler’s choice: if I
can’t be loved, I’ll be feared. It sounds like a strategy; it’s actually
a shortcut. Fear installs faster than trust and produces compliance
almost immediately, which is exactly why it’s tempting under deadline
pressure. But it’s a loan, not a purchase, and the interest comes due as
a Julius Caesar death — stabbed by your own people, the ones closest to
you, the ones with the most access and the least reason left to protect
you. A scared team doesn’t stay loyal under pressure. It hides bad news
and hands over the keys to whoever asks with enough apparent authority,
because that stranger is, in the moment, less frightening than the
manager they already have.

Panicked people do stupid things, and panic is a renewable resource a
bad manager keeps generating. I’ve written before about the alternative
— [owning the outcome
instead of managing the optics](https://sublimecoding.com/blog/professional-owns-the-outcome) means the manager sits with a miss in
daylight instead of outsourcing that discomfort to whoever’s easiest to
yell at. A team led that way doesn’t need fear, because the bar is
legible and the miss gets handled by the person actually accountable for
it.

## Blameless postmortems are incident-response infrastructure

Security already built the fix for this, mostly by accident, in the
blameless postmortem. The pitch usually gets sold as a kindness — don’t
be mean to the on-call engineer, it’s not their fault. That undersells
it. Blameless isn’t soft; it’s the fastest path to the truth, and speed
is the whole game in incident response.

A postmortem culture that hunts for who to blame teaches the same
lesson the fear-based manager teaches: minimize what you admit, protect
yourself first. That instinct doesn’t stay contained to postmortems —
it’s the same one that makes someone sit on a suspicious email for two
days instead of reporting it in the first five. [An agent postmortem needs the same
discipline pushed one step further](https://sublimecoding.com/blog/ai-agent-postmortem) — you can’t even blame the tool,
because blaming anything for the proximate mistake just teaches people
to hide the next one. Curiosity over blame isn’t a values statement on a
wiki page. It’s what makes the person closest to the anomaly say
something *fast*, while it’s still cheap to fix.

## In practice

The instinct that follows fear-based management is usually
surveillance: keyloggers, screenshot tools, mouse-jiggler detection. I
don’t run any of that, on principle — not because people never slack
off, but because surveillance like that signals exactly one thing: *I
don’t trust you*. Nobody responds to that by resolving to earn trust
back; they respond by learning what the monitoring measures and
satisfying that number instead. People route around surveillance.
Chair-time was never the same thing as output.

We’re all adults. The alternative to policing activity is measuring
outcomes, honestly, and giving people real chances before you act on a
miss — a basket of signals, not one metric: throughput relative to
peers, adjusted for task size, repeated misses against timeframes the
person agreed to after an explicit chance to course-correct. If a task I
know takes two days isn’t done after a week, that’s a real signal worth
a real conversation, not a snapshot of one bad afternoon turned into a
verdict. I’ve had to make calls like this — adding bandwidth where
someone genuinely needed it, and eventually letting go of people whose
performance stayed short of the bar after a fair shot to close the gap.
Framed as outcomes, not intimidation, that’s not a contradiction of
psychological safety; it’s what makes the safety credible. People trust
a bar that’s real and applied consistently a lot more than they trust a
boss who says “we’re a family” and then explodes without warning.

That same preference for real chances over surveillance is why [the muscle you actually want
on a senior team is the willingness to fail somewhere safe first](https://sublimecoding.com/blog/stop-making-senior-engineers) —
you don’t grow the judgment that catches a social-engineering attempt by
punishing every near-miss into silence. You grow it by making it cheap
to say “I almost got got” out loud.

## The vCISO close

When I run a security review, culture is in scope. Not as a soft
add-on next to the firewall rules — as an actual attack-surface
question, because a review that only checks tooling and skips the org’s
fear gradient has audited half the system. A fractional CISO who never
asks “what happens to the person who reports a mistake here” hands you a
report that says you’re covered, right up until someone on your team
gets a convincing phone call and hands over access rather than risk
being wrong in front of the wrong manager.

None of this is security-specific, either — it’s one face of the [broader operating system I
run teams on](https://sublimecoding.com/blog/trust-is-the-operating-system), where trust is the thing everything else is downstream
of.

If you want a second set of eyes on where your org’s actual attack
surface sits — tooling, process, or the fear gradient underneath both —
[that’s what a fractional security engagement is
for](https://sublimecoding.com/consulting).

---

## What Cyber Insurance Underwriters Ask AI Startups

URL: https://sublimecoding.com/blog/cyber-insurance-ai-startups
Published: 2026-07-10
Tags: security, AI startups, founders

>
**TL;DR:** The cyber insurance application that lands on
your desk is not an insurance document. It’s a security audit in
disguise — the same MFA, EDR, backup, and incident-response questions an
enterprise security review or a SOC 2 auditor will ask you, rephrased by
an actuary. That’s good news if you treat it that way: do the controls
work once and insurance, compliance, and enterprise sales all get
cheaper together. It’s very bad news if you treat it as paperwork,
because a checkbox you answered optimistically can void the whole policy
— carriers have already rescinded coverage over a misrepresented MFA
answer. And if you’re an AI company there’s a new layer: insurers are
quietly carving AI out of some policies while affirmatively covering it
in others, and the difference is in endorsement language nobody reads
until the claim is denied.

## Why this questionnaire is on your desk at all

Almost no pre-Series-A founder wakes up wanting cyber insurance. The
questionnaire shows up for one of three reasons: an enterprise
customer’s procurement team made coverage a contract condition, an
investor’s term sheet or board asked for it, or a compliance framework
you’re chasing listed it as an expected control. In other words, it
arrives the same way [SOC 2 arrives —
as a revenue gate, not a security initiative](https://sublimecoding.com/blog/soc-2-is-a-revenue-tool-not-a-security-tool).

That origin matters, because it tells you how to size the decision.
The question is almost never “do we philosophically believe in risk
transfer.” It’s “this $180K contract requires $2M in cyber liability
coverage — what’s the cheapest path to a policy that will actually pay
out.” Both halves of that sentence deserve attention. Founders reliably
optimize the first half and skip the second, and the second is where the
trap is.

## The application is a security audit with a premium attached

Ten years ago, cyber insurance applications were a page of revenue
questions. Today, underwriting has [quietly
become a technical audit](https://emergeits.com/blog/cyber-insurance-requirements-underwriting-has-quietly-become-a-technical-audit/). The 2026 application asks, in some order,
about four control families — and the bar for each has moved from “do
you have it” to “prove it”:

**MFA, everywhere that matters.** Not “do you offer MFA”
— *is it enforced* on email, VPN or remote access, cloud admin
consoles, and every privileged account. [Underwriters
treat available-but-optional MFA as a no](https://seedpodcyber.com/what-underwriters-look-for-cyber-insurance-application/), and surveys of carrier
requirements put enforced MFA on [effectively
every application in 2026](https://basg.co/blog/cyber-insurance-requirements-2026-what-insurers-demand). For a ten-person startup living in Google
Workspace, GitHub, and AWS, this is a config afternoon, not a project.
Do it before you apply, not in the two weeks after the incident.

**EDR on endpoints and servers.** Carriers ask what
endpoint protection you run, and [legacy
antivirus no longer counts](https://www.cyberduo.com/blog/cyber-insurance-renewal-denied-2026-checklist/) — the expectation is a real EDR agent,
increasingly with someone watching the alerts. For a small team this is
one of the few line items that costs actual money, and it’s also one of
the first things an enterprise security reviewer asks about, so you were
buying it anyway.

**Backups you have restored, not backups you have.** The
questions now cover frequency, immutability, and — the part teams fail —
[evidence
of a tested restore](https://www.prescientsolutions.com/cyber-insurance-requirements-2026-smb-checklist/). “We have backups” answered honestly means “we
have run a restore and it worked, and we can show the log.” If you
can’t, the honest answer is no, and you should go earn the yes before
you sign.

**A written incident response plan.** Underwriters ask
whether you have one and, increasingly, whether you’ve [exercised
it in the last year](https://www.cyberduo.com/blog/cyber-insurance-renewal-denied-2026-checklist/). At pre-Series-A this doesn’t need to be a
40-page binder — it needs to be a real document that names who declares
an incident, who talks to customers, and who calls the carrier’s breach
hotline (that last one is in the policy for a reason: call them first,
or you may find “unapproved vendor costs” aren’t covered).

Notice what that list is. It’s the same list [an AI-native
security program](https://sublimecoding.com/blog/running-security-at-an-ai-native-company-2026) starts with, and the same list on every enterprise
security questionnaire you’ll ever receive. The insurance application is
not extra work; it’s a third buyer for work you already owe two other
people. That’s the reframe that makes this whole topic cheap: build the
controls once, then let the auditor, the procurement reviewer, and the
underwriter each read their copy of the same evidence.

## The checkbox that voids the policy

Here’s the part the broker’s glossy PDF underplays. A cyber policy
application is a legal representation, usually signed by an officer of
the company. Answer a control question wrong — even sloppily rather than
maliciously — and the carrier has grounds to rescind the policy
*after* the incident, when you need it most.

This isn’t theoretical. In 2022, Travelers sued its own insured,
International Control Services, after a ransomware attack revealed the
company had [attested
to MFA it wasn’t actually using](https://www.insurancejournal.com/news/national/2022/07/12/675516.htm). The application had been signed by
the CEO and the person responsible for network security. The outcome:
ICS agreed to a judgment [rescinding
the entire policy](https://www.insurancejournal.com/news/national/2022/08/30/682564.htm) — not a denied claim, a policy treated as if it
never existed. Brokers now cite the case as the reason to [answer
application questions with care](https://global.lockton.com/us/en/news-insights/travelers-v-ics-underscores-need-to-respond-carefully-to-cyber-insurance), and carriers have kept raising
misrepresentation issues on incorrect application answers since.

The founder-brain failure mode here is exactly the one that fills out
enterprise security questionnaires optimistically: “we basically have
MFA, we’re rolling it out, close enough — check.” On a sales
questionnaire that costs you credibility. On an insurance application it
costs you the coverage. If the true answer is “partially,” say
partially, or fix it first. A policy bought with a wrong checkbox is
paper, not coverage — you’re paying premiums for a document the carrier
can walk away from precisely when it’s expensive for them to stay.

## The AI layer: covered, excluded, and silently ambiguous

If you’re building an AI product, there’s a second reading
assignment, because the insurance industry is actively redrawing lines
around AI right now and your policy sits on one of them.

Three things are happening at once:

**General liability and E&O are adding AI
exclusions.** In January 2026, ISO — the organization whose
standard forms most carriers build on — issued [three
new generative-AI exclusion endorsements](https://phl-firm.com/generative-ai-insurance-exclusions-2026/) for commercial general
liability, barring coverage for harms tied to generative-AI outputs:
defamatory content, IP infringement in generated material, damages
traceable to AI-driven errors. On the professional-liability side, at
least one carrier has filed an [absolute
AI exclusion](https://adversa.ai/blog/ai-risk-management-insurance-what-the-new-exclusions-mean/) for D&O and E&O lines covering “any actual or
alleged use, deployment, or development of Artificial Intelligence.” For
a company whose entire product is AI, an exclusion with that language is
close to an exclusion of the company.

**Cyber policies, by contrast, are mostly affirming
AI.** The cyber line has moved the other way: carriers are [affirming
coverage for AI-driven attacks](https://www.businessinsurance.com/insurers-brokers-adjust-as-ai-exclusions-emerge/) rather than excluding them. Coalition
added an affirmative AI endorsement clarifying coverage for AI-related
security events — including deepfake-driven fraudulent instructions —
and later folded it into its base policy. So the attack surface where AI
is the *weapon pointed at you* is increasingly explicitly
covered.

**A new product category is forming for the gap in
between.** Harms caused by *your model’s own behavior* —
hallucinations, model underperformance, output errors — are the gap
classic cyber never contemplated, and standalone products like [Armilla’s
Lloyd’s-backed AI liability policy](https://www.lexology.com/library/detail.aspx?g=b76e0dba-d9a8-44f1-9f5d-6fbd0a22f6b6) and Munich Re’s aiSure are
emerging to cover exactly that. Most pre-Series-A companies don’t need
these yet. You need to know they exist so you stop assuming your cyber
policy covers what they cover.

The practical takeaway is a reading exercise, not a buying exercise:
when the quote comes back, find the AI language. Is there an AI
exclusion, an affirmative AI endorsement, or silence? Silence is the
worst answer, because silent-AI ambiguity gets resolved in court after
the claim. And expect the application itself to grow AI-specific
questions — which third-party LLM providers you depend on, what customer
data reaches them, whether customer data trains models. Those are the
same questions [enterprise
security reviews already ask AI vendors](https://sublimecoding.com/blog/running-security-at-an-ai-native-company-2026), which is one more argument
for having real answers instead of aspirational ones.

## What it actually costs, and how founders get the sizing wrong

The good news: at startup scale this is one of the cheaper line items
in the security budget — security *leadership* is the expensive one, and the
[vCISO cost calculator](https://sublimecoding.com/tools/vciso-cost) sizes that half. Market surveys put small-business cyber premiums
around [$1,500–1,800
a year for $1M in coverage](https://www.insureon.com/small-business-insurance/cyber-liability/cost), with startup-specific estimates in the
[$750–2,500
range for basic coverage](https://blog.dazzee.com/how-much-is-cyber-insurance-for-startups) and typical policy limits running [$1M
to $5M](https://benzeninsurance.com/cyber-liability-insurance-cost-complete-2025-guide-for-small-businesses/). Your actual number moves with how much sensitive data you
hold and how good your controls look on the application — the same
controls above, which is why doing them first isn’t just integrity, it’s
a discount.

The two sizing mistakes I see, in both directions:

**Underbuying by accident:** taking the $1M minimum
because it’s the default, when the enterprise contract that triggered
the purchase explicitly requires $2M or names coverage types (breach
response, business interruption, tech E&O) the cheap policy omits.
Read the insurance clause in the customer contract *before* you
get quotes; it’s the actual spec.

**Overbuying from fear:** a five-person pre-revenue team
with no meaningful customer data buying a $5M tower plus riders because
an incident headline scared the board. If no contract requires it and
you hold nothing worth breaching yet, the premium is usually better
spent on the controls themselves — the [prioritization math is the same
as the security-hiring math](https://sublimecoding.com/blog/vciso-math-for-ai-founders): spend where the risk actually is at
your stage, not where the vendor’s fear content points.

The honest decision rule: buy when a contract, a board, or real
data-holding demands it; size to the strictest contract you’re signing;
and never buy coverage as a *substitute* for the controls,
because the application will force the controls anyway and a policy
without them is rescindable paper.

## Do the work once

Strip the actuarial wrapper off and the cyber insurance application
is the third appearance of the same exam: enforced MFA, real endpoint
protection, tested backups, a written IR plan, straight answers about
your AI supply chain. The SOC 2 auditor asks it with a different rubric.
The enterprise procurement reviewer asks it in a spreadsheet with 240
rows. The underwriter asks it with a premium — and, as ICS found out, a
rescission clause — attached.

Companies that treat those as three separate fire drills do the work
three times, badly, under three deadlines. Companies that build the
controls once, with evidence, answer all three from the same folder.
That’s the whole trick, and it’s very achievable at a five-person
company — most of the list is configuration and documents, not
headcount.

If a customer contract just made coverage urgent and you want the
application answered truthfully *and* quickly — controls stood up
in the right order, evidence that satisfies the underwriter and the next
security review in one pass — [that’s exactly the
kind of engagement I run](https://sublimecoding.com/consulting).

---

## Building an AI Memory That Outlives Any Single Agent

URL: https://sublimecoding.com/blog/ai-memory-outlives-any-agent
Published: 2026-07-06
Tags: AI, agents, AI tools

>
**TL;DR:** Every hour you spend teaching an agent your
codebase, your preferences, your project history is an investment — and
right now, most people are making that investment inside one vendor’s
proprietary memory format. Nobody knows who wins the model race, and the
leaderboard has already changed hands more than once this year. The
durable move is to keep your knowledge as plain, git-versioned files
that live outside any single tool, and let the agent’s own config be a
thin pointer into that vault instead of the vault itself. Any agent can
pick it up cold. Zero ramp-up, regardless of who’s winning when you
switch.

## Nobody knows who wins, and that’s the point

I’m not going to pretend I know whether Anthropic, OpenAI, Google, or
someone that hasn’t been founded yet ends up with the durable lead in
agentic AI. Nobody credible is claiming that with a straight face right
now. The frontier leaderboard has changed hands repeatedly over the last
eighteen months — Stanford’s 2026 AI Index puts it plainly: “U.S. and
Chinese models have traded places at the top of performance rankings
multiple times since early 2025” ([Stanford
HAI, 2026 AI Index Report](https://hai.stanford.edu/ai-index/2026-ai-index-report/technical-performance)). That’s the churn at the level tracked
across countries — inside just the U.S. labs, the same reshuffling
happens lab-by-lab, with different vendors leading different benchmarks
in the same quarter, let alone the same year.

That instability isn’t really the risk — it’s normal at this stage of
the race. The risk is what most people building agent workflows haven’t
priced in: while the leaderboard churns, you’ve been quietly
accumulating months of project context, coding preferences, and
institutional history *inside one vendor’s tool*. Every session
you run teaches an agent something about how you work. That accumulated
context has real value — and if it’s trapped in a proprietary chat
history or a tool-specific config format, its value is hostage to that
vendor’s roadmap, not yours.

This isn’t hypothetical vendor-switching anxiety. It already
happened, on a real timeline, to a lot of people. OpenAI retired GPT-4o,
GPT-4.1, GPT-4.1 mini, and o4-mini from ChatGPT this February, following
a sunset schedule the company set on its own clock, not the user’s ([OpenAI
Help Center, “Retiring GPT-4o and other ChatGPT models”](https://help.openai.com/en/articles/20001051-retiring-gpt-4o-and-other-chatgpt-models)). If your
accumulated memory lived inside conversations pinned to one of those
models, or inside a tool whose whole interaction model assumed that
model would always be there, you didn’t choose when to migrate. The
vendor did.

## What you’ve actually built, and where it’s sitting

Think about what a working agent relationship accumulates after a few
months of real use. It’s not just “the AI knows my codebase.” It’s:

- Project-specific operational trivia — the port conflict, the flaky
test, the deploy quirk — that took real sessions to surface and
correct.

- Your standing preferences: how you like code reviewed, what you
never want auto-committed, which shortcuts you’ve explicitly banned
after they bit you once.

- A running history of decisions and their reasoning, so the agent
stops re-litigating settled questions.

- The connective tissue between your projects — which pattern you use
where, and why one repo does something differently than another for a
documented reason.

I’ve written before about the first two layers of this — a
`CLAUDE.md` as [the
onboarding doc you never wrote for humans](https://sublimecoding.com/blog/claude-md-onboarding-doc), and the larger structured
knowledge base behind it, [the three-layer wiki
architecture](https://sublimecoding.com/blog/build-an-llm-wiki-for-your-agent) of immutable sources, model-written synthesis, and an
index. Both of those posts are about the *shape* of good agent
memory. This one is about a question neither of them answers:
**where does that memory actually live, and what happens to it the
day you switch tools?**

For a lot of setups right now, the honest answer is: inside the tool.
A vendor’s chat history that isn’t exportable in a form another agent
can consume. A config format specific to one CLI. A “memory” feature
marketed as magic that’s really a database row you can’t
`git clone`. None of that is a bug in any one product — it’s
just what happens by default when a tool’s job is to keep you inside
it.

## The failure mode: your memory, hostage to someone else’s roadmap

Here’s the scenario that should worry you more than it currently
does. You’ve spent six months building up real context inside an agent
tool — codebase quirks, project history, your accumulated preferences.
Then one of three things happens, on the vendor’s timeline, not
yours:

- **Pricing changes.** The tier you’re on gets more
expensive, or the usage model shifts in a way that makes your workflow
uneconomical.

- **The model gets deprecated or the product gets
sunset.** Not hypothetical — see above. OpenAI’s own 2026
deprecation wave touched everything from legacy GPT models to the
Assistants API, on schedules the company set unilaterally.

- **Someone else ships a genuinely better agent**, and
staying put starts costing you the productivity gap between what you
have and what’s now available.

None of those are hypotheticals — vendors make calls like this
constantly, and each one is a completely reasonable business decision
for them to make. The problem is entirely on your side of the
relationship: if your accumulated context only exists as that vendor’s
proprietary artifact, “switch to the better tool” now means “rebuild
months of context from scratch,” and that switching cost is exactly the
thing that keeps people locked into a worse tool past the point it makes
sense. That’s not a hypothetical about AI — it’s the same lock-in
economics that’s played out with every walled-garden platform before
this one, just compressed into a market that changes leaders every few
months instead of every few years.

You don’t need to predict who wins the model race. You just need your
knowledge base to not care who does.

## The alternative: a memory layer that lives outside any vendor’s walls

The fix isn’t complicated, and it isn’t new — it’s the same
discipline that’s always protected people from platform risk: keep the
asset in a format you own, and treat any single tool as a
*client* of that asset, not its owner.

Concretely, that means:

- **Plain, structured files** — markdown and JSON,
nothing proprietary, nothing that needs a specific app to open. Any
agent that can read a filesystem can read your memory.

- **An index file** that catalogs what’s there, so any
agent — not just the one you trained — can orient itself in one read
instead of scanning everything.

- **Git for versioning**, because you want history,
diffs, and the ability to roll back a bad edit, and because git is
itself a portable, vendor-agnostic standard that predates every AI tool
by two decades and will outlive most of them.

- **The whole thing living outside any single vendor’s
application** — not inside a chat history, not inside a
tool-specific database, not inside a format only one CLI parses.

The agent’s own config file — a `CLAUDE.md`, an
`AGENTS.md`, a system prompt, whatever your particular tool
calls the thing it reads on session start — stops being the store of
record and becomes a **thin adapter**. Its job shrinks to:
here’s who I am, here’s where the real memory lives, go read it. That’s
a much smaller surface to lose if you switch tools, because the valuable
part was never in there to begin with.

This is the same instinct that shows up anywhere lock-in risk
actually gets managed well — a thin integration layer over a third-party
dependency, so the dependency can be swapped without touching everything
built on top of it. Same principle, applied to your own accumulated
knowledge instead of a vendor API.

## Why the layered structure matters

A single flat folder of notes rots. That’s true whether the reader is
human or an agent, and I’ve gone deep on the mechanics of a healthy
layered wiki elsewhere — [the three layers, the
ingest/query/lint loop, the rules that keep it from turning into a junk
drawer](https://sublimecoding.com/blog/build-an-llm-wiki-for-your-agent). For the portability question specifically, the layering
matters for a slightly different reason: **different kinds of
memory have different shelf lives, and mixing them is what makes a vault
stop being trustworthy.**

Roughly, the useful split is:

- **Durable facts** — things true regardless of which
project you’re on or which tool is reading: your standing engineering
preferences, security defaults you never compromise on, how you like
work reviewed. These barely change and should be readable by literally
anything.

- **Project-specific state** — the operational trivia and
history for *this* codebase or *this* engagement. Changes
often, needs a clear “last updated” signal, and needs to be scoped so it
doesn’t bleed into unrelated projects.

- **Pointers to external systems** — a link into a ticket
tracker, a reference to a running service, a note that says “the real
detail is over there.” These are explicitly *not* duplicated into
the vault; the vault just knows where to look.

Collapse these into one undifferentiated pile and you get exactly the
failure mode that kills most personal knowledge bases: nobody can tell
what’s still true, what’s stale, and what was only ever relevant to a
project that ended eight months ago. Keep them separated and dated, and
the vault stays something you can actually trust years in, not just the
week you built it.

## Rehydration: any agent, same knowledge, zero ramp-up

The last piece is the mechanism that makes portability actually feel
portable instead of theoretical: something that automatically loads the
right slice of the vault at the start of a session, regardless of which
agent is running.

In practice this is usually a session-start hook, or the equivalent
your tool supports — a small script that runs before the agent’s first
real turn and pulls in the durable facts plus whatever project-specific
state is relevant to the current directory or task. The mechanism is
deliberately boring: read a few files, inject them into context, done.
The interesting part is what it implies. Because the hook is reading
plain files from a vault you own, it doesn’t care whether today’s
session is running one vendor’s CLI or a competitor’s — the vault
doesn’t know or care which agent showed up to read it, and neither
should you.

That’s the actual test of whether you’ve built this correctly: hand
the same vault to a different agent tomorrow and see how long it takes
to get productive. If the answer is “immediately, because it just reads
the same files,” you’ve built a memory layer. If the answer is “I’d have
to re-explain everything,” you’ve built a chat history, and it’s only
ever going to be as durable as the app it lives in.

## This is insurance, not paranoia

None of this requires you to predict the outcome of the model race,
hedge across three vendors simultaneously, or treat every tool with
suspicion. That’s not the ask, and it’s not how I run my own setup
either. Use whichever agent is best for the job today — the field
reshuffles too often to do otherwise.

The only discipline this actually requires is a habit, not a hedge:
when an agent learns something worth keeping, it goes into files you
own, not into a store only that tool can read. That’s a small tax paid
continuously, and it buys you the thing that actually matters — the
freedom to move to whatever’s better next quarter without paying for it
twice. You don’t need to guess who wins. You need your knowledge base to
not care who does.

If you’re standing up agentic workflows across a team and want a
second opinion on where the real risk sits — memory portability,
credential handling, or the security posture underneath the whole thing
— [that’s the kind of engagement I run](https://sublimecoding.com/consulting).

---

## Prompt Injection Defense for a 5-Person AI Startup

URL: https://sublimecoding.com/blog/prompt-injection-defense-ai-startup
Published: 2026-07-03
Tags: security, AI startups, AI

>
**TL;DR:** Almost everything written about prompt
injection is either an academic paper or an enterprise vendor pitch, and
neither is addressed to you: a five-person team shipping an LLM product
with no security hire. Here’s the operator version. You cannot filter
your way out of prompt injection — there is no patch, because the attack
input and the legitimate input arrive in the same channel. What you
*can* do is make injection unprofitable: keep Simon Willison’s
“lethal trifecta” (private data + untrusted content + external
communication) from ever coexisting in one agent context, shrink each
tool to the least privilege it needs, validate what comes *out*
of the model instead of trusting what goes in, and put a human gate in
front of anything state-changing. That’s roughly a week of work, most of
it deletion. And skip the guardrail-model arms race, the red-team
retainer, and the WAF-for-prompts SaaS — at your stage they’re
theater.

## The advice is written for teams you don’t have

Search for prompt injection defense and page one splits into two
camps. Camp one is research — taxonomy papers, benchmark suites, formal
definitions of indirect injection. Genuinely useful if you’re building a
guardrail model; useless if you’re trying to decide what to ship
Thursday. Camp two is enterprise vendors selling AI firewalls, red-team
platforms, and posture dashboards to companies with a CISO, a security
engineering team, and a procurement process.

Nobody is writing for the team that actually ships most new AI
products: five people, one of whom is “the security person” the way
someone is “the DevOps person” — by default, on top of a full product
job. That’s the team I work with as a fractional security lead, so
that’s who this post is for.

The good news: at five people you have an advantage the enterprise
doesn’t. Your attack surface is small enough to actually reason about,
and nobody has to file a ticket to delete a tool from the agent. Most of
what follows is subtraction.

## What you’re actually defending against

One paragraph of theory, because the mental model matters more than
the vocabulary. Prompt injection is not SQL injection with different
syntax. SQL injection had a real fix — parameterized queries separate
code from data, done. LLMs have no such separation: the system prompt,
the user’s request, and the contents of whatever document or webpage or
email the model just read all arrive as the *same kind of thing*
— tokens in a context window. Any text the model reads is text that can
instruct it. That’s why [OWASP’s
Top 10 for LLM applications](https://owasp.org/www-project-top-10-for-large-language-model-applications/) has ranked prompt injection as LLM01 —
the number one risk — since the list existed, and why it hasn’t moved:
it isn’t a bug that gets patched, it’s a property of the
architecture.

The cleanest way to think about when this property becomes an
*incident* is [Simon
Willison’s lethal trifecta](https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/): an agent with access to **private
data**, exposure to **untrusted content**, and the
ability to **communicate externally**. Any two of those is
survivable. All three in one context means a single poisoned input can
read your secrets and mail them out — no exploit code, no CVE in
*your* stack, nothing for a scanner to find.

This is not hypothetical. [EchoLeak (CVE-2025-32711)](https://arxiv.org/abs/2509.10540)
was a zero-click exfiltration chain in Microsoft 365 Copilot: a crafted
email, invisible to the human reading it, instructed Copilot to pull
sensitive data from the victim’s context and leak it through an image
URL. Microsoft rated it CVSS 9.3. That happened to a company with one of
the largest security organizations on the planet, because their agent
held all three legs of the trifecta at once. Your copy of the same
architecture doesn’t get a pass because you’re small — it gets *less
attention*, which is not the same thing.

## Day one: map your trifecta exposure

Before you build anything, spend an afternoon on inventory. For every
place your product calls a model, write down three columns: what private
data can reach the context (docs, tickets, emails, DB rows, other users’
content), what untrusted content can reach the context (anything a user
or the open internet can influence — including your own RAG corpus if
users can write to it), and what the model’s output can *cause*
(tool calls, links rendered to users, emails sent, rows written).

Most five-person teams find the same two surprises. First, the RAG
pipeline is an injection channel — if users can upload documents that
other users’ sessions later retrieve, user A is writing into user B’s
context, and your “input sanitization” on the chat box never sees it.
Second, rendered output is an exfiltration channel — if the model can
emit markdown images or links pointing at attacker-controlled domains,
it can smuggle data out one URL parameter at a time. That’s the EchoLeak
pattern, and it applies to any chat UI that renders model output as rich
content.

The deliverable from day one is not a document. It’s a list of places
where all three trifecta legs coexist. Each one is a design bug, and the
fix is architectural: split the agent, drop a capability, or gate the
action. Which brings us to the actual work.

## Cut the tool surface before you filter the prompts

The instinct is to add a filter in front of the model. Resist it —
the highest-leverage work is on the other side: making the model
*incapable* of doing damage even when it’s fully compromised.
Assume the model is an enthusiastic intern who believes everything they
read. You don’t fix that by screening the intern’s mail. You fix it by
not giving the intern prod credentials.

Concretely, for a week-sized effort:

**Scope every tool to the session’s user.** The
retrieval tool takes the authenticated user’s ID from *your*
code, not from a model-supplied argument. If the model can pass
`user_id` as a parameter, an injected prompt can pass someone
else’s. The single most common injection-adjacent bug I see in
small-team codebases isn’t exotic — it’s a tool signature that trusts
the model to say whose data to fetch.

**Give tools the narrowest verb that does the job.** A
`query_orders(user_id)` tool that your backend curries down
to the current user beats a `run_sql(query)` tool by such a
margin that it’s barely the same category of software. If any tool in
your agent accepts raw SQL, shell strings, or arbitrary URLs, that tool
is the incident report.

**Separate read contexts from write contexts.** If the
assistant that summarizes untrusted documents is the same agent instance
that can send emails or hit webhooks, you’ve built the trifecta on
purpose. Run untrusted-content processing in a context whose only output
is text returned to *your* code — no tools, or read-only tools —
and let a separate, clean context own anything that acts. The
orchestration cost is an extra model call. The security property it buys
is the whole ballgame.

## Validate outputs, not just inputs

Input filtering is where everyone starts and where attackers are
already waiting — obfuscation, encoding tricks, translation, and
payloads split across documents all sail past regexes and phrase
blocklists. Output validation is less fashionable and much more durable,
because it checks the thing you actually care about: what’s about to
happen.

Three checks, each about a day:

**Validate tool calls structurally.** Every tool
argument gets schema-checked and policy-checked in your code before
execution — is this argument type-valid, is it within this user’s scope,
is this action allowed in this state? Boring, deterministic, and it
works exactly as well on day 1,000 as day one.

**Sanitize rendered output.** Strip or proxy images in
model output; allowlist link domains; never render raw HTML from a
model. This single control kills the most practical exfiltration path a
chat product has.

**Detect data that shouldn’t leave.** A dumb,
deterministic scan of outbound model responses for things shaped like
secrets — API keys, connection strings, your own internal hostnames —
catches the embarrassing failures cheaply. It’s not sophisticated.
Neither are most incidents.

## Put a human gate on anything state-changing

I’ve written before about [the autonomy
ladder — when to trust an agent and when to step in](https://sublimecoding.com/blog/when-to-trust-an-agent-and-when-to-step-in), and prompt
injection is the sharpest version of the argument. An agent that can be
talked into things by its own inputs does not get to take irreversible
actions unsupervised. Sending money, deleting records, emailing third
parties, changing permissions: those actions get a confirmation step
where a human sees *what* is about to happen in plain language —
not “the agent wants to proceed,” but “this will email invoice.pdf to
billing@vendor-you-dont-recognize.example.”

Founders push back that this breaks the magic-agent demo. It does,
slightly. But the demo where your agent wires data to an attacker
because a PDF told it to is worse, and at five people you do not have
the incident-response capacity to treat that as a learning experience.
Approval gates are also the control that lets you ship *more*
agent capability later: every gate you can eventually remove is a rung
of autonomy you earned with logs instead of hope.

## Why the LLM-judge guardrail can’t be your primary defense

The tempting product on the shelf is the guardrail model — a second
LLM that inspects inputs or outputs for injection. Use one if you like,
but understand what you’re buying: a probabilistic filter in front of a
probabilistic system. It fails a few percent of the time on a good day,
novel attack phrasings roll off faster than vendors retrain, and — the
part that should bother you — the judge model reads the same untrusted
text and is itself injectable. EchoLeak walked straight past Microsoft’s
own cross-prompt injection classifier on the way to the data.

A detector that’s 97% effective sounds great until you notice
attackers get unlimited retries at near-zero cost. Against a motivated
adversary, “usually catches it” is a speed bump. The deterministic
controls above — scoped tools, output validation, approval gates — don’t
have a bypass rate. They’re either enforced or they aren’t, and that’s a
property a five-person team can actually maintain. Layer a detector on
top *after* the deterministic floor exists, as signal and
rate-limiting, not as the wall.

## What to deliberately not build

Half of security at your stage is declining work confidently. Skip,
for now: the prompt-WAF subscription (it’s a detector with a dashboard —
see above); a quarterly red-team retainer (a competent one costs more
than your entire security budget and will tell you to do the things in
this post); fine-tuning for adversarial robustness (real research area,
not a startup Tuesday); and building your own injection benchmark suite
(run one of the open ones twice a year and move on).

None of these are bad ideas at 50 people with a security hire and
something to lose that’s worth the spend. At five, every one of them
displaces the week of unglamorous work that actually changes your
exposure.

## Where this fits in the bigger picture

Prompt injection defense is one layer of the four-layer stack I laid
out in [how
I’d run security at an AI-native company in 2026](https://sublimecoding.com/blog/running-security-at-an-ai-native-company-2026) — alongside agent
credentials, secrets handling, and audit logging — and it’s the layer
enterprise security reviews now probe first, because buyers have read
the same incident reports you just did. Get the week of work done and
you’re not just safer; you have a concrete, honest answer for the
security questionnaire that would otherwise stall your first enterprise
deal.

If you want a second pair of eyes on where your product sits on the
trifecta map — or you’re staring down a security review and need the
whole stack, not just this layer — [the vCISO math post](https://sublimecoding.com/blog/vciso-math-for-ai-founders) covers
what fractional security leadership costs at your stage, and [this is the engagement I run](https://sublimecoding.com/consulting).

---

## Build an LLM Wiki Your Coding Agent Actually Reads

URL: https://sublimecoding.com/blog/build-an-llm-wiki-for-your-agent
Published: 2026-06-29
Tags: AI tools, agents, productivity

>
**TL;DR:** Andrej Karpathy’s “LLM wiki” idea is just a
knowledge base written for a reader who is a model, not a person. You
don’t need a vector database to start — you need a directory of markdown
with three layers (immutable sources, model-written synthesis, and an
index), three operations (ingest, query, lint), and two rules that stop
it from rotting. Here’s the architecture I run, generically, and the
discipline that keeps it useful past the first week.

Andrej Karpathy described keeping a personal “LLM wiki” — a knowledge
base he writes and curates so that a model can read it. The framing
stuck with a lot of people because it inverts the usual question. We
spend enormous effort making models read *our* documents. The
wiki asks the opposite: what does a document look like when its primary
reader is an agent?

I’ve argued before that [your
CLAUDE.md is the onboarding doc you never wrote](https://sublimecoding.com/blog/claude-md-onboarding-doc) — the file the agent
reads at the start of every session. That post is about the front door.
This one is about the house behind it: the larger, structured knowledge
base the agent reaches into when the answer isn’t on the welcome mat. If
CLAUDE.md is “here’s how we work,” the wiki is “here’s everything we’ve
learned, indexed so you can find it.”

You can build the whole thing with a text editor. No infrastructure
required. The interesting part isn’t the tooling — it’s the shape.

## The shape: three layers, not one folder

The failure mode for a knowledge base is a single flat folder of
notes that slowly turns into a junk drawer. The fix is to separate
things by *who owns them and whether they’re allowed to change*.
Three layers:

`wiki/
 raw/ # sources, captured verbatim — immutable
 synthesis/ # notes the model writes — concepts, comparisons, overviews
 maps/ # the index and the maps of content — read first
```

**raw/` is what you captured.** An article
you clipped, a transcript, a paper, a config dump, a thread you didn’t
want to lose. It goes in verbatim and it does not get edited. This is
your ground truth. When a synthesized note and a source disagree, the
source wins, and you can always tell which is which because they live in
different folders.

**`synthesis/` is what the model made of
it.** This is where the agent earns its keep: a concept note that
distills three sources into one explanation, a comparison note (“X
versus Y, when to reach for each”), an overview that stitches a topic
together. These are *derived* artifacts. They’re allowed to be
wrong, allowed to be overwritten, allowed to go stale — because they’re
cheap to regenerate from `raw/`.

**`maps/` is how anything gets found.** At
minimum, an index. Ideally a few “maps of content” — short hub notes
that link out to everything on a topic. This is the layer the agent
reads *first*, every time, before it opens anything else.

The point of the split is that it makes the dangerous operation —
editing — safe. The model can churn the synthesis layer freely. It can
never touch the sources.

## Two files carry most of the weight

Inside `maps/`, two plain files do more than any fancy
graph view ever will.

**`index.md` — one line per page.** A catalog
the agent reads to decide what to open. Not the content, just the
pointers:

`[](#cb2-1)# Index
[](#cb2-2)
[](#cb2-3)- [retry-on-429](../synthesis/retry-on-429.md) — backoff + jitter for rate-limited API calls
[](#cb2-4)- [streaming-vs-batch](../synthesis/streaming-vs-batch.md) — when to stream tokens, when to wait
[](#cb2-5)- [auth-token-lifecycle](../synthesis/auth-token-lifecycle.md) — where tokens are minted, stored, rotated
```

This is the trick that lets you skip embeddings for a long time. The
agent reads ~one screen of index, picks the two or three notes that
matter, and opens only those. You’re doing retrieval with a table of
contents instead of cosine similarity — and for a few hundred notes, a
table of contents is more precise, not less.

**log.md` — append-only history.** A dated
record of what changed and why:

`[](#cb3-1)# Log
[](#cb3-2)
[](#cb3-3)- [2026-06-28] Added comparison note on streaming vs batch; superseded the old "always stream" claim.
[](#cb3-4)- [2026-06-22] Captured the rate-limit thread into raw/; wrote retry-on-429 synthesis.
[](#cb3-5)- [2026-06-15] Marked auth-token-lifecycle 🟡 — provider changed rotation window, note may be stale.
```

The log is the thing people skip, and it’s the thing that makes the
wiki trustworthy six months in. It’s the difference between a note you
believe and a note you have to re-verify. Append only — you never
rewrite history, you only add to it.

## Three operations, not a hundred

A wiki you have to remember how to use is a wiki you stop using.
Collapse everything into three verbs and wire each one to a command or
skill so the agent runs it the same way every time.

**Ingest.** Pull a source into raw/`
verbatim → write or update a synthesis note → append a line to
`log.md` → add the pointer to `index.md`. Four
steps, one command. The discipline is that ingestion is never “just save
the link” — it always produces a synthesis note, because an unread
source is indistinguishable from a source you don’t have.

**Query.** Read `index.md` → open the two or
three relevant notes → answer, citing them. The agent should be able to
say *which* notes it used. If it can’t, the index isn’t doing its
job and you’ve got orphan notes nothing links to.

**Lint.** This is the operation almost nobody builds,
and it’s why most knowledge bases rot. A health-check pass over the
whole wiki that flags:

- Notes whose `updated:` date is older than the source they
cite (stale).

- Two synthesis notes that contradict each other.

- Orphans — files in `synthesis/` that nothing in
`index.md` points to.

- Any sign a source in `raw/` was edited (a cardinal sin —
see below).

Give each note a status the linter can set: 🟢 current, 🟡 suspect,
🔴 contradicted. Run it on a schedule, or before any session where
you’re going to lean on the wiki for something that matters. A knowledge
base without a linter is a knowledge base that’s quietly lying to you by
month three.

## The two rules that stop the rot

Everything above is structure. These two rules are what keep the
structure honest.

**1. Sources are immutable.** `raw/` is
read-only to the model. The moment you let an agent “tidy up” a source,
you’ve destroyed the one thing that made it ground truth — you can no
longer tell what was captured from what was inferred. This sounds
obvious and is violated constantly, because asking a model to
summarize-in-place feels efficient. It isn’t. Summarize *into a new
file in `synthesis/`*. Leave the original alone.

**2. The synthesis layer is the model’s to own.** The
flip side. Don’t hand-curate the synthesis notes into precious artifacts
you’re afraid to lose — they’re regenerable from `raw/`. Let
the agent overwrite them, merge them, split them. The value is in the
sources and the index; the synthesis is a cache. Treating it as
disposable is what lets you ingest aggressively without fear.

Together these two rules draw a clean line: humans (or capture tools)
own `raw/`, the model owns `synthesis/`, and
`maps/` is the contract between them.

## Conventions that make it machine-readable

Small things, but they’re what let the linter and the agent reason
about the wiki instead of just reading it:

`[](#cb4-1)---
[](#cb4-2)updated: 2026-06-28
[](#cb4-3)status: 🟢
[](#cb4-4)sources:
[](#cb4-5) - ../raw/rate-limit-thread.md
[](#cb4-6)---
[](#cb4-7)
[](#cb4-8)# Retry on 429
[](#cb4-9)...
```

Frontmatter on every synthesis note: when it was last touched, its
status, and which sources it derives from. That sources`
list is what lets the linter catch a stale note automatically — if any
source is newer than `updated:`, flag it. You get a
self-checking knowledge base out of three lines of YAML.

## Partition by domain so contexts don’t bleed

Once a wiki covers more than one area, give each domain its own
index. A working setup might keep entirely separate trees for, say,
infrastructure notes, a research reading pile, and a side project — each
with its own `index.md` and `log.md`, each
ingested and queried independently.

The reason is practical, not tidy: when the agent loads an index, you
want it loading *the relevant* index, not a merged megafile where
infrastructure trivia dilutes the research notes. Partitioning keeps
each query’s context small and on-topic. One wiki, several front
doors.

## When to reach for a real database

This is a directory of markdown read through file globs or an MCP
server. That’s the right tool from your first note to somewhere in the
**hundreds** of pages. It’s deterministic — the agent opens
the exact file you indexed, not the nearest vector — and it’s
debuggable, because you can read the whole thing yourself.

Past that scale, the index stops fitting in a sensible context window
and retrieval-by-table-of-contents breaks down. That’s the signal to add
embeddings — and only then. I’ve written about [building deterministic RAG in
Phoenix with pgvector](https://sublimecoding.com/blog/phoenix-rag-pgvector-arcana) for when you get there, and about [when a markdown file beats your
vector database](https://sublimecoding.com/blog/claude-md-onboarding-doc) for the decision itself. The mistake is starting
with the database. Start with the directory; graduate to the database
when the directory tells you to.

## The wiki reads itself

Karpathy’s framing — a knowledge base written for a model — lands
because it changes what “good documentation” means. A good human doc is
narrative and persuasive. A good agent doc is indexed, dated, sourced,
and linted. It’s less a book and more a small codebase: the index is the
entry point, the synthesis notes are the modules, the sources are the
vendored dependencies you don’t edit, and the linter is CI.

You don’t need infrastructure to start. You need a folder, three
layers, three operations, and the discipline to never edit a source.
Wire the operations to commands, run the linter on a cadence, and the
thing you build in an afternoon will still be telling you the truth a
year from now — which is more than most knowledge bases can say. The
same shift I described for [CLAUDE.md as a load-bearing
artifact](https://sublimecoding.com/blog/claude-md-after-50-commits) applies here: the wiki is only worth keeping if you treat
it as version-controlled, executable, and maintained — not as a place
notes go to die.

---

## Your CLAUDE.md Is the Onboarding Doc You Never Wrote

URL: https://sublimecoding.com/blog/claude-md-onboarding-doc
Published: 2026-06-29
Tags: AI tools, engineering leadership, agents

>
**TL;DR:** A `CLAUDE.md` is not
configuration. It’s the onboarding doc your team never wrote — the
tribal knowledge that normally lives in one senior engineer’s head,
written down at last because something finally needed to read it. The
twist: your “second brain” was never really for you. The agent is the
first reader your notes have ever had, and the first one that acts on
what it reads.

For about a decade we told ourselves a story about personal knowledge
management. Capture everything. Link your notes. Build a second brain.
Most of us built the vault, filled it for three weeks, and never opened
it again. The notes were real. The reader never showed up.

The reader showed up. It’s just not human.

When you write a `CLAUDE.md` — or an
`AGENTS.md`, or whatever your coding agent reads at the start
of every session — you are doing the thing PKM always promised and
rarely delivered: writing knowledge that gets consumed. Not filed.
Consumed, on every task, by something that turns it into action. And the
moment notes have a reader that acts, everything about how you should
write them changes.

## What a CLAUDE.md actually is

A `CLAUDE.md` is **executable tribal
knowledge**: the undocumented operational facts about a codebase,
written down because an agent now needs them to work, and loaded into
context on every session so they actually get used.

That’s the whole thing. It reads like config — a markdown file in
your repo root — but it isn’t. Config tells a machine *what to
do*. A `CLAUDE.md` tells an agent *what you know that
isn’t in the code*: the port another local app squats on, the test
that fails for unrelated reasons, the directory you must never touch,
the deploy that happens on push and not via the obvious command. The
knowledge a new hire extracts painfully over their first month by
breaking things and asking in Slack.

Anthropic’s own guidance lands in the same place: the question to ask
when writing one is “what would I tell a new engineer in their first
five minutes on this repo?” That is not a config prompt. That is an
onboarding doc.

## The onboarding doc you never wrote

Here’s the uncomfortable part for anyone running a team. You almost
certainly do not have a good onboarding doc for humans. Nobody does.
Onboarding docs rot the instant they’re written, nobody is incentivized
to maintain them, and the real knowledge stays where it always was — in
the head of the one person who’s been there longest. We’ve all just
agreed to call this “ramp-up time” and absorb the cost.

The agent doesn’t accept that arrangement. It has, in Karpathy’s
memorable phrase, **anterograde amnesia** — no memory
across sessions by default. Every session is its first day. So either
you write the knowledge down or the agent relearns the codebase, badly,
every single time. The economics that let human onboarding docs rot —
“we’ll just have someone explain it” — collapse, because there’s no
someone, and the explaining happens fifty times a day.

So teams that maintain a good `CLAUDE.md` are doing
something they never managed to do for people: **they’re keeping
an onboarding doc alive, because a daily user finally forces the
maintenance.** That’s not a side effect to shrug at. Tribal
knowledge becoming executable and version-controlled is one of the
quietly large shifts in how engineering orgs work, and most teams are
doing it by accident while thinking they’re just configuring a tool.

If you want the tactical version of how one of these files evolves
under real use, I wrote about [what my CLAUDE.md looked like
after 50 commits](https://sublimecoding.com/blog/claude-md-after-50-commits) — what stayed, what got cut, and why most of it was
operational trivia no human doc would have bothered to record.

## CLAUDE.md vs. the team wiki

This is where people get confused, so let me be precise about it. A
`CLAUDE.md` is not your Confluence. The two hold different
knowledge for different readers, and conflating them is how you end up
with a 4,000-line `CLAUDE.md` that the agent ignores.

**CLAUDE.md**
**Team wiki / Confluence**

Primary reader
The agent (every session)
Humans (occasionally)

Knowledge type
Operational, repo-specific, *executable*
Conceptual, organizational, narrative

Loaded when
Automatically, every task
When someone remembers it exists

Failure mode
Bloat — too much, agent skips it
Rot — nobody reads, nobody updates

Maintenance forcing function
Strong (agent breaks if stale)
Weak (no one breaks if stale)

Length discipline
Ruthless — every line costs context
None — pages grow forever

The rule that falls out of this: **put in
`CLAUDE.md` only what the agent needs to not screw up, and
put it there because leaving it out has a visible cost.**
Everything conceptual — why the architecture is the way it is, the
history, the org chart — belongs in the human wiki, which will continue
to rot, because nothing forces it not to. The lesson isn’t “make your
wiki more like CLAUDE.md.” It’s that the *forcing function* is
what was always missing, and the agent supplies it for free.

The difference is visible at the line level. This belongs in the
wiki, not here:

`[](#cb1-1)Auth was originally built in 2021 around session cookies; after the OAuth
[](#cb1-2)migration we moved to short-lived JWTs because the mobile team needed
[](#cb1-3)offline validation, and the refresh flow has some history worth knowing.
```

That’s narrative — context a human might want and an agent skims
straight past. This is the version that earns its tokens:

[](#cb2-1)Auth tokens live in `lib/app/auth.ex`. Never log them. Tests stub them
[](#cb2-2)via `AuthFixtures.token/0` — do not hit the real provider in tests.
```

Same subject, opposite discipline. One explains; the other prevents a
mistake. Only the second one belongs in a file that gets read on every
task.

## When a markdown file beats your vector database

The second-brain crowd’s reflex is to reach for retrieval. Embed
everything, stand up a vector store, RAG it back. And for a genuinely
large corpus, you’ll need that. But the interesting claim — Karpathy’s,
and increasingly mine after running it — is that the threshold where you
*need* it is much further out than people assume.

His framing is a literal wiki the agent maintains: a git repo of
markdown, a read-first index.md` that catalogs every page in
one line each, and the model just… reads the index and opens what’s
relevant. No embeddings. The analogy he uses is sharp: *Obsidian is
the IDE, the LLM is the programmer, the wiki is the codebase.* It
works, he argues, up to *hundreds* of pages before retrieval
earns its keep.

Why does the dumb version hold up so well? Because a curated index is
a precise, deterministic table of contents, and a context window is now
big enough to swallow what the index points to. RAG trades fidelity for
scale — it can miss the right chunk, surface the wrong one, and needs
reindexing every time you change embedding models. A markdown wiki
trades scale for fidelity: perfect recall of exactly what you wrote,
version history for free, nothing to drift. (When you *do* cross
into needing real retrieval, that’s its own discipline — I got into the
weeds of [running pgvector
RAG in Phoenix](https://sublimecoding.com/blog/phoenix-rag-pgvector-arcana) and the arcana is real.)

The decision, stated plainly:

- **Direct-load it** (`CLAUDE.md`, imports)
when it’s a small, always-needed core. Repo rules. The handful of facts
every session needs.

- **Link it** (a wiki of cross-referenced markdown) when
the corpus is up to a few hundred pages and the *connections*
between notes carry meaning.

- **Embed it** (RAG) only when you’ve genuinely outgrown
what an index-and-open workflow can hold.

Most people start at the bottom of that list and should start at the
top. Reach for the vector DB last, not first.

## So what do you actually put in it

Keep it to the things that are true, non-obvious, and costly to get
wrong:

- **Operational landmines.** The port conflict, the flaky
test, the build that lies, the directory that’s
gitignored-but-load-bearing. The stuff that makes a new session waste
twenty minutes rediscovering a fact you already knew.

- **Conventions the code doesn’t enforce.** “We use this
tag taxonomy.” “Never commit to main directly.” “Format on save or
precommit trips.” Rules that live in people’s habits, not in a
linter.

- **The shape of the workflow.** How to run things, what
to run before committing, what ‘done’ means here. This is where a
`CLAUDE.md` shades into the rest of your agent setup — [skills, not just one
mega-file](https://sublimecoding.com/blog/prompt-skills-not-claude), each loaded only when its task comes up, so the always-on
core stays lean. My [daily
agentic workflow](https://sublimecoding.com/blog/my-daily-agentic-ai-workflow) leans on exactly that split.

Here’s an actual slice of the file that runs this site — not a
sanitized template, the real operational sludge:

`[](#cb3-1)### Local dev server: use PORT=4001, not 4000
[](#cb3-2)Port 4000 is held on this machine by an unrelated app. Phoenix will fail
[](#cb3-3)to bind — or worse, you'll hit the wrong app and waste ten minutes on
[](#cb3-4)phantom 404s. Always start with `PORT=4001 mix phx.server`.
[](#cb3-5)
[](#cb3-6)### Tag taxonomy is enforced by tests
[](#cb3-7)`seo_health_test.exs` lists merged-out tag slugs that fail the build if
[](#cb3-8)used on a new post. Use the consolidated set — copy an existing post.
[](#cb3-9)
[](#cb3-10)### `.claude/launch.json` deletion is pre-existing
[](#cb3-11)Every session opens with this file showing as deleted in `git status`.
[](#cb3-12)Do not stage or restore it. Leave it alone.
```

None of that is clever. None of it is in the code. All of it is what
a new hire would burn an afternoon rediscovering — and every line earns
its place because leaving it out has a measurable cost: a wasted
afternoon, a red build, a confused commit. That’s the test for whether
something belongs: *does omitting it cost a session something
concrete?*

And keep it *short*. Every line you add is loaded on every
task and competes for the model’s attention with the actual work. A
CLAUDE.md` that tries to be the wiki becomes noise the agent
learns to skim. The discipline is the value. This is “context
engineering” — the term Karpathy popularized for filling the context
window with *just* the right information for the next step —
applied to the most mundane artifact in your repo.

## The reader that never forgets

There’s a longer arc here that matters if you lead engineers. We’ve
spent years worried about bus factor — the senior who carries the
codebase in their head, the knowledge that walks out the door when they
do. The standard fix is “write more docs,” and the standard outcome is
docs nobody maintains. I’ve argued before that the AI shift is quietly
[changing how we grow senior
engineers at all](https://sublimecoding.com/blog/stop-making-senior-engineers); this is the other half of it.

When the tribal knowledge has to be written down for the agent — and
*stays* written down because the agent breaks without it — the
bus factor problem gets attacked from a direction it never has been. Not
“please document this.” Instead: the documentation is load-bearing,
version-controlled, and decays visibly the moment it’s wrong, because
something fails immediately.

That’s the inversion worth sitting with. The second brain was never
going to work as long as the only reader was a future version of you who
would never actually open it. The notes needed a reader who shows up
every day, never gets bored of the maintenance, and turns what it reads
into action. That reader exists now. It just isn’t human — and it’s
reading your `CLAUDE.md` right now, treating your offhand
operational notes as the most important onboarding doc your team has
ever written.

Write it like you mean it.

---

## We're About to Stop Making Senior Engineers

URL: https://sublimecoding.com/blog/stop-making-senior-engineers
Published: 2026-06-17
Tags: AI, engineering leadership, hiring, engineering

>
**TL;DR:** For thirty years the way you became a senior
engineer was boring and reliable: you did a pile of unglamorous work —
the CRUD screens, the flaky-test triage, the “why is this query slow”
afternoons — and somewhere in that pile your hands learned things nobody
could have told your head. AI is exceptional at exactly that pile. So
we’re about to get very good at producing code and very bad at producing
the people who understand it. The gap between *knowing* an answer
and having *earned* it is widening, it hits juniors first and
hardest, and cutting juniors to “do more with AI” quietly burns the
bench you’ll need in five years. The fix isn’t fewer juniors. It’s
teaching them to be architects on purpose, instead of hoping it rubs
off. The counter-argument, from the other side of the same problem: [a junior leaning on the agent is fine, if you move the bar to verification](https://sublimecoding.com/blog/junior-dev-leaning-on-the-agent).

## The ladder is losing its bottom rungs

Becoming a senior engineer was never really about the senior work. It
was about the junior work, done enough times that it stopped being
junior. You wired up the forms. You chased the null that only showed up
in staging. You rewrote the same migration three ways before one of them
didn’t lock the table. None of it was prestigious, and all of it was the
curriculum. The architecture sense you wanted lived on the far side of a
thousand small, tedious encounters with how systems actually behave when
you connect them.

That’s the ladder. And AI is very good at the bottom rungs.

The optimistic framing — the one I mostly agree with — is that AI
amplifies human judgment rather than replacing it. I’ve [argued a version of that
myself in response to Nadella](https://sublimecoding.com/blog/nadella-token-capital-small-teams). But “amplifies judgment” has an
asterisk nobody likes to read aloud: amplification is multiplication,
and multiplying by a junior’s judgment gives you a junior’s judgment,
louder. Judgment isn’t a thing you can be handed. It’s sediment. It
settles out of work, slowly, and the work it settles out of is precisely
the work we’ve just handed to a machine.

## Knowing is not doing, and the gap is getting wider

There has always been a gap between knowing how something works and
being able to do it. You can memorize every design pattern in the
catalog and still reach for the wrong one the first time a real system
makes you choose. What’s new is how convincingly AI lets you skip the
doing while feeling like you did it.

A junior who prompts an agent into a working feature has
*knowledge* — they can describe what the code does, point at the
parts, explain the pattern. What they don’t have is the thing you only
get by writing it wrong first: the felt sense of why this boundary and
not that one, what breaks when load doubles, which clever line will be a
three-hour outage in eighteen months. [AI-assisted
engineering isn’t faster typing — it’s a different workflow](https://sublimecoding.com/blog/ai-assisted-engineering-is-a-new-workflow), and the
workflow is fundamentally one of review and direction. That’s a senior’s
job description. We’ve handed it to people who haven’t yet earned the
instincts review depends on.

Debugging is where the gap is starkest. I wrote years ago that [debugging is an exercise
in thinking](https://sublimecoding.com/blog/debugging-an-exercise-in-thinking), not a tool skill — the work is building an accurate
mental model of a system and then finding the one place reality diverges
from it. You don’t build that model by skimming generated code and
accepting it; passive acceptance teaches nothing. You build it the hard
way: from systems you assembled yourself, badly, and then had to
understand under pressure because production was down and it was your
fault. Take away the building-it-badly and the model has nowhere to come
from.

## Why juniors get hit first

Seniors are mostly fine. AI hands a staff engineer a draft and they
bring fifteen years of priors to bear — they read it the way a
structural engineer reads a blueprint, seeing the load paths and the
place it’ll crack. The model amplifies judgment they already have.
That’s the whole trick, and it’s real. Note the asterisk on “mostly,”
though: that judgment is a stock, not a flow — banked under the old
curriculum, the one we’re busy automating away. A senior who only
reviews and never builds is spending down priors they’ve stopped
replacing.

Juniors have nothing to amplify yet, and the tasks that used to build
the priors are the ones now evaporating. This is the trap: the same tool
that makes a senior modestly more effective makes a junior *look*
dramatically more effective while quietly starving them of the reps that
would make the gains real. The output goes up and the learning goes
down, at the same time, from the same cause.

The worst failure mode isn’t slow learning. AI is confidently,
fluently wrong on a regular basis — [an agent
deleted a production database in nine seconds and then explained,
articulately, why it shouldn’t have](https://sublimecoding.com/blog/ai-deleted-production-database-hire-more-engineers). Catching that requires knowing
when the plausible thing is the wrong thing, which is exactly the
judgment we just established juniors haven’t built. [Knowing when to
trust an agent and when to step in](https://sublimecoding.com/blog/when-to-trust-an-agent-and-when-to-step-in) is the load-bearing skill of
2026, and it’s the one that’s hardest to fake. A junior can’t override
what they can’t evaluate. They’ll ship the confident wrong thing,
because it looked like all the other confident things that happened to
be right.

## The lazy move is to cut juniors. It eats your future.

The market’s first instinct is to read all this as “so hire fewer
juniors.” It’s the wrong lesson, and an expensive one. [AI won’t shrink your team — it’ll
expose how much work you were leaving on the table](https://sublimecoding.com/blog/ai-wont-shrink-your-team), and the team
that handles that surfaced work still needs seniors. Seniors are a
renewable resource only if you keep growing them. Cut the junior
pipeline and you’ve optimized this year’s burn by quietly canceling
2031’s staff engineers. There is no senior-engineer factory that doesn’t
start with juniors; there is only the apprenticeship, and we’re
proposing to defund it right as it gets harder to run. And the dodge —
let everyone else grow juniors, then poach the finished seniors — only
works until enough firms try it that nobody’s growing any, at which
point bought-in seniors are scarce and expensive and the ones you grew
yourself are cheaper and likelier to stay.

So the question that actually matters isn’t whether to hire juniors.
It’s how to grow them now that the curriculum they used to learn from
has been automated out from under them.

## The field guide: grow architects on purpose

The old model worked by accident — do the grunt work, absorb the
lessons. With the grunt work gone, the lessons have to be taught on
purpose. That’s the shift: mentoring a junior in 2026 is less “here’s a
ticket” and more “here’s how to think like the person who’d architect
this.” Four things that actually move it:

**Make them build it by hand, once.** Before a junior is
allowed to generate the auth flow, they write one themselves, slowly,
and feel where it’s awkward. You earn the right to the abstraction by
having lived without it. This is deliberately inefficient, and the
inefficiency is the entire point — it’s the gym, not the commute.
Protecting that time is a real fight, because it never looks urgent; [making time to learn has always
required intent, not slack](https://sublimecoding.com/blog/making-time-to-learn), and that’s more true now, not less.

**Turn the AI into the thing they review, not the thing they
trust.** The new bug hunt is reading plausible generated code and
finding the flaw — the off-by-one in the pagination, the missing
transaction boundary, the auth check that’s subtly in the wrong place.
Have juniors critique agent output as a daily exercise. It rebuilds,
deliberately, the muscle the old debugging afternoons used to build by
accident.

**Assign systems, not tickets.** Hand a junior a whole
small service and make them defend the trade-offs — why this boundary,
what happens at 10x, where it fails safely. Make them [triage an unfamiliar
codebase and form a real point of view about it](https://sublimecoding.com/blog/triage-a-new-codebase-90-minutes), because reading
architecture is a skill you can practice directly instead of waiting
years to absorb. The goal is to move them up the abstraction ladder on
purpose, faster than the old osmosis ever did.

**Grade outcomes, not output.** When the cost of
producing code goes to zero, the cost of producing the *right*
code — the kind that fails safely and the next person can change without
spelunking — is the entire job. Make it a drill: before a junior starts,
have them write down what “done” means as outcome conditions, not a
diff. It degrades safely under 10x load. The next engineer can change it
without you in the room. The failure pages someone with a message they
can act on. Then grade against that list, not against whether the code
runs. [A professional owns
the whole outcome](https://sublimecoding.com/blog/professional-owns-the-outcome) — not the diff, the outcome — and owning the
outcome is the one thing on this list AI can’t do for them.

## The firms that win will manufacture seniors faster

The obvious objection writes itself: if AI just swallowed the bottom
of the ladder, won’t the next two model generations swallow the
architecture and the judgment too, making this whole project an
investment in a skillset with a three-year shelf life? Maybe. But it’s a
bet, and the asymmetry runs one way. If understanding stays a human job,
the firms that kept growing it win outright. If it doesn’t, everyone
loses that layer at once — and the firms that grew real engineers are
still best-positioned to work out what comes next, because judgment
about a moving target is the last thing to automate. Betting on
understanding pays in every world except the one where nothing you did
would have mattered anyway.

That gap is a problem, but it’s also a tell: the bottleneck has moved
from typing to understanding, and understanding is teachable if you
decide to teach it. AI can compress the feedback loop brutally — a
junior can now explore ten architectures in the time it used to take to
build one, *if* you point them at understanding the ten instead
of just shipping whichever the model produced first.

The companies that lose this decade will read “AI does the junior
work” as “so we don’t need juniors.” The ones that win will figure out
how to manufacture senior engineers faster than the old apprenticeship
ever could — by teaching, on purpose, the judgment that used to be left
to chance. The ladder lost its bottom rungs. The firms that bother to
rebuild them are the ones that still have a company in five years.

## Read this next

-
[**How I Triage a
New Codebase in 90 Minutes**](https://sublimecoding.com/blog/triage-a-new-codebase-90-minutes) — Reading architecture is a
skill you can practice directly; here’s the drill.

-
[**How
I’d Hire a Staff Engineer at an AI Startup**](https://sublimecoding.com/blog/how-id-hire-a-staff-engineer-at-an-ai-startup) — What the
destination role actually requires, and how to interview for it.

-
[**A Professional
Owns the Whole Outcome**](https://sublimecoding.com/blog/professional-owns-the-outcome) — The mindset you’re really trying
to grow when you grow a junior.

---

## A Professional Owns the Whole Outcome

URL: https://sublimecoding.com/blog/professional-owns-the-outcome
Published: 2026-06-07
Tags: engineering, engineering leadership, staff engineer

>
**TL;DR:** Professionalism in software isn’t the standup
cadence, the groomed tickets, the right vocabulary, or the title on your
badge. It’s taking ownership of the whole outcome — accepting
accountability for everything that happens because your code exists,
even the parts you don’t fully control and no one will ever trace back
to you. That boundary, not your title, is the thing. Naming the cost,
verifying before you claim, knowing when *not* to build, and
shipping something that fails safely are what that ownership looks like
in the places people quietly skip.

## Professionalism is not what it’s cosplaying as

Walk into most engineering orgs, ask what makes someone a
professional, and you’ll get a description of a costume. They stand up
at the right time in standup. Their tickets are groomed. They say “let’s
take that offline” and “what’s the blast radius” and “I’ll own that.”
They have the title — Senior, Staff, the one that took six years and a
calibration meeting. They look the part.

None of that is professionalism. It’s the uniform you wear to a job,
and like any uniform you can buy it without earning it. I’ve worked with
engineers who had every signal and shipped like tourists — clean diffs,
immaculate ceremony, and not one question about whether the thing
actually worked once it left their hands. I’ve also watched someone
three weeks out of a bootcamp sit through a 3am incident that wasn’t
technically theirs, because they were the one who’d noticed it breaking
and couldn’t talk themselves into pretending they hadn’t. One of those
two was being a professional. It wasn’t the one with the better
title.

Take the costume off and what it was imitating is ownership of the
whole outcome. Everything that happens because your code exists is
yours: the feature, sure, but also the cost, the failure mode, the next
engineer who has to read it, the claim in the PR description that turned
out to be wrong. A professional draws the boundary of “my work” around
the outcome. An amateur draws it around their diff.

## Craft is necessary, and it is not enough

The obvious objection is that professionalism is craft — write clean,
correct, well-tested code and you’ve done the job. Craft is real and
non-negotiable, but it’s the floor, not the ceiling. You can write a
flawless module that’s correct in isolation and still ship a bad
outcome: because it was the wrong thing to build, because it failed
silently when an assumption broke, because the next person couldn’t
safely change it, because nobody asked what happens when it’s actually
used. Excellent code pointed at the wrong problem is not professional
work. It’s a specialist admiring his own corner while the outcome he was
hired to produce quietly doesn’t happen.

So the boundary that matters isn’t “is my code good.” It’s “did the
thing we were trying to make happen actually happen, and is it still
happening at 3am on a Sunday.” That line is wider and more
uncomfortable, and drawing it there is the whole move.

## Where the line actually falls

The amateur’s boundary is legible and small: my code works. It passed
review. The bug’s in the other team’s service. The spec didn’t say to
handle that case. Every one of those can be *true* and still be a
dodge, because each is a way of saying the outcome stopped being my
problem at the edge of my diff.

But this isn’t a binary, and it isn’t a license to blame a junior for
everything downstream of their first commit. Ownership scales with
scope. The line falls where your visibility to influence the outcome
reaches — and a staff engineer’s reaches a lot further than a new
hire’s. If you could see the failure coming, because the signal was
there or the code path was your domain, you own it. If you genuinely
didn’t have the signal, you own finding out why the signal wasn’t there.
You never own never-failing. You own failing safely, and knowing why.
Seniority isn’t permission to draw the line tighter; it’s the obligation
to draw it wider, because you can see more.

## Accountability is the same act, before and after

Accountability and ownership get said in one breath so often they’ve
gone slack. They’re two distinct acts, and a professional runs both.

Accountability before the fact is naming the bar before you know
whether you’ll clear it. “This refactor cuts p95 by a third.” “This
migration is done when the old table is dropped, not when the new one is
written.” You fix the measure while being wrong about it is still cheap.
The engineer who’ll only tell you how to grade the work *after*
the results are in is grading himself.

Accountability after the fact is owning the miss with no sunk-cost
defense. The dead tool that stays installed because ripping it out would
admit the first call was wrong — that’s not a budget problem, it’s an
ownership problem wearing a budget costume. The professional version is
unglamorous: “I picked this, it didn’t work, here’s what it cost, we’re
pulling it.” You’re allowed to say you learned something *and*
say it was the wrong call. Both are true; only one of them is
accountability.

And the load-bearing part: the blast radius is yours even when the
blame isn’t. The outage is in a service your team doesn’t own, but you
can see the fix, so you fix it and sort out the org chart afterward.
Nobody hands you credit, because on paper it wasn’t your fire. That’s
the tell. If you extend ownership only to what will be credited to you,
it isn’t ownership — it’s reputation management.

## The four places it shows up that people skip

If ownership is the spine, these are the four places it has to show
up in actual behavior, and they’re the four most people walk past.

**Own the downside.** A recommendation that sells only
the upside is one you haven’t really owned, because if you’d owned the
miss in advance you’d have priced it — the hiring tax, the slower loop,
the thing that bites three weeks in. Name the cost in the same breath as
the benefit, not as a disclaimer at the bottom. If you never costed the
downside, you don’t know you’re right. You just know you’d like to
be.

**Own your words.** The moment you say a number out
loud, it’s credited to you. Repeating a vendor’s claim you never checked
isn’t reporting; it’s lending your credibility to a figure that may not
survive contact with its own methodology — and when it doesn’t, you’re
the one who gets credited for the damage. Read the code, not the README.
Cite the path, because the path is proof you looked.

**Own the decision not to do it.** Half the job is the
work you talk the room out of. The plausible migration, the well-demoed
tool, the rewrite every instinct says yes to — refusing those is
ownership too, because the time you spend is exactly as gone as the time
you waste. “What measurable thing gets worse if we don’t do this?” is a
question a professional asks first, not in the retro. Knowing when not
to reach for the thing is the most concentrated judgment there is.

**Own it past your commit.** Your outcome doesn’t end at
merge. The next person to touch this — possibly you, eighteen months
out, with no memory of why — is part of what you shipped. So you build
it to fail safely instead of loudly: isolation that doesn’t depend on
every future author remembering an incantation, errors a human can
actually read, a fault the system absorbs instead of paging someone.
Heroics at 3am aren’t professionalism. They’re the invoice for an
architecture that declined to own its failure modes in daylight.

## Where this gets expensive

Here’s the part the rest of this skips — the way I’d owe you the
costs of any tool I recommended, because a piece that sells only the
upside is one I haven’t owned.

The expensive failure of ownership isn’t too little of it. It’s the
engineer who owns *everything*: can’t let a thing ship without
their hands on it, becomes the only person who understands the system,
quietly arranges to be the only one who can answer the page. That looks
like the summit of professionalism and it’s the opposite, because it
breaks the fourth discipline — it builds an outcome that can’t survive
its owner being on vacation. Accountability for the outcome is yours.
Being the single point of failure for it is a different thing, and
confusing the two is how good engineers turn themselves into
load-bearing walls. You own the outcome by surfacing problems, chasing
failures, and helping fix them past your own code — not by making the
system need you in the room.

The harder cost is that ownership and control are not the same, and
organizations love to hand out the first without the second. This is why
the honest claim is accountability *even when you don’t fully control
it*: that’s not a loophole, it’s most of the job. You will own
outcomes you can’t fully steer, and doing it anyway is the work. But you
should be able to tell a place that’s cultivating ownership from one
that’s extracting it — handing you accountability for things it won’t
give you the authority to change — because the second will run you dry
and call it seniority. Owning the outcome includes owning the call on
whether the deal you’re being offered is an honest one.

## What you’ll put your name on

So forget the costume. The cadence and the groomed tickets and the
title are, at best, correlated with the real thing and, at worst, a
convincing disguise for its absence. The junior who says “that’s our
outcome, I’ll get it sorted” is being more professional than the senior
who says “not my code” — every time, in every org, whatever the badges
say.

Professionalism isn’t what you look like while you work. It’s the
size of the boundary you draw around the word *mine* — and
whether, when the outcome is bad and nobody is making you, you still
refuse to disown it.

## Read this next

- [**How I Triage a New Codebase in 90 Minutes**](https://sublimecoding.com/blog/triage-a-new-codebase-90-minutes) — Owning the outcome starts with knowing what you just inherited.

- [**When to Trust an Agent and When to Step In**](https://sublimecoding.com/blog/when-to-trust-an-agent-and-when-to-step-in) — Ownership doesn't delegate to the model — the autonomy ladder in practice.

- [**How I'd Hire a Staff Engineer at an AI Startup**](https://sublimecoding.com/blog/how-id-hire-a-staff-engineer-at-an-ai-startup) — Interviewing for outcome-ownership instead of years-of-experience.

---

## Prove the Return or Don't Spend the Time

URL: https://sublimecoding.com/blog/prove-the-return
Published: 2026-06-03
Tags: founders, business, productivity

>
**TL;DR:** You can raise more money, hire more people,
and buy more compute, but you cannot buy back a quarter. Time is the
only line item that never refills, which means every investment of money
*or* effort has to earn a demonstrable return — not a vibe, a
number. The most common way businesses violate this is buying software
nobody asked for: a tool sold to someone senior, justified by a pain no
one actually had, opened twice after onboarding, and quietly billed for
years. The fix isn’t a procurement policy. It’s one question you ask
before you spend anything: *what measurable thing gets worse if we
don’t do this, and how will we know in 90 days?* If you can’t
answer, you’re buying bloat.

## Time is the only non-renewable line item

Every other resource a business runs on can be topped up. Run low on
cash and you raise a round, take a loan, or sell more. Run low on people
and you hire. Run low on compute and you put it on a card. The supply is
constrained, sometimes painfully, but it exists.

Time is the exception. The quarter you spent migrating to a database
you didn’t need is gone. The two engineer-months poured into a process
that protected no one don’t come back when you realize the mistake.
There is no funding round for last spring. This is the asymmetry that
should govern how a business spends, and almost nobody runs their
company as if it were true.

I’ve watched smart operators agonize for a week over a $5,000 line
item and then wave through a decision that quietly burned a month of the
team’s attention — because the month never showed up on an invoice.
Money is legible. Time isn’t. So we guard the legible resource and
hemorrhage the irreplaceable one.

The entire discipline of running a business well comes down to fixing
that backwards. Treat time as the scarcest thing you own, because it is,
and force every claim on it — every tool, every initiative, every “we
should really” — to justify itself against that scarcity.

## “Demonstrable” is the operative word

It is easy to agree that investments should pay off. The agreement is
cheap because *pay off* stays comfortably undefined. The work is
in the word **demonstrable**.

A demonstrable return is one you can point at. A metric that moved. A
cost that dropped. A cycle that got measurably shorter. Revenue you can
trace back to the thing. The test isn’t “did this feel worth it” —
feelings are generous narrators — it’s “can I show you the number, and
would you have predicted that number before we started?”

Most spending that goes sideways fails precisely here. It produces
the *sensation* of progress without the substance. A new tool
gets adopted, dashboards light up, there’s a Slack channel and a launch
and a sense that the team is now Doing The Thing Properly. None of that
is a return. Activity is not output. Looking busy is not the same as
moving a number, and a surprising amount of what businesses buy is, on
honest inspection, the purchase of the feeling of being on top of
things.

If you can’t say in advance what number a spend is supposed to move,
you have not made a decision. You’ve made a purchase and attached a
story to it.

## The bloat tax

Here is the pattern, drawn from more companies than I’d like to admit
having seen it in.

A platform gets bought — call it forty thousand a year, all-in once
you count seats — to solve a problem. Someone senior sat through a very
good demo. The vendor’s deck described a pain in language crisp enough
that everyone nodded, and the tool was approved on the strength of that
nod. It onboarded. There were training sessions. For about six weeks it
had the glow of the new system everyone’s supposed to be using.

Then the usage graph did what those graphs do. A spike during
onboarding, a slope down through month two, a flat line by month three
that never recovers. Eighteen months later it’s still being billed,
nobody can quite remember who owns the contract, and the honest answer
to “what does this do for us” is a shrug and “it was supposed to help
with…” The sentence doesn’t finish because the pain it was bought to
kill was never one the company actually had. It was a pain the
*demo* had.

That’s the bloat tax. Not a single dramatic mistake but a slow
accretion of tools that each made sense in the room where they were
approved and add up to thousands of dollars a month solving nobody’s
problem. Seat sprawl. The analytics suite the marketing hire used at
their last job. The second project tracker bought because the first one
“wasn’t quite right,” now running in parallel with it. Each one
defensible alone. Together, a tax on the company that no one voted for
and no one can point to a return from.

The mechanism is always the same: the tool was matched to a
*described* problem, not a *measured* one. Nobody asked,
before signing, what specifically would get worse without it — and so
there was no way to ever tell whether it helped.

## Effort is spend too

Money at least leaves a trail. The more dangerous version of this
disease spends time, and time doesn’t invoice.

The migration nobody needed is the canonical case. The current system
is fine — boring, a little unfashionable, but fine. Someone makes the
case that the new thing is cleaner, more scalable, more correct, and the
case is *technically* true in the way that most architecture
arguments are technically true. A quarter disappears into the move. At
the end the company is in roughly the same competitive position it
started in, having spent its scarcest resource buying tidiness no
customer will ever feel.

Or the process ritual: the mandatory review, the extra approval step,
the meeting that recurs forever, each instituted to prevent some problem
that, examined closely, had happened once and wasn’t actually prevented
by the ritual anyway. These have negative return. They consume time
every single week and protect against nothing, but they survive because
removing them feels reckless and keeping them feels responsible.

This is the same illness as the unused platform, with the diagnostic
harder to run because there’s no line item to flag. You have to notice
it deliberately. The question that catches a $40k tool — *what gets
measurably worse without this?* — is exactly the question that
catches the needless migration and the protective ritual. We just rarely
point it at our own effort, because our own effort always feels like
work, and work feels like progress.

## Why smart teams keep doing this

None of this happens because people are foolish. It happens because
every force in the room pushes toward spending and almost none push
back.

Sunk cost keeps the dead tool alive: we’re already paying for it,
ripping it out would admit the original call was wrong, so it stays.
Optics reward motion: a leader who launches a new platform looks
decisive, while one who kills three redundant ones and buys nothing
looks like they’re not doing much — even though the second created far
more value. Fear of the gap drives the purchase: a competitor uses the
tool, a peer company has the process, and the absence feels like a risk
you can close with a credit card. And “everyone uses it” does enormous
unearned work, because matching what’s standard feels safe even when
standard is wrong for your stage and size.

Add it up and the default gradient of any organization slopes toward
more — more tools, more process, more spend — with nothing on the other
side of the scale unless someone deliberately puts it there. The
deliberate counterweight is the whole job.

## The test

You don’t need a procurement policy or a quarterly audit ritual, both
of which risk becoming their own bloat. You need one question, asked out
loud, before any meaningful spend of money or hours:

**What measurable thing gets worse if we don’t do this — and
how will we know in 90 days?**

That’s it. Two clauses, and most bad spending dies on the first one.
If you can’t name the specific thing that degrades without the tool, the
migration, the process, then you’ve found a solution shopping for a
problem, and the honest move is to not spend. If you *can* name
it but can’t say how you’d know in 90 days whether it worked, you’ve
named a hope, not a return, and you’ve given yourself no way to ever
shut it off.

The 90-day clause matters as much as the first. A return you can’t
check is a return you’ll never disprove, which is how the unused
platform survives to year two. Naming the signal in advance — the number
that should move, the date you’ll look — builds the off-switch at the
same moment you build the on-switch. You decide before you’re
emotionally invested how you’ll know if you were wrong.

It feels almost too simple to be a discipline. That’s the point. It’s
not hard to understand; it’s hard to *do*, because doing it means
saying no to plausible, well-demoed, everyone-uses-it spending that
every instinct says yes to.

## What running a business actually is

Strip away the rest and running a business is this: spending a
finite, non-renewable resource on the small set of things that
demonstrably pay you back, and refusing — actively, against the room’s
gradient — the much larger set of things that merely feel like
progress.

The tools that became bloat weren’t bought by careless people. They
were bought by competent operators who skipped one question because the
demo was good and the spend was approved and saying no would have felt
like falling behind. The cost wasn’t only the money, though the money
was real. It was the attention the team spent half-using something that
solved nothing, and the quarter that bought tidiness instead of
advantage — time that doesn’t come back, in the one budget that never
refills.

So prove the return, or don’t spend the time. Everything else is a
story you tell yourself while the only resource that matters drains
away.

## Read this next

- [**SOC 2 Is a Revenue Tool, Not a Security Tool**](https://sublimecoding.com/blog/soc-2-is-a-revenue-tool-not-a-security-tool) — The same return-on-spend framing applied to the audit every enterprise deal demands.

- [**How We Cut $350K in Cloud Spend in Six Months**](https://sublimecoding.com/blog/cut-350k-cloud-spend-six-months) — What proving the return looks like when the line item is infrastructure.

- [**vCISO Math for AI Founders**](https://sublimecoding.com/blog/vciso-math-for-ai-founders) — Run the same math on security leadership: $2–4K/month fractional vs $300K full-time.

---



---

End of file. For the structured guide and the full archive, see
https://sublimecoding.com/llms.txt and https://sublimecoding.com/blog.