Multi-Tenant AI SaaS in Phoenix: Isolation That Holds

TL;DR: Row-level tenant isolation in Postgres is a solved problem — a tenant_id column, an index, and discipline. What’s not solved, because almost nobody has written it down, is what happens to that isolation once every request also flows through an LLM call and a retrieval query. Three places break in ways ordinary Ecto hygiene doesn’t catch: your provider API keys (one shared key means one shared blast radius and one shared rate-limit pool), the context-building code that assembles a prompt (easy to leave an unscoped call site if scoping isn’t structural), and — the sharp one — a RAG retrieval query that’s flawless Ecto and still returns another tenant’s documents, because storage was scoped correctly and the read wasn’t. None of this shows up as an error. It shows up as tenant A’s onboarding contract answering tenant B’s support question, with a clean log and a passing test suite that never tested the thing that mattered.

The question a diligence reviewer already asks you

If you’ve read what enterprise security reviews actually probe on an AI startup, you’ve seen the line that stops more deals than any other item on the questionnaire: does one customer’s data ever influence another customer’s outputs? It’s a fair question and a specific one, and “we use Postgres row-level scoping, we’re fine” is not actually an answer to it — it’s an answer to a different, easier question. This post is the implementation answer: what tenant isolation has to look like once the data doesn’t just sit in a table, it flows through a prompt and a vector search on its way to another tenant’s chat window.

I’m going to spend one paragraph on the storage-layer decision and then leave it, because better posts already own that ground. Everything after that paragraph is the part that’s specific to AI products, and it’s the part I don’t see written up anywhere with actual code.

Schema-per-tenant vs. shared schema: not this post

You have two mainstream options for where tenant data physically lives in Postgres: a shared schema with a tenant_id column on every table (cheap to run, cheap to migrate, isolation enforced by your queries), or schema-per-tenant using Postgres’s native schema namespaces and Ecto’s prefix option (stronger blast-radius containment, more expensive migrations, a ceiling on how many tenants you can comfortably run). Curiosum’s multi-tenancy guide and Ecto’s own query-prefixes howto cover that tradeoff well and I’m not going to re-litigate it. Pick shared-schema unless you have a specific reason not to — a small number of large regulated tenants, usually — and move on. The rest of this post assumes shared-schema-plus-tenant_id, because that’s what most AI SaaS products actually run, and because the interesting failures live one layer up from that decision either way.

Per-tenant provider keys vs. one shared key

The first AI-specific isolation decision most teams make by accident: one API key to OpenAI or Anthropic, shared across every tenant, with usage attributed after the fact in your own logs. It’s the obvious starting point — one secret to provision, one bill to reconcile — and it’s fine for a while. It stops being fine at a predictable point, and it’s worth deciding on purpose instead of drifting into it.

The core tradeoff is blast radius versus operational overhead:

The rate-limit line is the one people get wrong, so it’s worth being precise. OpenAI enforces rate limits at the organization and project level, not per raw API key — a project carries its own token and request-rate allocation, separate from other projects in the same org. So “per-tenant key” isolation in practice means giving a tenant (or a tenant tier — enterprise tenants get their own project, everyone else shares one) its own project, not just a distinct-looking secret pointed at the same shared pool. Do that with a shared key instead, and one tenant running a batch job at 2am throttles every other tenant’s live traffic through the same 429s, and you’ll spend the incident review explaining why “we have rate limiting” didn’t stop it.

You don’t need per-tenant keys from day one. You need to know which side of that line you’re on, and you need it to be a decision, not the accidental consequence of copying OPENAI_API_KEY into one .env file and never revisiting it. The trigger to move off shared is usually the first tenant whose contract asks about it directly, or the first noisy-neighbor incident — whichever comes first.

Pin tenant_id where the prompt gets built, not at every call site

The second failure mode is more insidious because it doesn’t look like a security bug — it looks like a normal Ecto query that someone wrote slightly wrong. The pattern I’ve seen repeated across teams: tenant_id scoping gets added correctly at some call sites and quietly dropped at others, because scoping is enforced by convention — “remember to filter by tenant” — instead of by the shape of the code.

The fix is the same discipline I laid out for the has_many/belongs_to split in Ecto: put the rule where a call site can’t accidentally skip it, not in a comment reminding the next engineer to remember. For tenant scoping specifically, that means the tenant-scoped queryable is the only entry point into the context — there is no unscoped function to reach for by mistake:

defmodule MyApp.RAG.Query do
  import Ecto.Query
  import Pgvector.Ecto.Query

  # The only way to get a Chunk queryable.
  # There is no unscoped version to reach
  # for by accident.
  def for_tenant(tenant_id) do
    from c in MyApp.RAG.Chunk,
      where: c.tenant_id == ^tenant_id
  end

  def nearest(tenant_id, embedding, k \\ 5) do
    for_tenant(tenant_id)
    |> order_by(
      [c],
      cosine_distance(c.embedding, ^embedding)
    )
    |> limit(^k)
  end
end

Every function that touches Chunk starts from for_tenant/1, so tenant_id isn’t a WHERE clause someone remembers to add — it’s a parameter you can’t call the module without providing. Context-boundary functions that assemble the actual LLM prompt — pulling retrieved chunks, prior messages, tenant-specific system prompt fragments — should take the authenticated tenant from your code (the current session, never a model- or client-supplied argument) and pass it straight into for_tenant/1. That’s the same rule from prompt injection defense for AI startups: scope every tool and every query to the session’s user, taken from code you control, not from anything the model or the request body hands you.

The bug that passes code review: storage scoped, retrieval isn’t

Here’s the failure mode that’s specific to RAG, and it’s the one that scares me most because it’s invisible in a diff. Say your ingestion pipeline is disciplined — every chunk is written with tenant_id set, there’s a foreign key, there’s an index, code review would catch a write that skipped it. And say your retrieval function is also well-written Ecto: correct cosine_distance call, correct limit, no N+1, nothing a linter or a reviewer would flag.

# Clean Ecto. Wrong result. Nobody catches
# this in review because nothing about it
# looks wrong.
def nearest_unscoped(embedding, k \\ 5) do
  from(c in MyApp.RAG.Chunk,
    order_by: cosine_distance(
      c.embedding, ^embedding
    ),
    limit: ^k
  )
  |> MyApp.Repo.all()
end

This function is missing exactly one where clause, and that’s the entire bug. It will never error, because top-k cosine similarity across the whole chunks table always returns something — it just returns the nearest chunks from every tenant, not the calling tenant. Storage is scoped perfectly. The write path has a foreign key and an index proving it. The read path — the one query that decides what actually lands in tenant B’s context window — has none of that, because nothing about a missing WHERE shows up as a schema violation, a type error, or a slow query. It shows up as tenant A’s onboarding contract or support transcript answering tenant B’s question, word for word, in a response your product renders as if the model just knew the answer.

This is the same architecture I described in the hand-rolled pgvector post: retrieval is one Ecto query, which is the whole appeal — composable, WHERE-able, no separate metadata-filter DSL to fight. That composability is exactly why the failure is so cheap to introduce: adding a filter is a one-line change, so is forgetting one, and the query that forgot looks identical in a code review to the one that didn’t, unless the reviewer is specifically checking for a tenant_id clause every single time. At scale, “check for it every time” is not a control. A structural rule — one queryable, no unscoped path in — is.

Prove it: the test that has to exist

If tenant isolation on the retrieval path matters — and per the security-questionnaire framing above, it’s often the single question that decides whether a regulated buyer signs — then it needs a test that fails the moment someone reintroduces nearest_unscoped/2, adds a join that drops the where, or refactors the query module and loses the guarantee. Not a manual QA pass before launch. A test that runs on every commit, forever.

defmodule MyApp.RAG.QueryTest do
  use MyApp.DataCase, async: true

  alias MyApp.RAG.Query

  test "retrieval never crosses tenants" do
    tenant_a = insert(:tenant)
    tenant_b = insert(:tenant)

    insert(:chunk,
      tenant_id: tenant_a.id,
      body: "tenant a's onboarding contract"
    )
    insert(:chunk,
      tenant_id: tenant_b.id,
      body: "tenant b's support policy"
    )

    [embedding] = embed!(["onboarding"])

    leaked =
      Query.nearest(tenant_b.id, embedding, 10)
      |> MyApp.Repo.all()
      |> Enum.filter(
        &(&1.tenant_id == tenant_a.id)
      )

    assert leaked == []
  end
end

Stub embed!/1 to a deterministic fixture so this doesn’t hit a real embeddings API in CI — the assertion doesn’t depend on which embedding you get back, only on whether the result set ever contains a row from tenant_a. That’s the whole test. It’s cheap to write, it runs in milliseconds, and it’s the one piece of evidence that turns “we scope by tenant” from a claim in a security questionnaire into something you can point at. A reviewer who asks the tenant-isolation question and gets shown this test, passing, is having a very different conversation than one who gets shown an architecture diagram.

Where this actually gets tested is the interview, not the demo

The demo will never expose this bug, because a demo has one tenant, or a handful of friendly ones with nothing sensitive in the same corpus. It surfaces in production, at the exact moment two real customers’ documents happen to be semantically close enough that the top-k result set for one includes the other — and the person who finds it first is either your customer’s security team during an enterprise deal, or, worse, the customer themselves, reading an answer in their chat window that quotes language they’ve never seen before and don’t recognize.

Cross-tenant retrieval leakage is also a sharper problem than the prompt-injection trifecta I’ve written about elsewhere, precisely because it needs no attacker. Simon Willison’s framing — private data, untrusted content, and external communication converging in one context — describes an adversary exploiting a gap. This doesn’t require anyone malicious. It requires one missing WHERE clause and a corpus large enough that the gap gets exercised by ordinary traffic. That makes it a baseline hygiene problem you solve before you get anywhere near adversarial threat modeling, not an advanced case you defer until you have a security hire.

What this buys you in the room that matters

Go back to the questionnaire line this post opened with: does one customer’s data ever influence another customer’s outputs? Most AI startups answer it with an architecture diagram and a sentence about Postgres row-level scoping, and a reviewer who’s read a few of these knows that answer describes the write path and says nothing about the read path — which is exactly where the gap in this post lives. The better answer is structural: one scoped queryable per tenant-touching resource, no unscoped path into it, and a test in CI that fails the day someone adds one. That’s not a bigger engineering lift than the naive version — it’s roughly the same code, written so a call site can’t skip the part that matters, plus one test file.

If you’re staffing this now, or you’ve got an enterprise security review on the calendar and you’re not confident your retrieval path would survive the tenant-isolation question asked directly, that’s exactly the kind of architecture review I run as a fractional CTO for AI startups. Let’s talk about what that looks like for your stack before the reviewer asks the question you haven’t tested the answer to.