Building an LLM Audit Trail That Passes SOC 2

TL;DR: Every SOC 2 guide that mentions LLMs stops at policy language — “log access to AI systems” — and never shows you the table. Here’s the table. An append-only llm_audit_events schema in Ecto, the exact fields that make an entry defensible in an auditor’s sample versus decorative, and the field most teams skip that ends up saving them in the room: the provider’s own request ID. Also covered honestly: whether you have to log the raw prompt (usually yes, with a redaction step, not a blanket no), and how long to keep any of it before “audit trail” becomes “unmanaged PII warehouse.”

The instrumentation point is the same call, different consumer

Metering LLM usage per customer in Phoenix covers the append-only ledger you build to bill a tenant for token usage — integer micro-dollars, idempotent writes keyed on a provider request ID, never :float. This post instruments the same outbound call, but the record it produces answers a different question and lives under different rules. A billing ledger exists to defend an invoice line for as long as you keep customers; it can be thin — tenant, tokens, cost, done. An audit trail exists to defend a claim about control, to a third party you didn’t choose, on a schedule you don’t control, and it has to survive scrutiny of who called what, with what input, under whose authority, for as long as your retention policy says it does. Same :telemetry hook. Different table, different columns, different retention clock.

There’s a third consumer of that same call worth separating out. The agent evidence pack for a security review specifies the tool-call record as a reviewer-facing artifact — the field set you hand someone evaluating your product, and how to generate it so it can’t drift from the code. This post is the storage side of that record: the schema it lives in, the retention clocks it runs on, and what an auditor pulls out of it. Read that one for what to show; read this one for what to build.

If you haven’t wired the underlying telemetry yet, the ReqLLM integration walkthrough covers the event shapes this post assumes — specifically [:req_llm, :request, :stop], whose metadata already carries request_id, provider, model, and finish_reason before you’ve written a line of your own code.

What SOC 2 actually asks you to log

The reframe I give founders heading into their first SOC 2 is that there is no “AI” criterion to satisfy. There are criteria that predate LLMs by a decade and apply to them without modification, and knowing which ones is what separates a defensible answer from a guess in the audit room. SOC 2 as a revenue tool rather than a security project is why you’re in the room at all; this is the artifact that has to survive once you’re there.

CC7.2 (System Operations) requires the entity to monitor system components for anomalies and analyze them to determine whether they’re security events — this is the criterion an LLM call log most directly satisfies, because a prompt-injection attempt or a call from a deprovisioned service account is the anomaly it’s asking you to be able to detect. The criterion’s own words are “monitors system components and the operation of those components for anomalies that are indicative of malicious acts, natural disasters, and errors” (AICPA, TSP Section 100, 2017 Trust Services Criteria). CC6.1 and CC6.3 (Logical and Physical Access Controls) govern who’s authorized to trigger the call in the first place, which is why “calling user or tenant” is a required column, not an optional one — an audit log with no actor attached can’t prove access was restricted to authorized parties. CC7.3 (System Operations) picks up where CC7.2 stops: it covers evaluating the security events CC7.2 surfaced to determine whether they rise to security incidents — a failure to meet your objectives — and taking action if they do. That evaluation only works if the log entry carries enough context (input, output, timestamp, actor) to judge against.

None of that is AI-specific. What’s new is the volume and the surface area: a REST API you called twelve times a day now gets called on every user keystroke, through an SDK whose retry behavior you don’t fully control, against a vendor whose model version changes under you. The criteria didn’t change. The number of events that need to satisfy them did. Controlling what you send in the first place is the other half of that: throttling against the provider’s real TPM and RPM ceilings keeps the 429 from happening at all.

An append-only Ecto schema for LLM audit events

Same discipline as the billing ledger — no update, ever, corrections are new rows — but a wider column set, because the auditor isn’t sampling for a dollar amount, they’re sampling for a story: who did what, to which model, with what result.

create table(:llm_audit_events,
             primary_key: false) do
  add :id, :binary_id, primary_key: true
  add :occurred_at, :utc_datetime_usec,
      null: false
  add :actor_id, :binary_id, null: false
  add :actor_type, :string, null: false
  add :tenant_id, :binary_id, null: false
  add :provider, :string, null: false
  add :model, :string, null: false
  add :provider_request_id, :string,
      null: false
  add :input_tokens, :integer, null: false
  add :output_tokens, :integer, null: false
  add :latency_ms, :integer, null: false
  add :tool_calls, {:array, :map},
      default: []
  add :finish_reason, :string
  add :prompt_digest, :string, null: false
  add :prompt_redacted, :text
  add :outcome, :string, null: false

  timestamps(updated_at: false)
end

create unique_index(
  :llm_audit_events,
  [:provider, :provider_request_id],
  name: :llm_audit_events_provider_request_idx
)
create index(
  :llm_audit_events,
  [:tenant_id, :occurred_at]
)
create index(
  :llm_audit_events, [:actor_id]
)

prompt_digest (a SHA-256 of the raw prompt) always gets written, even when prompt_redacted doesn’t — it lets you prove which prompt produced a given output, and lets you re-associate a customer complaint with an exact call, without keeping the raw text around by default. More on that split below.

The Ecto schema, with the invariant a future reader needs written where they’ll see it:

defmodule MyApp.Audit.LLMEvent do
  use Ecto.Schema
  import Ecto.Changeset

  @primary_key {:id, :binary_id,
                autogenerate: true}
  schema "llm_audit_events" do
    field :occurred_at, :utc_datetime_usec
    field :actor_id, Ecto.UUID
    field :actor_type, :string
    field :tenant_id, Ecto.UUID
    field :provider, :string
    field :model, :string
    field :provider_request_id, :string
    field :input_tokens, :integer
    field :output_tokens, :integer
    field :latency_ms, :integer
    field :tool_calls, {:array, :map},
      default: []
    field :finish_reason, :string
    field :prompt_digest, :string
    field :prompt_redacted, :string
    field :outcome, :string

    timestamps(updated_at: false)
  end

  # append-only: no update_changeset/2 exists
  # on this module. Corrections are new rows
  # with :outcome set to "superseded".

  @fields ~w(occurred_at actor_id actor_type
             tenant_id provider model
             provider_request_id input_tokens
             output_tokens latency_ms
             tool_calls finish_reason
             prompt_digest prompt_redacted
             outcome)a

  def changeset(event, attrs) do
    event
    |> cast(attrs, @fields)
    |> validate_required(@fields -- [
         :prompt_redacted, :finish_reason
       ])
    |> unique_constraint(
         :provider_request_id,
         name: :llm_audit_events_provider_request_idx
       )
  end
end

Writing this from inside the same call wrapper the billing ledger uses is deliberate, not laziness — one project-owned MyApp.LLM.call/3 that requires actor_id and tenant_id as arguments, emits one billing row and one audit row from the same :telemetry handler, off the same event. Two consumers, one instrumentation point, no code path that reaches the provider without both attached.

Why the provider request ID is the field that saves you

The provider request ID is the join key that lets you prove your log matches the vendor’s log, which is the difference between an entry an auditor believes and one they don’t. Every major provider stamps a request ID on its response — x-request-id from OpenAI, request-id from Anthropic, x-amzn-requestid from Bedrock — and ReqLLM already surfaces it in the [:req_llm, :request, :stop] metadata as request_id, so capturing it costs you nothing beyond a column. One gotcha carried over from that post: model arrives as an LLMDB.Model struct rather than a string, so take .id before it reaches the changeset.

Here’s the failure mode it prevents. An auditor doing Type 2 testing doesn’t take your word for a control’s operating effectiveness across the review period — they pull a sample of events and check each one has proof. If their sample includes an LLM call and your row has no way to independently verify it happened as described, you’re asking them to trust your application’s own account of itself, which is close to the definition of an unauditable control. A provider_request_id gives a reviewer something to check that you didn’t author. How strong that check is depends on the provider. On Bedrock it’s the best case: model invocation logging writes each call’s requestId, calling principal ARN, and input/output token counts to CloudWatch Logs or S3 — though it is disabled by default, so turn it on before you need it rather than after. With OpenAI and Anthropic the path is narrower: the ID is what their support resolves a disputed call against, and your token counts still reconcile against the usage they invoice you for. Either way your row stops being a claim and becomes a claim with a witness.

It’s also the field that makes incident response tractable under CC7.2/CC7.3: “this model call misbehaved” becomes “here’s the exact provider-side request, escalate it to their support with this ID” instead of “here’s an application log we hope corresponds to something.”

Do you have to log the prompt?

Usually yes, but not the raw text by default — you need enough to reconstruct what happened without warehousing customer PII you have no retention plan for. The prompt_digest / prompt_redacted split is the practical answer: hash every prompt unconditionally (cheap, reversible only with the original text, and enough to prove which input produced which output), and only store redacted or truncated prompt text when your data-handling policy explicitly allows it for that tenant and that data class.

Redaction in practice means running a PII scrubber (structured field masking for known formats — emails, SSNs, card numbers — plus a conservative regex pass) on the prompt before it’s written to prompt_redacted, and treating anything you can’t confidently classify as “don’t store it, keep the digest.” This is a judgment call your data classification policy should make explicit, not one an engineer should default on ad hoc per feature — which is the same governance gap running security at an AI-native company argues costs teams the most when it’s left implicit.

The trap to avoid is the binary version of this decision: “we log everything” turns your audit trail into the biggest unencrypted PII surface in the company, and “we log nothing” turns it into an audit trail an auditor can’t actually use. The digest-plus-conditional-redaction split is what lets both things be true — you can prove exactly which prompt produced a disputed output when you’re authorized to look, and you’re not carrying raw customer text you never needed for anything but that one dispute.

Retention: the tension nobody writes down

Retain audit records only as long as your documented policy says, and no policy should say “forever” — SOC 2 doesn’t mandate a duration. If your report includes the Confidentiality or Privacy categories, C1.1 and P4.2 put you on the hook to define a retention period and dispose on it, and an auditor will check your documented period against what’s actually in the table. If you scoped Security only — which most startups do — nothing forces the question, which is exactly why teams never answer it and end up with a five-year-old prompt table nobody owns. A common, defensible split: prompt_digest, token counts, actor, and provider_request_id (the audit-value columns) live for the length of your audit window plus a buffer — 13 months covers a Type 2 period with room to spare — while prompt_redacted gets a shorter clock, often 30–90 days, because raw text is the highest-liability column in the table and the one with the least ongoing audit value once the immediate incident-response window has closed.

Build the shorter clock as an actual job, not a policy document nobody automates:

defmodule MyApp.Audit.PurgeRedactedPrompts do
  use Oban.Worker, queue: :maintenance

  import Ecto.Query

  @retention_days 60

  @impl Oban.Worker
  def perform(_job) do
    cutoff = DateTime.utc_now()
    |> DateTime.add(-@retention_days, :day)

    from(e in MyApp.Audit.LLMEvent,
      where: e.occurred_at < ^cutoff,
      where: not is_nil(e.prompt_redacted)
    )
    |> MyApp.Repo.update_all(
         set: [prompt_redacted: nil]
       )

    :ok
  end
end

Notice this nulls prompt_redacted and leaves everything else in the row — the digest, the actor, the request ID, the token counts all survive, because those are the columns that carry audit value across the full retention window. The record isn’t deleted; the liability is.

This is the one sanctioned mutation of an otherwise append-only table, and it’s worth writing down as an invariant rather than rediscovering it during an audit: the purge job may null prompt_redacted and nothing else. No column an auditor samples is ever rewritten. If you want the database holding that line instead of code review, a column-level grant or a Postgres rule will do it.

What does an auditor actually ask for?

A sample of events across the review period, and for each one: proof the actor was authorized, proof the call happened as logged, and proof an anomaly (if any occurred) was evaluated per your documented process. What makes a sampled row pass or fail comes down to whether each field is independently checkable or just asserted.

A row missing provider_request_id or actor_id isn’t wrong, exactly — it’s unfalsifiable, and an auditor’s job is to reject unfalsifiable evidence.

What makes an entry defensible instead of decorative

The short version: every field a stranger could independently check, versus every field that only your own application vouches for. A log statement that says "LLM call succeeded" in your Sentry breadcrumbs is decorative — it’s true only because your code says it’s true. A row with a provider request ID, an authenticated actor, a token count that reconciles against the vendor’s own invoice, and a timestamp precise enough to sequence against a rate-limit event is defensible, because three of those four fields can be checked against a source you don’t control.

This is the same distinction security controls that assume a team you don’t have makes about controls generally — a policy document is decorative until something makes it operational, and an audit trail is the specific place that gap shows up first, because it’s the artifact the auditor actually opens. If you’re heading into a SOC 2 process for the first time and want a second set of eyes on which controls in your stack are load-bearing versus theater before the auditor tells you, that’s exactly the kind of gap review I do as a fractional CTO / vCISO for AI startups — cheaper to find in a working session than in a finding.

Read this next

If SOC 2 readiness for an AI product is new territory generally, surviving enterprise security reviews as an AI startup covers the broader review a prospect’s security team runs before signing, of which an audit trail like this one is a single, load-bearing piece.