Streaming LLM Tokens in LiveView, the 2026 Way

Jared Smith July 14, 2026 15 min read Elixir Phoenix AI

Token-by-token LLM streaming in Phoenix LiveView, no React — the 2026 async APIs, plus the production failure modes most tutorials skip.

TL;DR: You do not need React, a WebSocket microservice, or a client-side state library to stream an LLM’s response token-by-token into your UI. Phoenix LiveView already holds an open connection to the browser, and a spawned process feeding tokens back over it is all the plumbing you need. The naive version — accumulate each token into an assign and re-render — works beautifully in a demo and falls over in production for three specific reasons: it keeps paying the model when the user navigates away mid-stream, it re-renders your entire template on every single token, and it leaves a half-finished message on screen when the upstream call dies. This post shows the modern build using LiveView 1.1’s async primitives and Req’s streaming :into callback, then fixes all three failure modes the tutorials skip.

The connection is already open — stop reaching for a second one

Every few weeks someone building their first AI feature in Phoenix asks the same question: “How do I stream the model’s tokens into the UI as they arrive?” And almost every answer they find points them somewhere strange — spin up a Channel, bolt on a React island, proxy the OpenAI stream through a Node sidecar.

None of that is necessary. A LiveView is a stateful server process with a persistent connection to exactly one browser tab. That is the entire problem already solved. The model emits tokens; you push them down the wire that is already open. The only real work is wiring an HTTP streaming response on one side to that socket on the other, and doing it without lying to yourself about what happens when things go wrong.

If you want the deeper argument for why the BEAM is unusually well-suited to this — cheap processes, per-user isolation, supervision — I made it in the BEAM is quietly the best runtime for AI agents and, more fundamentally, in how Elixir’s concurrency model actually works. This post is the concrete, runnable version of that thesis for the single most common AI UI task.

A note on versions: the code below targets Phoenix LiveView 1.1+ and Req 0.5+. The async callbacks (start_async/4, handle_async/3, Phoenix.LiveView.AsyncResult) and the send/2 + handle_info/2 pattern are stable from 1.1 onward; I verified them against the current LiveView docs (1.2.1 at the time of writing) and they are unchanged. Req’s streaming :into callback shape is likewise stable in the current 0.6.x line. The LLM calls target OpenAI’s Responses API (POST /v1/responses) — the current surface as of 2026, not the older Chat Completions endpoint.

The naive version that demos beautifully

Here is the version you will write first, and honestly it is a reasonable starting point. The model’s tokens land in an assign, the assign drives the template, LiveView ships the diff. Token appears. Magic.

defmodule MyAppWeb.ChatLive do
  use MyAppWeb, :live_view

  def mount(_params, _session, socket) do
    {:ok, assign(socket, prompt: "", answer: "")}
  end

  def handle_event("ask", %{"prompt" => prompt}, socket) do
    parent = self()

    # Spawn a task that streams tokens back to us.
    Task.start(fn -> stream_completion(prompt, parent) end)

    {:noreply, assign(socket, prompt: prompt, answer: "")}
  end

  # Each token arrives as a message and gets appended to the assign.
  def handle_info({:token, text}, socket) do
    {:noreply, assign(socket, answer: socket.assigns.answer <> text)}
  end

  defp stream_completion(prompt, parent) do
    Req.post!("https://api.openai.com/v1/responses",
      headers: [{"authorization", "Bearer #{System.fetch_env!("OPENAI_API_KEY")}"}],
      json: %{
        model: "gpt-4o-mini",
        stream: true,
        input: prompt
      },
      into: fn {:data, data}, {req, resp} ->
        for token <- parse_sse_tokens(data), do: send(parent, {:token, token})
        {:cont, {req, resp}}
      end
    )
  end
end

<div class="chat">
  <form phx-submit="ask">
    <input type="text" name="prompt" value={@prompt} />
    <button type="submit">Ask</button>
  </form>
  <div class="answer">{@answer}</div>
</div>

Run that locally, type a prompt, and you will watch text materialize word by word. It feels like you shipped something real. You have not — you have shipped a demo. Here is what it does the moment it meets actual users.

The three ways it breaks the instant it meets real users

1. It keeps paying for tokens nobody will ever see. The user fires a long prompt, watches three sentences stream in, decides it is wrong, and closes the tab. Their LiveView process terminates. But the Task.start process is not linked to it in any way that matters here, and even if it were, Task.start gives you no handle to cancel it. The Req request upstream keeps the socket to OpenAI open and keeps pulling tokens — which you keep being billed for — until the model emits its final token into a mailbox that no longer has a reader. On a chat feature with any volume, this is a line item.

2. It re-renders the entire template on every token. socket.assigns.answer <> text replaces the whole answer assign on each message. LiveView is smart about diffing, but you have still made the entire answer string a single changed assign, so the full text re-serializes and re-diffs on every token. A 600-token response is 600 diffs of a string that grows to 600 tokens long — quadratic work for what should be append-only. Add a few concurrent users and you will see the LiveView’s reduction count climb for no good reason.

3. A mid-stream error leaves a corpse on screen. OpenAI returns a 500 four tokens in. Req.post! raises inside the task. The task dies. Your LiveView never finds out — it just stops receiving {:token, _} messages and sits there forever showing half a sentence, no spinner, no error, no retry. The user reloads and tries again, because you gave them nothing else to do.

All three share a root cause: the naive version models the happy path and nothing else. The streaming part was never the hard part. Lifecycle is the hard part. Let’s build the version that takes it seriously.

The right build: own the request, track its lifecycle, append cleanly

The fix starts with one decision: wrap the LLM call behind a project-owned module and give the LiveView a real handle to the work in flight. We use start_async/4 so LiveView supervises the task and tells us — via handle_async/3 — when it finishes or crashes. Inside that task, the streaming :into callback forwards tokens back to the LiveView pid as they arrive. The async result is the terminal signal (done, or failed); the send/2 messages are the incremental stream. That split is the whole design.

First, the boundary module. Never call Req against a third-party API directly from a LiveView — wrap it so the streaming protocol, auth, and SSE parsing live in one testable place:

defmodule MyApp.LLM do
  @moduledoc "Project-owned boundary around the streaming completions API."

  @endpoint "https://api.openai.com/v1/responses"

  @doc """
  Streams a completion, sending `{:llm_token, ref, text}` messages to `dest`
  as tokens arrive. Returns `:ok` on a clean finish or raises on transport
  error (so the caller's async task surfaces it as `{:exit, reason}`).
  """
  def stream(prompt, dest, ref) do
    Req.post!(@endpoint,
      headers: [{"authorization", "Bearer #{api_key()}"}],
      json: %{
        model: "gpt-4o-mini",
        stream: true,
        input: prompt
      },
      # Bound the wait between chunks so a stalled upstream can't hang forever.
      receive_timeout: 30_000,
      into: fn {:data, data}, {req, resp} ->
        for text <- parse_sse(data) do
          send(dest, {:llm_token, ref, text})
        end
        {:cont, {req, resp}}
      end
    )

    :ok
  end

  # The Responses API streams typed Server-Sent Events: each is an
  # `event: <type>` line plus a `data: {json}` line. Text arrives in
  # `response.output_text.delta` events (the JSON carries a `delta`); the
  # stream ends with a `response.completed` event, not a `[DONE]` sentinel.
  # Matching on the payload's `type` means interleaved events (created,
  # output_text.done, completed) are simply ignored. Real code should
  # buffer partial lines across chunks; kept inline here for readability.
  defp parse_sse(chunk) do
    chunk
    |> String.split("\n")
    |> Enum.flat_map(fn
      "data: " <> json ->
        case Jason.decode(json) do
          {:ok, %{"type" => "response.output_text.delta", "delta" => text}} -> [text]
          _ -> []
        end
      _ -> []
    end)
  end

  defp api_key, do: System.fetch_env!("OPENAI_API_KEY")
end

Now the LiveView. Note Phoenix.LiveView.AsyncResult driving the UI state machine, and that we accumulate tokens into an IO list, not by repeatedly concatenating a string:

defmodule MyAppWeb.ChatLive do
  use MyAppWeb, :live_view
  alias Phoenix.LiveView.AsyncResult

  def mount(_params, _session, socket) do
    {:ok,
     socket
     |> assign(:prompt, "")
     |> assign(:stream_ref, nil)
     |> assign(:tokens, [])          # IO list of received tokens
     |> assign(:completion, AsyncResult.loading() |> AsyncResult.ok(:idle))}
  end

  def handle_event("ask", %{"prompt" => prompt}, socket) do
    # A ref lets us ignore stray tokens from a request we've since abandoned.
    ref = make_ref()
    pid = self()

    socket =
      socket
      |> assign(:prompt, prompt)
      |> assign(:stream_ref, ref)
      |> assign(:tokens, [])
      |> assign(:completion, AsyncResult.loading())
      # start_async supervises the task and routes its result to handle_async/3.
      |> start_async(:completion, fn ->
        MyApp.LLM.stream(prompt, pid, ref)
      end)

    {:noreply, socket}
  end

  # Incremental: one token arrived. Only append if it belongs to the
  # current request — a late token from an abandoned stream is dropped.
  def handle_info({:llm_token, ref, text}, %{assigns: %{stream_ref: ref}} = socket) do
    {:noreply, update(socket, :tokens, &[&1 | text])}
  end

  def handle_info({:llm_token, _stale_ref, _text}, socket) do
    {:noreply, socket}  # token from a cancelled/superseded request
  end

  # Terminal: the task returned cleanly. The stream is done.
  def handle_async(:completion, {:ok, :ok}, socket) do
    {:noreply, assign(socket, :completion, AsyncResult.ok(socket.assigns.completion, :done))}
  end

  # Terminal: the task crashed mid-stream (HTTP 500, timeout, etc).
  def handle_async(:completion, {:exit, reason}, socket) do
    {:noreply, assign(socket, :completion, AsyncResult.failed(socket.assigns.completion, reason))}
  end
end

<div class="chat">
  <form phx-submit="ask">
    <input type="text" name="prompt" value={@prompt} />
    <button type="submit" disabled={@completion.loading}>Ask</button>
  </form>

  <div class="answer" id="answer">{@tokens}</div>

  <.async_result :let={_status} assign={@completion}>
    <:loading><span class="cursor">▌</span></:loading>
    <:failed :let={_reason}>
      <p class="error">
        The model dropped the connection.
        <button phx-click="ask" phx-value-prompt={@prompt}>Retry</button>
      </p>
    </:failed>
  </.async_result>
</div>

This is already a different animal. The start_async task is supervised by the LiveView; if the LiveView dies, the task dies with it. handle_async gives us a real terminal event, so a crashed stream flips the UI into a failed state with a retry button instead of freezing. The ref guard means tokens from an abandoned request are silently dropped instead of corrupting the next answer.

But “the task dies with the LiveView” is necessary, not sufficient. Killing the task does not, by itself, hang up on OpenAI. That is failure mode one, and it deserves its own section.

Cancellation done right: hang up on the model, not just the socket

Here is the subtle, expensive part. When start_async’s task is killed — because the LiveView terminated, because the user hit “stop,” because they navigated away — the task process disappears. But the Req request running inside that task opened a TCP connection to OpenAI and is mid-stream. Does killing the task close that connection?

It depends entirely on whether the HTTP work happens in the task’s own process. With Req’s default function-form :into (the shape above), the request is driven synchronously inside the calling process — the start_async task itself. When the BEAM kills that process, the socket it owns is closed as part of process teardown, which tears down the upstream connection. That is the behavior you want, and it is the reason to keep the Req call inside the start_async function rather than handing it off to some longer-lived pool worker.

The trap is into: :self. That mode spawns a separate process to pump the socket and mails chunks to you. Kill your task and that helper can outlive it, happily draining (and billing) the full response into a mailbox nobody reads. For LiveView token streaming, prefer the function-form :into so the request’s lifetime is bound to the task’s lifetime. Cancellation becomes free: it is just process death.

For the explicit “stop generating” button, cancel the named async and let teardown do the rest:

def handle_event("stop", _params, socket) do
  # cancel_async kills the supervised task; its Req socket closes on teardown,
  # which closes the upstream connection. We also clear the ref so any
  # tokens already in our mailbox are ignored on arrival.
  {:noreply,
   socket
   |> cancel_async(:completion)
   |> assign(:stream_ref, nil)
   |> assign(:completion, AsyncResult.ok(socket.assigns.completion, :cancelled))}
end

For the navigate-away case you usually do not need to write anything: when the client disconnects, the LiveView process terminates, which kills its start_async task, which closes the Req socket. The chain holds as long as the HTTP request lives inside the supervised task. If you have moved the LLM call into a separate long-lived GenServer or a pooled worker — a reasonable thing to do for other reasons — then you have severed that chain and you are back to leaking requests. In that case you must trap the LiveView’s terminate/2 (or monitor the LiveView pid from the worker) and explicitly cancel the in-flight request. The general rule: the process that owns the HTTP socket must die when the user goes away. Architect so that happens for free, or wire it up explicitly. There is no third option that does not leak money.

If you want belt-and-suspenders confirmation in dev, watch your provider dashboard’s active-request count while you spam open-and-close on a long prompt. If it climbs and never settles, your cancellation chain is broken somewhere.

Backpressure and the re-render storm — and the actual fix

Now failure mode two: re-rendering the world on every token. The naive version stored the answer as one big string and rebuilt it per token. Two things fix this, and they compose.

Append, don’t concatenate. The right build above stores tokens as an IO list: [&1 | text] conses the accumulated iodata with each new fragment in O(1) and renders in arrival order — no Enum.reverse needed. The point is you are no longer rebuilding an ever-growing binary on the server every message the way answer <> text did; that concatenation was the quadratic cost. Shrinking what actually goes over the wire on each diff is the next paragraph’s job.

Render the streamed region with phx-update="ignore" and let the client own appends, or batch on the server. Two legitimate strategies:

The server-light approach adds phx-update="ignore" to that answer div: LiveView stops diffing the node after first render, and a small client hook appends incoming token text directly to the DOM. The server still sends tokens, but it is not re-diffing a growing block on every one. This is the lowest-overhead option for high token rates.

The server-only approach, if you would rather not write a hook, is to batch tokens before assigning. Do not push one diff per token — buffer tokens for, say, 50ms or 20 tokens, whichever comes first, then flush one update. The user cannot perceive sub-50ms batching, but your reduction count drops by an order of magnitude:

# In the LiveView: accumulate, and flush on a short timer instead of per-token.
def handle_info({:llm_token, ref, text}, %{assigns: %{stream_ref: ref}} = socket) do
  socket = update(socket, :buffer, &[&1 | text])
  # Schedule a single flush if one isn't already pending.
  socket =
    if socket.assigns.flush_pending do
      socket
    else
      Process.send_after(self(), {:flush, ref}, 50)
      assign(socket, :flush_pending, true)
    end

  {:noreply, socket}
end

def handle_info({:flush, ref}, %{assigns: %{stream_ref: ref}} = socket) do
  {:noreply,
   socket
   |> update(:tokens, &[&1 | socket.assigns.buffer])
   |> assign(:buffer, [])
   |> assign(:flush_pending, false)}
end

Either way, the principle is the same: the rate at which the model emits tokens should not equal the rate at which you re-render. Decouple them. A model that emits 80 tokens/second does not need 80 LiveView diffs/second; it needs a smooth-looking stream, which 20 batched updates/second delivers indistinguishably while doing a quarter of the work. Multiply that saving across every concurrent chat session and the difference is the number of users one node can hold.

There is no Req-level backpressure knob to reach for here — the model sets the pace and you cannot ask it to slow down. Backpressure in this design means not amplifying the model’s token rate into an equal-or-greater render rate on your side. Batching is how you absorb it.

When NOT to build it this way

I am bullish on LiveView for this. I am not unconditional about it.

Skip LiveView if you genuinely need offline or sub-frame client rendering. A native mobile app, an offline-capable PWA, or anything where the UI must keep updating with the network cut — LiveView’s whole model is a live server connection, so there is no server to stream from when you are offline. Run the model client-side or against a local edge and render in whatever owns the device. LiveView is the wrong layer for that, full stop.

Reconsider at extreme fan-out. If a single generated stream must be delivered to thousands of simultaneous viewers — a broadcast, not a per-user chat — then one LiveView process per viewer, each holding its own assigns, is a lot of duplicated state. That is the case where a dedicated Phoenix.Channel (or a PubSub topic that all viewers subscribe to, with the model streamed once into the topic) earns its complexity. For the overwhelmingly common case — one user, one private generation — that machinery is pure overhead, and the per-user LiveView process is exactly right.

Skip the streaming UI entirely for short, non-interactive completions. If the model returns two sentences in 400ms, token streaming is theater. Show a spinner, await the result with a plain assign_async/4, render it whole. Streaming earns its lifecycle complexity only when the response is long enough that watching it arrive is genuinely better than waiting for it.

Verdict

Streaming LLM tokens into a Phoenix UI is not a React problem, a Channel problem, or a sidecar problem. It is a process-lifecycle problem wearing a streaming-UI costume. The connection you need is already open; the supervision you need is already in the box.

Build it with start_async/4 so the work is supervised and its terminal state is a real event you can render. Stream tokens back over send/2 and append them — as an IO list or via a client hook — so you are not rebuilding the answer on every message. Keep the Req call inside the supervised task so cancellation is free: when the user leaves, the process dies, the socket closes, the model stops billing you. Batch your re-renders so the model’s token rate is not your diff rate. Do those four things and the version that demos well is also the version that survives contact with real traffic.

The naive version is not wrong, exactly. It is just unfinished — it solved streaming and skipped lifecycle, which is the actual job. Finish it.

Two follow-ups are coming soon: running these generations as durable, retryable Oban jobs when they outlive a page view, and the RAG retrieval layer that feeds the prompt before any of this streaming begins. Both build directly on the boundary module and supervision patterns above.

All writing