Real-Time Agents: streaming, barge-in, and session state that doesn’t collapse

Tool use gave agents hands.

RAG gave them memory (the grounded kind).

But in November, the hard truth shows up:

Real-time agents are not a model problem.
They’re a latency + state + cancellation problem.

The moment you add streaming audio and allow the user to interrupt mid-sentence, you’re no longer “calling an LLM”.

You’re operating a conversation machine whose parts run at different speeds, fail independently, and must remain coherent anyway.

The real architecture: three concurrent loops

A real-time agent is three loops running concurrently:

1. Input loop — capture user intent (audio/text), detect turn boundaries, stream partials.
2. Thinking loop — plan, retrieve, call tools, generate response (often streaming).
3. Output loop — speak/render tokens, allow interruption, reconcile partial outputs into a final trace.

If you treat this as a single “request/response”, you’ll ship a demo.

If you treat it as a set of coordinated loops with shared state and cancellation, you can ship a product.

Streaming UX is not “nice to have” — it’s how you buy time

In non-real-time systems, latency is hidden behind a spinner.

In real-time conversation, a spinner is social death.

Streaming buys you time in three ways:

• Perceived responsiveness: the user sees progress immediately.
• Progressive commitment: you can emit “I’m doing X…” before you finish doing X.
• Early error surfacing: tool failures and missing permissions can be shown before the user waits 10 seconds.

Barge-in is a product requirement (and a cancellation contract)

Voice agents feel magical… until the user can’t interrupt.

In real conversation, interruption is normal:

• “Wait—no, not that…”
• “Actually, use my work email.”
• “Stop, I already know.”

If your agent can’t stop speaking and adapt, it feels less intelligent than a phone menu.

The architectural rule

Every output must be cancellable, and cancellation must be observable.

That means:

• cancel TTS playback immediately
• cancel any in-flight model generation
• cancel in-flight tool calls when possible
• and record what happened in the session timeline

Session state that doesn’t collapse: build a timeline, not a blob

Most agent demos treat “conversation state” as a blob of messages.

Real-time agents can’t.

Because streaming and barge-in create partial, overlapping reality:

• partial transcript updates (ASR (Automatic Speech Recognition) revisions)
• partial model output (tokens)
• partial tool results
• partial audio playback
• cancellations and restarts

So instead of a blob, you need a session timeline: an append-only sequence of events with strong ordering guarantees.

The mental model

A session is a timeline of facts.
UI views are projections of that timeline.

That’s the only way to keep audio, text, tools, and UI aligned.

A practical event model (what I actually ship)

Your event model doesn’t need to be perfect.

It needs to be:

• append-only
• idempotent
• causally linked (what caused what)
• and replayable (for debugging + evaluation)

Here’s a minimal event envelope:

{
  "session_id": "sess_123",
  "event_id": "evt_456",
  "ts": "2024-11-24T10:12:33.120Z",
  "type": "asr.partial",
  "turn_id": "turn_7",
  "parent_event_id": "evt_455",
  "payload": {}
}

And here are the event types that matter most in real-time agents:

Notice what’s missing:

There is no “final answer” field.

There is only a timeline that allows you to reconstruct what happened.

Turn detection: your VAD is part of the product

Turn detection is not an implementation detail.

It decides:

• when to stop listening
• when to start thinking
• when to start speaking
• and when to yield to the user

For voice agents, you typically mix:

• VAD (voice activity detection) to detect speech/silence boundaries
• ASR stability heuristics (how often the partial transcript is changing)
• explicit user controls (tap-to-talk, push-to-talk, “stop” button)

Streaming output: how to avoid talking yourself into a corner

Streaming text is easy. Streaming speech is where mistakes become audible.

Because once you speak, you commit socially — even if you later “correct” yourself.

So you need a simple discipline:

The two-phase response

1. Low-commitment preface (safe, reversible)
2. High-commitment content (only after you’re confident)

Example:

• “Okay — I’m checking your last invoice…” (safe)
• “Your October invoice is 82.50€” (committed, must be correct)

This is the same “truth boundary” idea from 2023, now applied to realtime.

The cancellation ladder: what to cancel first (and what to never auto-cancel)

When barge-in happens, you need a deterministic order of operations.

Here’s the cancellation ladder I use:

Side effects: design commit points like you’re processing payments

If tools can do real work (send, buy, delete, book), real-time makes safety harder.

Your agent needs explicit commit points.

A practical pattern:

• LLM proposes an action (structured)
• UI confirms (or policy auto-confirms only for low-risk scopes)
• only then do you commit side effects
• and you record the commit with an idempotency key

Example:

{
  "type": "action.proposed",
  "payload": {
    "action": "send_email",
    "to": "billing@company.com",
    "subject": "Invoice request",
    "body": "..."
  }
}

Then:

{
  "type": "action.confirmed",
  "payload": { "confirmation_id": "conf_901" }
}

Then:

{
  "type": "action.committed",
  "payload": { "idempotency_key": "send_email:sess_123:turn_7:conf_901" }
}

This is not “compliance theater”.

It’s how you avoid the worst failure mode in real-time agents:

The user interrupts…
but the agent still does the thing.

Session state: split it on purpose

“State” becomes unmanageable when everything shares the same memory object.

So split it.

I like four layers:

This separation is what prevents “session collapse”:

• ephemeral state can glitch without corrupting the session
• projections can be re-derived after reconnect
• the timeline gives you debuggability and evaluation

Transport choices: SSE vs WebSockets vs WebRTC (the practical view)

You can build real-time agents with multiple transports. The choice is less ideological than it looks:

• SSE: simplest for streaming text tokens server → client (one-way).
• WebSockets: good for full-duplex text + control messages (cancel, ack, state).
• WebRTC: best for low-latency audio with jitter handling, but operationally heavier.

The important part is not which transport you pick.

It’s that your session timeline and cancellation semantics are consistent across them.

Failure modes you must design for (real-time edition)

Observability: your latency budget needs a breakdown chart

If you can’t break down latency, you can’t improve it.

Log per-turn timing like a trace:

• input capture → turn end
• turn end → model first token
• model first token → tool call
• tool call → tool result
• tool result → first committed claim
• first committed claim → first audio sample
• cancel.requested → cancel.ack

The punchline: real-time agents are a convergence problem

Streaming and barge-in create parallel realities:

• what the user said
• what the model thought they said
• what the agent started to say
• what was actually spoken
• what tools actually did

Your architecture must force all of that to converge back into a coherent session record.

And the only reliable way I’ve found is:

1. timeline as source of truth
2. projections for UI
3. explicit commit points
4. end-to-end cancellation with acknowledgements

That’s what turns a “voice demo” into an operable system.

Resources

What’s Next

Real-time agents increase surface area:

• more sessions
• more integrations
• more tools
• more security boundaries
• more protocol choices

So the next step is inevitable:

Standards.

Next month: Standards for the Agent Ecosystem: connectors, protocols, and MCP