Dissecting ChatGPT: The Product Architecture Around the Model

If you’ve ever tried to “just add an LLM” to a real product, you’ve felt it:

The demo works.

Then reality shows up.

• the model answers confidently with no grounding
• the user asks a question that requires state or permissions
• a tool call succeeds but the user never sees it
• latency spikes, cost spikes, and now you’re in incident mode
• prompt injection lands, and suddenly the model is executing the user’s instructions

This is why I like to say:

ChatGPT is not a model.
It’s a product architecture wrapped around a model.

And that distinction matters, because that architecture is what makes the system feel coherent, safe-ish, and usable.

ChatGPT as a product loop (not a single inference)

Most people picture this:

user prompt → model → answer

Real systems look closer to this:

request → policy → context assembly → model → (tools) → model → response shaping → logging → feedback loop

You can think of it like a modern full-stack runtime:

• a scheduler (when and how the model runs)
• a memory system (what context gets injected)
• an IO layer (tools / retrieval / browsing / code execution)
• a permission system (what the model is allowed to do)
• a debugging surface (traces, token usage, evals, regressions)

The prompt stack is a policy stack

Chat-style systems aren’t “prompted” once. They’re programmed continuously.

In practice, ChatGPT-like products have layers of instruction, typically in this shape:

• System instructions: the product’s non-negotiables (identity, safety posture, formatting, tool rules)
• Developer instructions: the app’s task framing (role, output schema, domain constraints)
• User instructions: the user’s request (which can be malformed, malicious, or ambiguous)
• Tool results: structured state from the outside world (retrieval, APIs, calculations)

The critical point:

Not all instructions are equal.

You need an explicit hierarchy, because users will try to override your system (sometimes accidentally, sometimes not).

Context assembly is the “real work” nobody budgets for

Most failures I see in production are not “bad model”.

They are bad context.

The model can only reason over what you give it in the window:

• missing facts → confident hallucination
• too many facts → the wrong ones win
• stale facts → wrong answers with high confidence
• sensitive facts → privacy incident

So the product needs a context assembly pipeline that behaves like an engineered subsystem, not a prompt hack.

Tools are side effects: treat them like production IO

Tool use is where LLM apps stop being “chat” and start being systems.

If the model can:

• send an email
• trigger a refund
• deploy infrastructure
• modify a database record

…then you are building a stochastic actor in your production environment.

That’s not scary if you treat tool use like any other side-effectful architecture:

capabilities, permissions, idempotency, and observability.

Safety is layered: policy outside the model is not optional

ChatGPT-like products generally rely on multiple layers, because no single layer is perfect.

At a high level, you want defense in depth:

• Input checks: detect disallowed content, sensitive data, injection patterns, weird payloads
• Prompt-level policy: explicit safety posture and tool rules
• Tool sandboxing: scoped credentials, constrained schemas, rate limits, audit logs
• Output checks: refuse, redact, or reformat responses that violate constraints
• Product UX: nudges, confirmations, citations, “I don’t know” patterns

The key mental shift is this:

Safety is an engineering discipline, not a moral posture.

It’s about bounding failure modes.

Observability: treat the LLM layer like a distributed system

By July 2023, my strongest opinion about LLM products is simple:

If you can’t trace it, you can’t ship it.

In classic systems, we ask:

• what request path was taken?
• what dependencies were called?
• where did latency occur?
• what error rates spiked?

LLM systems need the same, plus LLM-specific signals:

• token counts (prompt / completion / total)
• model selection (which model, which version)
• context sources (what was retrieved and why)
• tool calls (which ones, params, outcomes)
• refusal / safety events (what was blocked)
• user feedback (thumbs down, edits, retries)

A reference architecture you can steal

Here’s the “boring” architecture that tends to work.

Not because it’s fancy.

Because it makes responsibilities explicit.

The product decisions that matter more than the model

If you’re building something ChatGPT-like inside your domain, these are the decisions that usually dominate outcome:

Anti-patterns (and how they fail in production)

1) “One big prompt”

You paste policies, product docs, and the user request into a single blob.

It works until it doesn’t:

• the window fills
• the wrong instruction wins
• your policies become “optional”

Fix: split responsibilities:

• policy layer
• context assembly layer
• tool layer
• response layer

2) “The model can just call the API”

You give the model broad access and trust it to behave.

It works until it doesn’t:

• it calls tools with wrong parameters
• it leaks data across tenants
• it performs actions the user didn’t authorize

Fix: tools as capabilities + sandbox + audit + confirmation boundaries.

3) “We’ll add evals later”

You won’t. You’ll drift slowly until you have no idea why quality changed.

Fix: minimum viable eval harness before you scale usage.

Resources

FAQ

What’s Next

Now that we’ve treated ChatGPT as a product runtime, we can talk about the most visible user-facing failure mode of that runtime:

hallucinations.

Next month:

Hallucinations: A Probabilistic Failure Mode, Not a Moral Defect

The goal is to replace outrage with engineering:

• why hallucinations happen structurally
• what makes them worse (bad context, bad incentives, ambiguous tasks)
• and what practical guardrails reduce them in real products