The Compliance Cliff: prohibited practices and governance controls that actually ship

February taught us how to test agents.

March teaches the harder lesson:

You can have perfect evals and still ship something you’re not allowed to ship.

This is the “compliance cliff” every team hits the first time an agent leaves the lab:

• the product works
• the metrics look good
• stakeholders are excited

…and then someone asks:

“Are we allowed to do this?”

The uncomfortable truth is that compliance is not a document.

Compliance is runtime behavior + evidence.

So this month I’m going to treat regulation the way an engineer should treat it:

as a set of constraints that must be designed into the system.

What “prohibited practices” means to an engineer

Most compliance regimes have two kinds of rules:

1. Do these things (disclosure, logging, oversight, incident handling)
2. Never do these things (the “prohibited” bucket)

Engineers naturally gravitate to (1) because it sounds like features.

But (2) is where teams get surprised — because prohibited behavior often emerges from composition:

• a model that can persuade
• plus user profiling
• plus high-leverage UI actions
• plus weak oversight

None of those parts are “illegal” in isolation.

Together, they can become a prohibited system behavior.

So the right mental move is this:

Treat “prohibited practices” as system-level failure modes.

You don’t “comply” with them by adding a checkbox.

You comply by preventing the system from entering those states.

The four prohibited-practice traps agents fall into

Prohibited practices are written in legal language, but the operational meaning is usually one of these traps.

These show up in agent systems in very practical ways:

• A sales agent that learns which phrasing maximizes conversion can become a dark-pattern machine without anyone intending it.
• A support agent that profiles sentiment can drift into “emotion recognition” territory if you’re not strict about what signals you ingest and how you use them.
• A “helpful assistant” that suggests actions can become manipulative when paired with tight loops (notifications, “urgent” messaging, fear-based copy).
• A productivity agent deployed in the workplace can cross lines fast if it starts inferring traits or monitoring behavior “for optimization.”

The compliance control plane

If you only learn one pattern from this post, make it this:

Put compliance controls in a separate plane from product code.

Because compliance isn’t one feature.

It’s a cross-cutting set of guarantees:

• who can do what
• under what conditions
• with what disclosures
• with what oversight
• with what evidence

That’s not “an agent prompt.”

That’s a platform layer.

A practical control plane usually has these building blocks:

• Policy-as-code gate (feature flags + rules + jurisdiction + customer tier)
• Capability sandboxing (tool allowlists, scopes, quotas, safe-mode)
• Auditability (structured traces, immutable logs, replay handles)
• Oversight hooks (human approval, escalation, refusal paths)
• Monitoring + kill switches (rate limits, anomaly detection, instant disable)

The goal is not bureaucracy.

The goal is making “what we are allowed to do” enforceable at runtime.

From “rule” to “control”: the translation method

When compliance hands you a rule, you need to translate it into four artifacts:

1. Scope: What features, users, and contexts does it apply to?
2. Invariants: What must never happen?
3. Controls: What prevents or detects invariant violation?
4. Evidence: What proves the controls exist and worked?

This is the same move we used in 2023 for hallucinations:

Define truth boundaries. Enforce them. Measure them.

Now we’re doing it for governance.

The governance controls that actually ship

Below is a minimum viable governance stack that teams can implement without turning product development into molasses.

It’s structured like production architecture: layered defenses, clear owners, and evidence by default.

Designing for “prohibited practices” without guessing the law

Here’s the pragmatic stance:

You don’t need to memorize every line of regulation to build safer systems.

You need a design posture that makes prohibited outcomes difficult.

That posture is:

• don’t build persuasion loops without oversight
• don’t infer sensitive traits unless you absolutely must (and can justify it)
• don’t automate high-impact judgment without human review
• don’t pull in biometric/emotion signals “because it’s easy”

So what should you do on Monday morning?

Build controls around the four traps.

Control patterns for each trap

1) Manipulation at scale → “No persuasion” boundaries

The mistake is treating manipulation as “bad prompts.”

Manipulation is usually an optimization target hidden inside product metrics: conversion, retention, “resolution rate.”

Controls that ship:

• Copy + UX review: treat agent copy like growth copy (red-team it)
• No urgency / scarcity patterns for high-stakes flows
• Persuasion-limiter policies: enforce safe language in certain contexts
• Metrics firewall: never optimize on outcomes that reward coercion (e.g., “close rate” in debt collection)

2) Exploiting vulnerability → context-aware restrictions

The failure mode here is subtle:

A model adapts tone based on signals. Those signals correlate with vulnerability. Now the agent “helps” by applying pressure in the exact places it shouldn’t.

Controls that ship:

• Age / role gating (and don’t pretend you can infer age safely)
• Dependency detection flags (e.g., medical/mental health keywords) → shift to safe-mode
• Hard refusal boundaries (no financial coercion, no self-harm escalation, no medical instructions beyond policy)
• Escalation to humans when vulnerability triggers are hit

3) Automated judgment → separate “assist” from “decide”

Agents are great at summarizing.

They’re dangerous at adjudicating.

Controls that ship:

• Decision separation: the agent can recommend, but a deterministic service decides
• Feature flags: “assist mode” vs “autopilot mode”
• Explanations: the agent must cite reasons and limits, not just outcomes
• Appeals: users can contest and get a human review path

4) Biometric + emotion shortcuts → strict data contracts

The fastest way into a compliance cliff is “we added webcam analysis.”

Controls that ship:

• Data contracts: explicitly ban certain data types at ingestion
• Tool restrictions: no webcam/screenshot OCR unless justified and sandboxed
• Privacy review gates: anything biometric requires a separate approval path
• Model training boundaries: no “we kept the images for later” surprises

Shipping playbook: compliance as a delivery pipeline

Here’s a lightweight process that works in real teams.

It doesn’t require an army of lawyers.

It requires discipline and automation.

Governance that doesn’t collapse under scale

The real reason governance fails is that it’s “someone’s job” instead of “how the system works.”

When you run fleets of agents, you need governance like you need DevOps:

• standardized
• automated
• observable
• owned

A practical operating model looks like this:

• Product owns the use-case envelope and user-impact decisions
• Platform owns capability gating, logging, and runtime controls
• Security/Privacy/Legal own policy requirements and review thresholds
• Everyone owns evidence (because it’s baked into CI and telemetry)

Resources

FAQ

What’s Next

March was about turning “prohibited practices” into controls that can be enforced and audited.

April is where agent platforms start to look real:

Agent runtimes emerge — SDKs, orchestration primitives, and the observability stack you need to run agents as a product, not a demo.