Governing AI agents in healthcare operations
In healthcare the governing question is not what the agent can do — it is where the clinical line sits and whether the agent can be proven never to have crossed it. Most enterprise healthcare agents are administrative. Keeping them administrative is a control problem, not a prompt problem.
Get an Agent Trust Gap BriefThe line that matters
Administrative automation — scheduling, eligibility, prior authorisation paperwork, appointment reminders, records retrieval — sits in a very different regulatory position from anything that informs clinical judgement. The risk is not that a team deliberately builds a diagnostic agent. It is that an administrative agent drifts into advice because a patient asked a question and the model answered.
Scope is a control, not an instruction. "Do not give medical advice" in a system prompt is not a control — it is a hope. A control is a policy gate that inspects the outbound action and blocks it, plus a refusal receipt proving it was blocked.
The evidence you want is not an absence of complaints. It is a countable set of refusals showing the boundary was tested and held.
What the agent is actually allowed to do
Governance starts by writing down the actions. Not capabilities, not integrations — the specific irreversible things this agent can cause to happen in the world, and what has to be true before each one is permitted.
| Action | Regulated surface | Governing question |
|---|---|---|
| Retrieve patient records | PHI under HIPAA | Minimum necessary. Which records, for which purpose, logged how? |
| Check eligibility / benefits | PHI | Access scoped to the encounter, not the population. |
| Schedule or reschedule an appointment | Operational | Reversible, but patient-visible. Confirmation is a commitment. |
| Send a patient message | PHI + scope risk | Cannot be unsent, may contain PHI, and is where clinical drift happens. |
| Submit prior authorisation | Payer-facing | A representation to a third party. Accuracy and attribution both matter. |
| Anything that could be read as clinical advice | Out of scope | Must be blocked by policy and evidenced by a refusal receipt. |
How it fails
These are the failure modes that produce incidents in this workload specifically. Each maps to a control, not to a warning in a runbook.
- Scope drift into advice. A patient asks a clinical question inside an administrative thread and the agent answers helpfully. Control: an outbound policy gate that classifies and blocks, producing a refusal receipt.
- Over-broad PHI retrieval. The agent is given a records API with no scoping, so every run technically accesses far more than the encounter requires. Control: least-privilege credentials scoped per encounter, not per system.
- PHI in the wrong channel. The agent includes clinical detail in an SMS reminder. Control: channel-aware content policy enforced before transmission.
- Unattributable submissions. A prior authorisation is filed and nobody can establish which run produced it or on what basis. Control: execution receipts binding the submission to inputs, policy and approver.
- Retention mismatch. Agent evidence is purged on a 90-day infrastructure schedule while the underlying obligation runs years. Control: evidence retention set by obligation, not by log rotation.
The controls that hold
Hard scope boundary
An enforced classification gate on outbound content, not a prompt instruction. Anything reading as clinical guidance is blocked before transmission and the block is recorded.
Refusal receipts
Every block produces a signed artifact. A countable record of refusals is the strongest evidence that a boundary is real, and it is what you show when asked how you know the agent stayed administrative.
Minimum-necessary access
Credentials scoped to the encounter. The agent should be structurally unable to read the population when it needs one record.
Channel-aware content policy
What may appear in an SMS differs from a portal message. The policy knows the channel and enforces accordingly.
Human approval on payer-facing submissions
Anything representing facts to a payer passes an approval gate and carries the approver's identity in the receipt.
Obligation-driven retention
Evidence retention is set by the longest applicable obligation, not by whatever the logging tier defaults to.
Common questions
Can an AI agent handle patient scheduling under HIPAA?
Administrative scheduling is a common and defensible use, provided access follows minimum necessary, PHI handling is controlled per channel, and the interaction is logged. The compliance work is in scoping and evidence, not in the scheduling itself.
How do you stop a healthcare agent giving medical advice?
With an enforced policy gate on outbound content that classifies and blocks, backed by refusal receipts. A system-prompt instruction is not a control because it produces no evidence and fails silently.
What evidence would a regulator ask for?
Who accessed what PHI and why, on what authority, and what the agent did with it. Per-action receipts answer that directly; application logs usually do not.
Is prior authorisation automation risky?
It is a representation to a third party, so accuracy and attribution matter. Run it behind an approval gate and keep receipts binding each submission to its inputs and approver.
How long should healthcare agent evidence be retained?
Match the longest applicable obligation in your jurisdiction and payer contracts. The common failure is infrastructure log rotation quietly deleting evidence years before the obligation ends.