Summary: AI agent security depends on more than prompt instructions or model accuracy. Enterprise systems need enforceable controls over what agents can access, which actions they can perform, how changes are validated and when human approval is required.
Five established principles help explain how to design those controls: least privilege, Kerckhoffs's principle, Chesterton's fence, the Swiss cheese model and Hyrum's Law. Together, they provide a practical way to assess the risks of compromised agents, unintended changes and hidden dependencies.
In This Article:
- Overview: five laws, five design rules
- The Principle of Least Privilege: the credential sets the blast radius
- Kerckhoffs's Principle: assume the system prompt will leak
- Chesterton's Fence: an agent may only delete what it can explain
- The Swiss Cheese Model: independent layers, not more layers
- Hyrum's Law: every consistent output becomes a promise
- What 2025–2026 security guidance confirms about these laws
- Where to start
Overview: five laws, five design rules
AI agent security depends less on how capable the model is and more on what the system lets a wrong agent do. Each of these laws predates large language models, and each one maps to a concrete design rule.
| Law | Origin | Core idea | Design rule for AI agents |
|---|---|---|---|
| Principle of Least Privilege | Jerome Saltzer and Michael Schroeder, 1975 | Use only the privilege the job needs | Scope credentials per agent. The credential, not the prompt, sets the blast radius. |
| Kerckhoffs's Principle | Auguste Kerckhoffs, 1883 | A system must not depend on secrecy | Assume the system prompt will leak, and keep every control outside it |
| Chesterton's Fence | G. K. Chesterton, 1929 | Don't remove what you can't explain | Require a stated purpose before any delete or simplification |
| Swiss Cheese Model | James Reason, 1990 | Accidents happen when holes in several defenses line up | Build independent layers, each designed as if the others have already failed |
| Hyrum's Law | Hyrum Wright, Google | Every observable output gets depended on | Declare which outputs are promises, and version them like an API |
The first four laws limit four things: what an agent may touch, where enforcement lives, what it may remove and what stands behind each control. The fifth covers what happens when an agent, or the model under it, changes.
The Principle of Least Privilege: the credential sets the blast radius
An AI agent's worst possible action is fixed by the credentials it holds, not by its intelligence or its prompt. Jerome Saltzer and Michael Schroeder stated the rule in 1975:
"Every program and every user of the system should operate using the least set of privileges necessary to complete the job."
They wrote it decades before prompt injection existed, and it's still the strongest defense against it.
Problem: An email triage agent holds an organization-wide admin token, granted "for flexibility." A crafted inbound email injects an instruction to forward CRM exports to an external address. The model does what it was built to do: it reads a message, follows an instruction and uses a permission it holds. The failure is the credential, not the model.
Capable models make this worse. The smarter the agent, the more tempting the broad grant: "Give it the wide token so it can figure things out." Intelligence attracts authority, and authority multiplies the payoff of every successful injection. A prompt that says "please be careful" is a hint. The credential is the enforcement.
Solution: Scope each agent to its own task. On our migration platform, the scopes look like this:
| Agent | Can read | Can write | Never holds |
|---|---|---|---|
| Discovery and analysis agents | Only their own artifact type | Only the shared knowledge model | Repository, production or deployment credentials |
| Generation agent | Evidence for its task | Candidate code in a staging workspace | Rights to certify, promote or deploy |
| Reviewer council | Evidence | Review findings | Any access to code |
| Arbiter | Evidence and reviews | Approve or reject decisions | Any access to code |
| Human approval gate | The full evidence trail | Promotion approval | n/a |
The shared knowledge model is the only writable common state. Human approval sits before promotion as a security boundary, not a courtesy.
Now assume the generation agent is fully compromised tomorrow, through poisoned context, a hostile injection or a plain hallucination. The mechanism doesn't matter. Its worst possible act is putting bad code into a staging queue that certification will reject. No production notebook is overwritten, no branch is merged and no job is deployed.
Note: Least privilege has a real cost. Scoping each agent separately means more credentials, more setup and the occasional ticket asking why an agent can't do something. That friction is the insurance premium, paid in small amounts and often. The alternative is paid once, after an incident.
The card below sums up least privilege for AI agents: the law, the failure it predicts and how scoped credentials fix it.
In fig.: The principle of least privilege applied to AI agents. The credentials set the blast radius, so authority is scoped per task, writes go to staging only and sensitive actions wait for approval.
Kerckhoffs's Principle: assume the system prompt will leak
Any AI agent control that depends on a secret system prompt isn't a control. It's one extraction attack away from nothing. Auguste Kerckhoffs set out the rule in 1883, in La Cryptographie Militaire, writing about military ciphers. His second principle says a system:
"should not require secrecy, and it should not be a problem if it falls into enemy hands."
Cryptographers summarize it as a system that stays secure even if everything about it except the key is public. For agents, the translation is direct: prompts aren't keys.
Problem: A support agent's safety depends on a confidential prompt that forbids refunds above a threshold. A jailbreak exposes the prompt, and a follow-up injection overrides it. Every safeguard lived in the one artifact the attacker now controls. The design didn't fail at the injection. It failed months earlier, when someone treated a sentence as a security boundary.
Solution: Run the Kerckhoffs test. Imagine your entire agent repository becomes public tomorrow: every agent prompt, every reviewer prompt, the conversion rules, the scoring logic and the orchestration flow. Can an attacker now deploy to production, certify an output or promote anything?
We ran this test on our migration platform. The answer is no, because the prompts grant nothing:
| If this is published… | …the attacker learns | …but still can't |
|---|---|---|
| Generation agent prompt | How code is drafted | Write outside staging, because its credentials end there |
| Reviewer council prompts | How evidence is judged | Approve anything, because the prompts define reasoning and carry no authority |
| Certification rules | What passes review | Certify, because that needs an authorized approval workflow in infrastructure |
| Orchestration flow | The order of steps | Deploy, because deployment permissions live outside every model |
The only real secrets are credentials and keys, exactly as Kerckhoffs specified. Prompts define reasoning, and credentials define authority. Least privilege and Kerckhoffs are two halves of one design. The first limits what an agent may do. The second makes sure that limit doesn't depend on anyone keeping a paragraph secret.
Prompt secrecy is also misleading security. It feels like a safeguard and demos like one, so a team ticks the box and stops building real controls. A hidden instruction saying "never overwrite production" is a polite request to a language model. An access policy in infrastructure is enforced.
Note: Kerckhoffs never said secrecy is worthless. Keys and credentials must stay secret. You may also keep prompts private for business reasons, such as protecting competitive IP or making jailbreaks less convenient. Those are valid choices. Just don't let the word "confidential" carry any weight in your security architecture.
The card below shows Kerckhoffs's principle as a test for AI agent security: would the deployment still be safe if its full system prompt were published?
In fig.: Kerckhoffs's principle for AI agents. Assume the system prompt will leak, and enforce limits in infrastructure the model can't rewrite, such as tool authorization checks, payment caps, egress filtering and output scanning.
Chesterton's Fence: an agent may only delete what it can explain
Cleanup is the most dangerous job you can give an AI agent, because deleted things prove their importance only by their absence. Generated output can be reviewed before it matters. A deletion often shows its cost weeks later, at the worst possible moment.
G. K. Chesterton described the rule in The Thing (1929). A reformer finds a fence across a road and says, "I don't see the use of this; let us clear it away." The wiser reformer answers:
"If you don't see the use of it, I certainly won't let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it."
Agents work through context windows, which are keyholes. If an artifact's purpose isn't visible through the keyhole, it looks useless. That shows the agent's limited view, not that the artifact has no use. A cron job with zero references in the repository may be called externally by the month-end close. The agent can't see the month-end close. It can only see zero references.
Problem: A converter finds a sort step between two transforms. Nothing downstream in the graph seems to need the ordering, so the step is removed, and the pipeline gets faster. Three weeks later, customer reconciliation fails. An old batch system on the far side of the boundary expects ordered records. The sort wasn't an optimization. It was a contract that only history knew about.
This law covers the case the first two can't. Least privilege limits what an agent may touch. Kerckhoffs moves enforcement out of the prompt. Neither one catches an agent with legitimate authority that is simply wrong. The most dangerous deletion is the authorized one, because the credential was valid.
Solution: Our migration platform generates code last, not first. Before any conversion, the pipeline:
- Reconstructs the topology
- Extracts metadata contracts
- Reads runtime logs
- Builds lineage
- Assembles all of it into an evidence model
That evidence model is the fence investigation. Every unexplained transformation is treated as intentional until evidence proves otherwise. The reviewer council doesn't ask, "Can we convert this?" It asks, "Do we understand why this exists?" If the answer is no, the item goes to a person for review. It's never silently simplified away.
The operating rules that came out of this:
| Rule | What it prevents |
|---|---|
| The agent must state an artifact's purpose before any delete or simplification | Removing a fence because its purpose isn't visible |
| Unknown purpose goes to escalation, never to deletion | Guesswork turning into data loss |
| Provenance lookups are mandatory tool steps: who created it, what calls it and what last ran it | Decisions made with only the context window in view |
| Every removal stays reversible for a grace window | Permanent loss from one wrong call |
| A speed gain from removing something unexplained is logged as a warning | Celebrating the change that breaks reconciliation later |
Note: Chesterton's fence is a gate, not a wall. It doesn't say "never remove." It says "explain first, then remove freely." Many legacy fences really are useless, and an evidence model lets you remove them with confidence. A migration that keeps every quirk forever isn't modernization. It's embalming.
The card below applies Chesterton's fence to AI cleanup agents. Missing evidence of purpose is treated as a limit of the agent's view, not as proof that an artifact is useless.
In fig.: Chesterton's fence for AI agents. An agent must state an artifact's purpose before removing it. Unknown purpose goes to escalation, provenance lookups are mandatory and every removal stays reversible.
The Swiss Cheese Model: independent layers, not more layers
Every AI agent guardrail has holes, so security depends on stacking independent layers whose holes don't line up. Prompts can be injected, validators have blind spots, judge models make mistakes and people get tired.
James Reason, who studied why planes crash and reactors fail, set out the model in Human Error (1990). It later became known through the image of stacked slices of Swiss cheese. Each defense is real and useful, and each has holes. An accident happens when holes in several layers line up at the same moment. Major accidents are rarely one big failure. They're several ordinary weaknesses lining up on the same day.
Problem: A production agent whose only defense is one carefully engineered system prompt has a single slice. One successful injection is a complete breach, because nothing stands behind it.
Solution: On our migration platform, suppose the generation agent hallucinates a transformation. Before that code can reach production, five layers must all miss it:
| Layer | What it checks | Mechanism |
|---|---|---|
| 1. Metadata checks | Schema mismatches | Rules |
| 2. Mechanical validation suite | Output correctness | Test execution |
| 3. Trust score threshold | Confidence across the evidence | Scoring |
| 4. Reviewer council with an evidence advocate | Whether the change is justified | Model-based review |
| 5. Human certification gate | Final approval | Human judgment |
All five could fail on the same artifact, but the odds are against it, as long as the layers are independent.
The earlier laws are slices too:
- Least privilege credentials (law 1)
- Enforcement in infrastructure, not prompts (law 2)
- Purpose gates before any destruction (law 3)
- Input sanitization at the edges, and schema validation on every output
- An independent reviewer council feeding one arbiter
- Staged writes with rollback, and a human gate at the end
The key word is independent. Our first Laws of AI Agents post used Condorcet's Jury Theorem to show that correlated reviewers share errors. Five safeguards that fail the same way are really one slice. Independence has to be engineered, with different mechanisms, different failure modes and different owners.
The design rule follows from this: build each layer as if every other layer has already failed. That isn't pessimism. It's the specification.
There's also a habit almost nobody has. When layer four catches a bad output, most teams celebrate the catch. A mature team also logs it as a near miss and asks why layers one to three let it through. A hole that almost lined up is tomorrow's incident report, delivered early.
Note: Each layer adds latency and cost. The model doesn't ask for infinite layers. It asks for independent ones. Two independent layers beat five correlated ones, and most budgets allow three or four good slices. Spend on independence, not on count. The human slice has holes too, and approval fatigue is the biggest one.
The card below shows the Swiss cheese model as a defense-in-depth design for AI agents, where safety comes from several independent layers rather than one perfect guardrail.
In fig.: The Swiss cheese model for AI agent guardrails. Every safeguard has holes, so independent layers are stacked, each designed as if the others have already failed, and near misses are tracked.
Hyrum's Law: every consistent output becomes a promise
Anything an AI agent does consistently will be depended on by someone, whether or not you promised it. Hyrum Wright, a Google engineer, described the pattern, and his colleague Titus Winters named it in Software Engineering at Google:
"With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody."
Problem: Migration discovery keeps finding examples. A month-end job writes two words to its log: "Migration Complete." For years, an external scheduler has watched for exactly those words and used them to trigger a downstream workflow. No specification mentions it, and no diagram shows it. Remove the log line, and month-end stops.
In another case, a legacy job writes 00123 and the modern replacement writes 123. The values are numerically identical. But a downstream consumer expects fixed-width values, and production breaks.
What the documentation promised and what production depended on are often far apart:
| Documented | Actually depended on |
|---|---|
| Four named fields | Column order, decimal formatting, fixed-width padding, null representation |
| An output file | File names, arrival times, sort order |
| Error handling | Reject file naming, specific error codes, log messages |
Agents make this worse, because their output is full of incidental patterns: phrasing, section order, Markdown style, JSON field order, tone and length. Consumers, both people and programs, build on all of it. A harmless prompt change that reorders two sections can break three pipelines in other teams overnight, and no test ever covered the dependency.
The agent-specific version is the model upgrade. Swap the underlying model and many incidental patterns shift at once, so every unofficial dependency breaks on the same day. The team blames the new model's quality. The real failure is older. Consumers were parsing weather.
Solution: A contract's real job is to declare which outputs are promises and which are weather. Our first post covered Postel's Law: emit strictly against a contract. Hyrum's Law explains what happens to everything outside the contract. On our migration platform:
- Discovery collects every observable artifact, including scripts, schedules, file names and reject paths, not just code.
- Runtime logs reconstruct undocumented dependencies, like the two-word trigger above.
- Lineage hunts for hidden consumers, such as the forgotten report, the vendor feed and the regulatory extract.
- Validation checks ordering, counts, formats, reject handling and business totals, because those are observable too.
- On every proposed simplification, the evidence advocate asks how we know nobody depends on this.
Here's a test you can run this week: change nothing but your agent's phrasing and watch what breaks downstream. That difference is your accidental API surface, mapped for free.
Note: Hyrum's Law doesn't require freezing every output forever. That would be the embalming that Chesterton's fence warns against. It requires knowing. Keep the official promise list small and structured, version it like an API, and publish deprecation windows for everything else.
The card below explains Hyrum's Law for AI agent outputs, where phrasing, ordering and formatting become an accidental API that downstream systems depend on.
In fig.: Hyrum's Law for AI agents. Publish an explicit output contract, version prompts like APIs, run contract tests on every prompt or model change, and give consumers structured fields instead of prose.
Where to start
Start by running one test per law against the agents you already have in production. These laws aren't libraries to install. They're the reasons architectures hold or fail.
| Law | Test to run this week | Warning sign |
|---|---|---|
| Least privilege | List the worst thing each agent could do with the credentials it holds today | Any agent with an organization-wide or admin token |
| Kerckhoffs | Assume the full prompt set is public. What breaks first? | A safeguard that exists only as prompt text |
| Chesterton's fence | Find every agent that can delete or simplify. Does it have to state a purpose first? | Deletions with no escalation path |
| Swiss cheese | Count the layers between a bad output and production, and check that each uses a different mechanism | Several model-based checks and nothing else |
| Hyrum's Law | Change only your agent's phrasing in a test environment and watch downstream consumers | Consumers parsing free text with no versioned contract |
For the architectural laws that come before these, read Laws of AI Agents: Architecture from Postel's Law.
You don't secure an agent by making it smarter. You secure it by making its worst day boring.
Talk to Tarento's Generative & Agentic AI team about stress-testing your agent security
