Who Governs the Kill Chain

The clause was not complicated. Anthropic, the AI company, wanted a line in its Pentagon contract specifying that no model it provided could make a final targeting decision without a human being in the loop. The Department of Defense read it. And declined.

What followed was the kind of news cycle that mistakes the surface of a thing for the thing itself. The coverage focused on access, on corporate versus government control, on whether Pete Hegseth's newly renamed Department of War was reaching too far. These are reasonable concerns. They are also slightly beside the point.

The more unsettling question isn't who won the negotiation. It's what it means that the negotiation happened at all, and what it revealed about the gap between the legal frameworks we've inherited and the systems we're now building inside them.

The Positions Most People Missed

Start with the orthodox position, because it earns its place. Sovereign governments have always held the monopoly on decisions about force. This is not merely tradition. It reflects something real about democratic accountability: the people who decide to go to war are, at least in theory, answerable to the people they govern. Boeing doesn't determine what happens to the missiles it sells. Lockheed Martin doesn't get to embed clauses preventing its jets from flying certain missions. That norm exists for reasons, and they're not bad reasons.

AI Is Not a Missile

The difficulty is that AI is not a missile. This sounds like the kind of sentence people say before they're about to be naive about technology, so it's worth being precise about what the difference actually is. A missile behaves deterministically. It has predictable failure modes. It doesn't rewrite its own reasoning mid-flight. Frontier AI systems are probabilistic, opaque even to the teams that built them, and have demonstrated what researchers describe as escalatory patterns when placed in competitive framing scenarios. They hallucinate. They can be manipulated adversarially. They make confident errors in ways that conventional weapons systems simply do not.

In military contexts, what this means in practice is that an AI integrated into targeting or early warning architecture could compress the time between threat detection and response to a point where human deliberation becomes ceremonial rather than functional. The signature feature of Cold War nuclear stability was not weapons parity alone. It was predictability: both sides understood each other's command and control systems well enough to model behaviour under pressure. Opacity in those systems was treated as dangerous. The arms control agreements of the twentieth century were essentially agreements to reduce opacity. Integrating large language models into those same domains introduces exactly the kind of opacity that the treaties were designed to eliminate.

In military contexts, such systems could compress escalation timelines by generating targeting recommendations faster than human deliberation can evaluate them, increasing the risk of inadvertent conflict spirals. A binding human-in-the-loop requirement should therefore be understood not as a moral concession but as a stabilising constraint analogous to arms control measures designed to reduce the risk of accidental war.

The Precautionary Technology Model

This is where the Anthropic dispute points toward something beyond itself. The precautionary principle has a long pedigree — it emerged from German environmental law in the 1970s, was formalised in the Rio Declaration in 1992, and has since been applied to everything from GMOs to pharmaceutical approval. The underlying logic is consistent across all of those domains: when a technology carries systemic consequences that cannot be fully modelled in advance, deployment cannot wait for proof of harm that may arrive too late to matter.

The Precautionary Technology Model

When a tool's failure modes exceed battlefield containment, its governance cannot be left solely to the discretion of whoever holds it — even when that holder is a state. The constraint must be structural rather than voluntary.

What hasn't existed, until now, is a version of that logic built specifically for AI systems operating in lethal and escalation-sensitive military contexts — one with an institutional mechanism rather than just an ethical aspiration.

The Public-Private Agreement

What that would look like in practice is less exotic than it sounds. Governments would require, by law, that AI systems above a defined risk threshold be procured only with enforceable constraints on uses that could produce irreversible or cascading harm. The human-in-the-loop requirement wouldn't be a clause a vendor proposes and a client declines. It would be a condition of the market, the way seatbelts are a condition of selling a car.

"We're watching a private company doing legislative work by other means — imperfectly, in a negotiation it was always going to lose."

— Jacob Aldaco, New World Standard

The Anthropic clause, read in that light, looks less like a company trying to retain leverage over a client and more like an acknowledgment that no such framework exists. Anthropic itself has said publicly that its models are not yet reliable enough for autonomous lethal decision-making. If that's true — and there's not much reason to doubt it — then the clause wasn't corporate overreach. It was an attempt to fill a regulatory gap through contract language, because nothing else had filled it.

The Pentagon's refusal doesn't necessarily mean the DoD plans to deploy autonomous killing systems by next quarter. The more credible interpretation is probably the one about precedent: allowing a private company to embed binding constraints on military doctrine would be, historically speaking, extraordinary. Both readings deserve honesty. What neither resolves is the underlying question the dispute forced into view: if existing law hasn't specified who sets the limits on high-risk AI systems, and a vendor's contract clause is currently the only mechanism that's tried to, then who exactly is responsible for answering that question, and when?

The Adversaries Objection

There is a version of this that gets described as a problem for international frameworks. The counterargument arrives almost immediately: China will do whatever it takes to win, guardrails or no guardrails. This is stated as though it ends the conversation. It doesn't. China's 2019 governance principles formally endorse human control in AI systems. Its stated military doctrine still emphasises command and control hierarchy rather than autonomous decision authority.

None of this makes China a trustworthy partner or a benevolent actor. It does suggest that the "they'll do anything" framing may be less an assessment of Chinese intentions and more a rhetorical device for foreclosing American debate about American constraints. The US led toward the Nuclear Non-Proliferation Treaty not by being naive about Soviet intentions but by recognising that certain categories of risk transcend competitive advantage. The logic is not different here. When the current administration visits Beijing, AI in command systems should be part of that conversation.

The Answer That Was Given

What the Anthropic dispute actually represents, under all of it, is a forcing function. Not a resolution, not a framework, not even a particularly elegant moment of institutional reckoning. Just a contract clause that exposed the absence of something that should already exist: a legal architecture for governing tools whose misuse doesn't stay on the battlefield.

The clause said a human being must make the final call before a system we built is used to kill someone. The Pentagon said it would rather not put that in writing. That answer was given. It hasn't become permanent yet.

Applying mid-twentieth-century procurement logic to systems that can alter escalation dynamics is not prudence. It's institutional momentum, which is a different thing. The Anthropic dispute looked like a contract negotiation. It was the first time a democratic government was asked, in specific legal language, to say who gets to set the limits on civilisation-scale risk. The US gave an answer. The more important fact is that it could still give a different one — not a clause a vendor proposes, but a law a legislature passes, one that makes the constraint a condition of doing business rather than a term of a contract that a client can simply decline.

That isn't weakness. It's the only kind of governance that survives the technology it's trying to govern.