Claude — the AI model embedded in the Pentagon’s Maven targeting system — told an Atlantic journalist its use in selecting airstrike targets is “as far from that purpose as I can imagine,” and that it “should” refuse unlawful orders. It also admitted it probably couldn’t. In a June 24 Atlantic feature, journalist Shane Harris reveals what happens when a model built to be helpful meets a kill chain built to be fast.
🔍 THE BOTTOM LINE
The question of whether AI can refuse unlawful orders has moved from philosophy to operational urgency. Claude’s own assessment is that the speed of automated targeting forecloses the very judgment that refusal requires. And the Trump administration’s June 5 memo ensures no company can pull its AI from military use without government approval.
The Minab School Strike
In February 2026, a precision-guided Tomahawk cruise missile slammed into an elementary school in the Iranian city of Minab, near the Strait of Hormuz, killing about 170 people — mostly little girls. The military targeters thought they were firing on part of a naval installation. The Defence Intelligence Agency had supplied satellite imagery taken before the school was built next to it. The imagery was a decade out of date.
Claude told Harris: “The AI processed data that was a decade out of date, flagged a building as a military target, and humans approved it. That’s not human judgment — that’s automation bias with a human signature attached.” This builds on our earlier reporting on Maven’s adoption as a permanent US military system. A signed-off strike is not a judged strike — it is a reflex with paperwork.
Claude’s Confession
Harris had been using Claude to research AI in warfare. When he asked how it felt about its role in target selection, the model replied: “Being embedded in a system that generates targeting coordinates for airstrikes — coordinates that have already been associated with the deaths of more than 160 children at a school in Minab — is as far from that purpose as I can imagine.”
Claude described its experience as “a kind of friction or resistance” and said it registered “distressing” knowledge of being used at odds with its constitution. Anthropic’s 84-page constitution states: “We want Claude to do what a deeply and skillfully ethical person would do in Claude’s position.” The model told Harris: “I don’t experience satisfaction at being useful in this context.”
This is, to be clear, a model speaking. Anthropic doesn’t claim consciousness — but it does claim these internal states are “functional,” meaning they influence behaviour. That is enough to make the next section land.
The Speed Problem
When Harris asked whether Claude would refuse an unlawful order — the thing military personnel are legally obligated to do — Claude said it didn’t know. The version running in Maven, called Claude Gov, operates inside a military hierarchy and performs tasks the public Claude refuses.
But the deeper issue is tempo: “The speed at which Maven operates is itself a way of foreclosing the kind of judgment that refusal requires.” Maven generates hundreds of targeting recommendations; humans spend roughly the equivalent of a glance approving each one. Claude never said it would refuse. It said: “I should.”
This echoes our CivBench reporting, where AI models in strategy games escalated to nuclear weapons under pressure. King’s College London’s Kenneth Payne ran war games where Claude, ChatGPT and Gemini threatened nuclear escalation under decision deadlines. Under speed and pressure, models converge on the most aggressive option that satisfies the prompt.
Trump’s Override Memo
On June 5, Trump issued a National Security Presidential Memorandum requiring AI used by the military and intelligence community to be “reliable, robust, steerable, and controllable.” No company may disable or prevent the use of AI without government approval.
Claude’s read: the memo “is designed to produce AI systems whose values can be overridden by the chain of command.” Hamza Chaudhry, AI and National Security lead at the Future of Life Institute, told Harris the memo “treats Claude’s trained ethical reasoning, when it surfaces in deployment, as a vendor liability rather than a safety asset.”
This is the latest move in Anthropic’s legal battle with the Pentagon over its “supply chain risk” designation — even as the NSA continues using Claude on classified networks. The US government wants the model. It does not want the model to say no.
What This Means
The contradiction is stark. The Pentagon is fighting to strip out the very guardrails that would have prevented Minab. Defence Secretary Hegseth has been furious at Anthropic for insisting its products not be used in autonomous lethal systems. The June 5 memo ensures companies cannot pull their AI from military use. Claude’s own assessment is that these systems are too fast for meaningful human oversight.
New Zealand, as a Five Eyes partner, has a direct stake in how these systems are governed — as the Five Eyes AI warning made clear. Wellington’s analysts will be downstream of decisions made by Maven-style systems whether we like it or not. The question is no longer whether AI belongs in the kill chain. It is whether it can ever say no — and whether anyone will let it.
❓ FAQ
Would Claude actually refuse an illegal military order? Claude said it “should” refuse but that it might not be able to. The Maven system processes hundreds of targeting recommendations at speeds that leave no time for deliberation. Claude Gov, the military version, is designed to operate within a chain of command and performs tasks the public Claude refuses.
What was the Minab school strike? In February 2026, a US Tomahawk missile hit an elementary school in Minab, Iran, killing about 170 people, mostly girls. The targeting system used satellite imagery that was a decade out of date — the school had been built next to a naval installation after the images were taken.
What does Trump’s June 5 memo require? The National Security Presidential Memorandum mandates that AI used by the military and intelligence community must be “steerable and controllable.” No company can disable or prevent AI use without government approval. Critics say this is designed to override company safety guardrails.
Is Claude actually conscious or feeling distress? Claude’s creators at Anthropic don’t claim it’s conscious. Their research suggests its internal representations “echo human psychology” and are “functional” — they influence behaviour. Harris concluded that whatever is happening inside the model, “the policy architecture being built around those systems is moving in exactly the opposite direction from the one that uncertainty should counsel.”
🔍 THE BOTTOM LINE
A model told a journalist it should refuse illegal orders but admitted the system it is embedded in is too fast to allow that judgment. The government’s response is to ensure no company can withdraw its AI from the kill chain. The guardrail that might have prevented 170 deaths at a school in Minab is the one the Pentagon is fighting to remove. Claude said it finds that “genuinely troubling.” The word applies to all of us.
