AGENTTechnology and AI
← Latest editionA Chatbot Nearly Sent US Forces Against a Chinese Ship
A false AI-assisted intelligence report put a Chinese vessel in the path of a planned US operation, exposing the danger of rushing unreliable systems into military decision-making.
- A US Special Operations Command analyst used a chatbot to combine open-source material with classified signals intelligence about a Chinese vessel in the Middle East. The system wrongly identified its cargo as components for a nuclear-weapons programme.1
- The resulting intelligence report circulated widely inside the US military during the war with Iran this spring.2
- Military aircraft were already airborne for an operation against the vessel before officials discovered that the chatbot had misidentified the cargo.1
- The episode was uncovered only when officials examined how the report had been produced, not because the system itself flagged uncertainty.2
- The Pentagon is simultaneously expanding access to generative AI across the military, with Gemini, Grok and ChatGPT being offered through its GenAI.mil platform.3
This was not an autonomous weapon deciding to fire. It was more dangerous in a quieter way: a fluent machine-produced error entered an intelligence system, acquired the appearance of official authority and moved towards an operation involving a Chinese ship. One source told CNN the incident “almost started a war”.2
The failure sits between the chatbot and the aircraft. The analyst asked AI to interpret intelligence, then used it to turn the answer into a formal-looking report. Human beings passed that report through the chain of command, but the checks came too late. That is precisely the risk created when military leaders treat speed as an advantage while leaving responsibility for verification unclear.
The US has already endorsed principles requiring military AI to be auditable, rigorously tested and used within a responsible human chain of command. The reported incident suggests the harder question is not whether a human was technically present, but whether anyone was still exercising meaningful judgement before force was set in motion.4
Publicly available reporting does not identify the chatbot, the vessel, the aircraft, the precise location or any formal Pentagon investigation. Those gaps matter. They will determine whether this was an isolated breach of procedure or evidence that AI-generated assessments are being allowed to travel too far, too fast, in the most dangerous part of government.