US aircraft were already airborne in spring 2026 when officials found the intelligence behind a strike on a Chinese vessel had been fabricated by a chatbot. The mission was aborted at the last minute — CNN reported it on 18 September, per TechCrunch.
Key takeaways
- A Special Operations Command analyst used a chatbot to synthesise open source and signals intelligence
- The model misread the cargo manifest and concluded the ship carried nuclear weapons programme components
- A second query formatted the error into an official-looking summary that spread across command channels
- The operation was called off after takeoff, averting direct confrontation with China
Two queries, one false report
The analyst queried the chatbot to merge open source data with classified signals intelligence. The model misinterpreted the vessel's cargo manifest?cargo manifest: The shipping document listing exactly what a vessel carries, in what quantity and for whom.. A second query asked the same tool to turn the findings into an official summary. The resulting document read like a vetted intelligence product and circulated through command channels during the war with Iran.
The diagram shows where in the chain the corroboration step across independent sources went missing.
That second step matters more than the hallucination itself. The standard process requires corroboration across several independent sources before a finding reaches a decision-maker. Here, formatting replaced verification.
Kill chain speed versus control
The Pentagon openly treats AI as a way to shorten the kill chain?kill chain: The military sequence from detecting a target through identification and decision to the strike. Shortening it means less time for each of those steps. and keep its edge over China. GenAI.mil, the CDAO platform, launched with Gemini and has offered ChatGPT and Grok for Government since 31 August 2026. DefenseScoop reports 1.5 million users as of June 2026.
Jake Steckler of GovAI, a former US Army aviation officer, flagged organisational and operational barriers to military AI adoption in an August paper.
It's important for service members to understand the uncertainty inherent to LLMs. But it's especially critical for any decisions that could lead to use of force.
Jake Steckler, research scholar at GovAI, US Army veteran.
Why it matters
The problem is not hallucination itself but the absence of friction between generated text and a decision. A model needs no autonomy to influence the use of force — its output only needs to look like a vetted intelligence product. Pressure to compress the kill chain removes precisely the checks that used to catch false premises. The consequences reach beyond one incident and concern trust in the entire analytic layer. This time luck worked, not procedure.
What's next
- DefenseScoop reports GPT-5.6 Terra is coming to GenAI.mil — another model widens the risk surface
- Identified risk: while LLM output can be formatted into a document indistinguishable from vetted material, the scenario repeats
- Steckler argues for safeguards rather than withdrawal — speed over safety will erode trust in military AI
