An AI-Written Report Nearly Put U.S. Forces on a Chinese Ship. Brussels Backs an AI Slowdown That Mistral Rejects. AI Borrowers Now Compete With Governments for Capital. (FREE VERSION)

GTR Terminal: Subscribers get full access to our live dashboard tracking the signals shaping the future. Your access link and password are one click away.

CNN reports that a chatbot invented nuclear cargo on a Chinese vessel this spring and that armed U.S. personnel were preparing to board before the error was caught. A Pentagon inquiry into a February strike on an Iranian school points to the same habit of trusting software output without checking it.

BRIEFING: A CNN report on September 18 said an intelligence report this spring stated that a Chinese ship in the Middle East was carrying components of a nuclear weapons program. The military planned an interception, armed personnel were preparing to board, and aircraft were already in the air before the raid was called off. Shortly before the operation, officials looked into how the report had been produced. An analyst at a special operations command had asked an AI chatbot about the ship's cargo list. The tool combined public information with classified intercepted communications, known as signals intelligence, and reached a false conclusion about the cargo. The analyst then used AI again to format the finding as a standard intelligence report – the kind commanders are trained to trust. CNN's sources said such fabrications have not been isolated across the intelligence agencies, and there is no real guidance on how a human check is supposed to prevent civilian deaths or attacks on friendly forces as AI moves into targeting. In this case, the operation was halted by a person who checked the source, which is the system working, albeit late.

The same day, Bloomberg reported that Pentagon investigators discovered that flawed intelligence, outdated imagery and an overreliance on AI contributed to the February 28 missile strike on an elementary school in Minab, Iran, which killed 123 children. The administration had ordered an opening assault that hit more than 1,000 targets in 24 hours, which compressed the time available to confirm each one. The site had once been part of a military compound but had been a school for years, and the targeting database had not been updated. Target review ran through the Maven Smart System, software made by Palantir that merges many intelligence feeds to propose and rank targets. Palantir told Bloomberg it is not responsible for the underlying data or for identifying gaps in intelligence, and that there is no evidence its software was at fault. Officials pointed to cuts of about 90 percent in the Pentagon teams that work to limit civilian harm. 

In both cases, people treated a confident machine summary as verified, in one case because of time pressure and in the other because the format looked official.

SO WHAT

For Executives:   The lesson transfers directly to any company that lets staff use AI to draft analysis that others act on. The ship analyst's error was ordinary. The damage came from a lack of appropriate human fact checking and the packaging of the output as a finished report, which lost any sign of how it was made, evoking unwarranted trust. Require that every AI-assisted analysis in your company carries a visible label and a source list that a second person can check in minutes. Decide now which decisions in your business need a named human to confirm the underlying source before action, and write that into procedure this quarter. Shipping, energy, and insurance firms operating near U.S. forces in the Gulf should add one more item – a contact route to the relevant naval command, because a false report about your vessel is now a plausible event. Hedge: There is no hedge. Human review and trust assurance must be essential elements of AI use, whatever the finished inquiries say about the details of these two cases.

For Policy Makers:  The U.S. and Chinese presidents meet at the White House on Thursday, September 24, and their governments now have a concrete case for the incident channel that has been discussed since the summer (Issue 36, September 10). In the case of the Chinese ship, neither side had a way to ask the other a question in time. Examine and safeguard the ratio of human oversight capacity to AI-generation volume and speed. Propose a military-to-military hotline protocol that covers AI-assisted intelligence specifically, and propose it at the UN Security Council meeting on Wednesday so it has multilateral standing. Defense ministries that are buying decision-support software, including NATO members and Gulf states, should write into contracts who is responsible for flagging stale data or flawed processes, since Palantir's statement makes clear that the vendor does not accept that duty by default. Restore civilian-harm review teams before expanding AI in targeting. Hedge: A reporting channel can be used for propaganda as easily as for safety. Limit it to factual queries with a fixed response time, and test it in an exercise before relying on it.

For Investors:  Defense software valuations assume that speed is the product, but these two incidents show the liability of that perspective. Expect procurement rules that require audit trails, provenance labels, and independent monitoring and testing, both in the U.S. and among allies, and favor vendors that can already show how an output was produced and veracity assured. Such vendors should have formal mechanisms for human governance of AI inputs and outputs, which should include safeguards against human attention spans being stretched by agent chains and orchestras that are too long or complex. Palantir's exposure is reputational more than contractual since government buyers carry responsibility for data quality, but a congressional inquiry is likely and would weigh on the sector. Firms that sell verification, data lineage, and model evaluation to defense customers gain from the same events. Hedge: Wartime demand for faster targeting has so far outweighed every caution. If the Iran war continues, budgets will keep favoring speed, and the audit requirement might arrive as guidance with no money attached.

For Service Providers:  Three pieces of work follow. For corporate clients, an AI-assisted analysis policy with labeling, source lists, and a second-reader rule, deliverable in weeks. For defense suppliers, a contract review that assigns responsibility for data freshness and error flagging and fixing between vendor and customer. For governments, the design of an incident-hotline protocol, including what gets reported, to whom, and how fast. Law firms should prepare clients across industries for the question of who is liable when a state acts on a false machine-generated report about their asset. Hedge: Clients will ask for a tool that detects hallucinations. None is reliable enough to promise. Sell process and accountability, and be plain about what software cannot yet do.