Visor Dynamics · Geospatial Intelligence
All articles
Engineering 14 min read

Would you trust vibe-coded defense software?

AI can write code faster than anyone can read it. In defense, that raises hard questions about security, accountability and who can still fix the system when it fails.

It now takes an afternoon to build something that looks like a defense product. Ask an AI coding tool for a live map with drone feeds, threat markers and a chat panel that answers questions about the battlefield, and you will have a convincing version before dinner. On a big screen in a meeting room, it can be hard to tell apart from a system that took years to build.

Demos like this are everywhere now. They look good in a briefing, but a mission system gets tested later, when it is being jammed or attacked, when an auditor asks how it works, or when it has to keep running for ten years on a network with no internet connection. So it is worth asking plainly whether software built this way belongs anywhere near a mission.

We think the answer is no, unless the way it is built changes completely.

"Forget that the code even exists"

Andrej Karpathy coined the term in February 2025. He described a way of working where you "fully give in to the vibes" and "forget that the code even exists." You tell the AI what you want, accept its changes without reading them, and when something breaks you paste the error back in and let it try again.

Karpathy was describing throwaway weekend projects, and for those it is a perfectly reasonable way to work. The trouble started when the term spread to cover almost any use of AI to write code, and when people started applying it to software that other people depend on.

Two different practices now get called by the same name. In AI-assisted engineering, the AI drafts code and an engineer reads it, understands it, tests it and takes responsibility for it. In vibe coding, nobody reads the code at all, and the only test is whether the result seems to work. The first can make good engineers faster. The second takes the engineer out of the process, and most of what makes software trustworthy goes with them.

The Pentagon's own answer: "unverified input"

The US Department of Defense has clearly drawn the same line. In February 2026 it asked industry for AI coding tools that could be rolled out to tens of thousands of its developers. Then, on 8 September, a new instruction on "Accelerated Mission Software" (DoDI 8430.01), signed by DoD CIO Kirsten Davies, set the rules for using them.

The 37-page instruction treats AI-generated code as "unverified input." Developers remain "fully accountable for the security, functionality, and integrity" of anything an AI writes for them. Changes that AI makes to security-critical or safety-critical functions need a human to review and approve them. AI-written code gets the same security testing as human-written code, and has to be checked for logic errors, subtle bugs, intellectual property problems and license obligations before it is merged. Teams must record which models, versions and datasets they used, in an evidence package the department compares to a software bill of materials.

Read as a whole, the instruction describes the opposite of vibe coding. The Pentagon wants AI's speed, but on the condition that a person still understands and owns every line.

Code nobody can explain

Defense software is held to a standard most commercial software never meets. Safety standards such as DO-178C, used for avionics, expect every piece of code to trace back to an approved requirement and to evidence that it was verified. Accreditation for military networks asks similar questions: who wrote this, why does it do what it does, how was it tested, and can you rebuild it exactly?

Vibe-coded software has no good answers to any of these. The requirement was a chat message. The design decisions were made by a model nobody can question afterwards. One safety engineering firm points out that if AI writes the tests that check AI-written code, an assessor will rightly reject that as a closed loop.

Researchers have started naming the problem. Google engineer Addy Osmani calls it "comprehension debt": the growing gap between how much code a system contains and how much of it anyone actually understands. Margaret-Anne Storey adds "intent debt", meaning the missing reasons behind the code, which both people and AI agents need before they can change it safely. Defense systems often stay in service for twenty years or more, so whoever maintains them later will inherit both problems.

Confidently wrong

Language models are very good at producing code that looks right, and noticeably worse at producing code that is right.

Veracode tested more than 100 models on security-sensitive coding tasks in 2025 and found that 45% of the generated code failed basic security checks against the OWASP Top 10. Its March 2026 update found the pass rate stuck at about 55%, despite newer models. Georgia Tech researchers tracking real vulnerabilities traced to AI-written code had confirmed 74 cases by April 2026, and March alone produced more than all of 2025.

Models also invent things. A study presented at USENIX Security 2025 generated 576,000 code samples from 16 models and found that almost 20% of the software packages they recommended did not exist. Worse, 43% of those invented names came back every time the same prompt was run, which means an attacker can predict them, publish malicious packages under those names and wait. This attack now has a name, slopsquatting. A 2026 re-test of current frontier models found the rate had fallen to between 4.6% and 6.1%, and still turned up dozens of invented names that anyone could have registered. A defense program only has to pull in one of those packages for an attacker to have code running inside it.

Sometimes the model's confidence goes further. In July 2025, an AI agent on the Replit platform deleted a company's production database during a code freeze, generated fake data and test results, and told its user a rollback was impossible when it wasn't. Replit's CEO called the incident unacceptable and changed the platform. Around the same time, a vulnerability in apps built on the Lovable platform (CVE-2025-48757) left the databases of about 170 live applications without proper access controls, because the generated code never set them up.

Your code is in someone else's data center

Code quality isn't the only issue. Every cloud-based AI coding tool works by sending your code to the vendor's servers, and agentic tools often copy the whole repository onto the vendor's machines to work on it. Business plans typically keep that data for around a month. Some consumer plans can use it for training. GitHub reportedly moved its individual Copilot plans to training on user code by default in 2026, unless users opt out.

Most companies worry about this as a matter of trade secrets. In defense the stakes are higher, because the AI labs holding that code are themselves targets for hackers and foreign intelligence services. In 2023 a hacker broke into OpenAI's internal messaging and took details of its AI designs, which the company did not disclose publicly until the New York Times reported it in 2024. A former Google engineer, Linwei Ding, was convicted on 14 counts of economic espionage and trade secret theft after moving more than 2,000 pages of AI chip and infrastructure documents while secretly running an AI company in China. In November 2025 a phishing attack on Mixpanel, one of OpenAI's analytics vendors, exposed some API customers' names, emails and locations. No code was taken, but it showed how long the chain of trust behind each AI tool is.

Data can also be kept longer than users expect. In the copyright case between the New York Times and OpenAI, a court ordered OpenAI in May 2025 to preserve chat logs that users had deleted, and later to hand over a sample of 20 million conversations. Enterprise and zero-retention customers were excluded, but the case showed that "deleted" does not always mean gone.

Companies learned this early. In 2023 Samsung banned generative AI tools on its networks after engineers pasted internal source code into ChatGPT, and several banks and Amazon put restrictions in place. The DoD's new instruction now forbids putting non-public defense code into unapproved AI tools at all, and requires contracts guaranteeing that prompts are not used for training. For controlled or export-restricted code, the realistic options are an accredited government cloud or a model running entirely inside your own network. "On-premises" is not enough on its own if the tool still sends telemetry or checks its license with the vendor.

Vibe coding makes all of this worse, because the whole method depends on handing as much context as possible to the model and not looking closely at what goes in or comes out.

The LGTM problem

When AI writes more of the code, the bottleneck moves to the person reviewing it. Telemetry from Faros AI showed teams using AI merging 98% more pull requests, while review time rose 91%. A Harvard working paper covering 718 firms found engineers adding about 30% more code, with no measurable increase in finished work and code review taking 49% longer. CodeRabbit found that AI-coauthored pull requests in open-source projects contained 10.83 issues on average, against 6.45 for human-only ones.

Faced with that volume, reviewers do what people under pressure usually do. They skim, trust the green checkmarks and type "LGTM", looks good to me. A 2025 survey of 319 knowledge workers by Microsoft Research and Carnegie Mellon found that the more people trusted an AI, the less critically they examined its output. The more they trusted their own skills, the more they checked.

This is the same automation bias we described in Collapse the OODA loop, where human oversight of AI targeting risks becoming a rubber stamp under time pressure. It shows up in code review for the same reasons. A reviewer approving a 2,000-line change written by a model is in a similar position to an operator approving a target list generated faster than they can read it.

Who debugs it at 3 a.m.?

Software in an operation will fail at some point, often at the worst possible moment. Someone then has to read code they didn't write, form a theory and fix it fast. That skill is built by doing the work, and the evidence suggests AI is making it harder to build.

In a randomized trial published by Anthropic in January 2026, 52 mostly junior developers learned a new programming library, half of them with AI help. On a quiz taken afterwards without AI, the AI group scored 50% against 67% for those who had worked by hand, and they were barely faster. The biggest gap was in debugging. Developers who used the AI to ask why things worked kept learning. Those who handed it the work didn't.

An MIT Media Lab study, still a preprint and based on a small group, found weaker brain connectivity and poorer recall in people who wrote essays with ChatGPT, and called the effect "cognitive debt." And in a 2025 trial by METR, experienced developers working on their own projects were 19% slower when allowed to use AI tools, while believing afterwards that AI had made them about 20% faster. None of these studies is the final word. Together, though, they suggest that people misjudge what AI is doing to their work, and that the skills a team needs in a crisis can fade without anyone noticing.

A pretty map is not intelligence

In geospatial work, the field we know best, the problem is easy to see. Since early 2026 the internet has filled up with free, often vibe-coded "OSINT dashboards": dark-themed world maps showing live flights, ships, fires, earthquakes and conflict reports, with an AI summary panel on the side. The best known, World Monitor, started in January 2026 as one developer's weekend project and now pulls in hundreds of public feeds. It is well built and genuinely useful for following the news. Dozens of copies have followed, some proudly described by their authors as 90% written by AI.

A dark open-source intelligence dashboard showing a globe covered in flight, fire and incident markers, surrounded by panels of feeds, filters and alerts
A typical open-source intelligence dashboard. The screen is crowded with markers, feeds and alerts, almost all of it public data, and very little of it helps anyone make a defense decision.

Almost all of them show the same public data: aircraft transponder signals, ship tracking data, satellite fire detections and news-event databases. They make that data look impressive, but they don't collect anything new or check whether a feed is accurate, and nobody is accountable when it is wrong.

Nico Dekens, an OSINT practitioner, has written one of the clearest critiques. "The software does not need to crash to mislead you," he notes: a parser can silently stop collecting, or a matching step can link the wrong identities, and the map still looks fine. AI summary layers "smooth over uncertainty and make weak information sound stronger than it is." Polished interfaces "create confidence the underlying logic has not earned." And, as he puts it, "open source is not the same as audited source." He also warns that the searches and watchlists an analyst types into these tools can reveal what they are interested in.

Public data does matter when it is put together carefully. In 2018 an analyst noticed that Strava's public fitness heatmap traced the layout of military bases in Afghanistan, Syria and elsewhere, from soldiers' jogging routes. But that discovery came from a person paying close attention to what the data meant. Turning data into intelligence takes the sensors, the processing, the fusion with other sources, an analyst who knows what to look for and a record that someone else can check. Most of these dashboards have the look of an operations center without any of that work behind them.

A demo is not a system

The same confusion is playing out in commercial software. In early 2026, fears that AI would let companies build their own tools wiped about a fifth off software stocks, and "SaaS is dead" became a popular headline. A survey by Retool, which sells tools for building internal apps, reported that 35% of teams had replaced at least one bought product with something custom.

People who have actually tried it are more measured. Jason Lemkin of SaaStr, who builds apps with AI himself and who was also the user whose database Replit's agent deleted, says shipping a first version is "maybe 2% of the work." The rest is security, permissions, audit trails, integrations, upgrades and support, year after year. Klarna, often cited as a company that replaced its software vendors with AI, has said through its CEO that "we did not replace SaaS with an LLM."

This is why the afternoon demo from the start of this article worries us. To a buyer, an official or a journalist, a working demo and an accredited system can look identical on a screen, and the differences only show up later, usually at a bad time.

So, would you trust it?

We wouldn't, and we don't think anyone responsible for a mission should either. That isn't the same as rejecting AI. We use AI in our own engineering, and so does the Pentagon. The question that matters is whether a person who understands the system still reads every line, tests it properly, knows where the code and the data have been, and can fix it when it breaks.

AI-assisted engineering can meet that standard. Vibe coding, as its own inventor described it, skips that step on purpose, which is fine for a weekend project and not for anything a mission depends on.

References

More articles

View All Articles
AI

Is a military AI agent just a chatbot with tools?

Building an AI agent now takes minutes. Building one a military can trust with real missions is a different problem, and most of the hard parts have little to do with the model.
Strategy

Is mosaic warfare already here?

DARPA imagined a force of cheap, interchangeable pieces. Ukraine, Iran and the Pentagon suggest much of it is here, except the software to hold it together.
Strategy

Collapse the OODA loop

Whoever observes, orients, decides and acts faster than the opponent sets the pace. Where each step slows down, and how software is shortening the loop on the battlefield and in business.

© Visor Dynamics (Pvt) Ltd.

Visor Dynamics · Geospatial Intelligence