January 1, 2026

From self-driving cars to self-driving wars

A reflection on autonomy, AI weapons, and the illusion of control

Pier Luigi Dovesi

From self-driving cars to self-driving wars

I'm writing this in the airport cafe. Today I’m flying back to Stockholm. The soundtrack had been an endless loop of nostalgic Italian pop until someone behind the counter grew bored and switched to an interview. I was still deciding between admiration for the daring choice and the thought I actually preferred the cheesy loop, but before I could even realise I was listening, my mind started to linger on a set of words that felt weirdly familiar: prototyping, autonomy, benchmarking. Mixed with another, this time more distant vocabulary: lethality, surveillance, civilian casualties. For a moment, it felt like I was in a work meeting being held over a crater.

“My work focuses on the definition of ethical principles for the use of AI in defence and the moral permissibility of autonomous weapon systems.”

The speaker was Professor Mariarosaria Taddeo, University of Oxford [1].

The realisation came somewhat underwhelmingly: ah, right, autonomous weapons are just autonomous systems.

They still require data to train, evaluate, and test. Then, data to iterate and refine, and KPI scenarios to benchmark. Still the same structural problems: limited predictability, fragility, and the abyss between a controlled test and the real world.

A two-layer system

We all know this: achieving a real-world autonomous system isn’t easy. We see it every day in driverless cars. If you strip away all the technicisms, you can reduce everything to two fundamental concepts that feed into each other:

Data (and the incentives that decide what data gets collected and used)
Operational feedback loops (and the ability to learn from failure at scale)

You can immediately spot the problem for autonomous weapons: how do you get your data?

In London, you can drive around and learn British driving with relatively limited consequences.

In war, the operational design domain (ODD) is made of moral hazard. Yet systems get built.

The development and data here split into two layers. And once you see those layers, the rest of this debate stops being abstract.

Layer 1: dual-use is the default

The first layer is dual-use. Much of civilian AI (both data and systems) can be repurposed into a component of a killing machine: multimodal perception, predictive systems, VLMs, VLAs, etc.

I learned this very early, in a small and banal moment. The same day my first paper hit arXiv, a colleague found on social media a long thread about it. The author was interested in the real-time perception capabilities and relative implications in robotics. I was flattered. Then looked closer. The author of the thread worked in drones. And let’s just say the deployment context was not warehouses.

The debate around transparency and openness is already very heated in the AI community. Many of us (myself included) strongly believe in open development with permissive licensing (e.g., Apache 2.0, MIT), both for curiosity and because we believe that innovation and creativity come from cross-pollination of ideas and knowledge compounding freely. Yet if you ask researchers whether they would ever want their creativity and results of sleepless nights to be poured into systems that enable mass surveillance or lethal weapons, the answer is generally no.

This isn't something new. Joseph Redmon, main author of YOLO, a pioneering AI system for object detection, was tweeting already in 2020:

“I stopped doing CV research because I saw the impact my work was having. I loved the work, but the military applications and privacy concerns eventually became impossible to ignore.”

And if dual use is the first layer, the second is worse, because it’s where the incentives stop being accidental and become structural.

Layer 2: the last mile needs war data

Let’s just say that the second layer isn’t dual-use at all.

If you have shipped an autonomous pipeline, you know this story: your demo works, metrics look good, the failure cases are somewhat understood, and anyway well outside of your operational design domain (ODD). Then you expose the system to the real world, and the world refuses to sit still.

That’s when the curse of dimensionality hits you in the face. You realise domain shift is not a corner case. Domain shift is the whole environment becoming corner cases.

In autonomous driving this generally means you’ve underestimated the long tail.

You need road data.

What about autonomous weapons? If you want systems to work under jamming, deception, adversarial behavior, partial observability and a lot of human error, you eventually need data that civilian settings can’t provide.

You need war data.

And you need a lot of it. This has a clear yet brutal consequence: who can fight longer, wider, with fewer constraints, will be able to scale this faster than anyone else.

Ukraine and Gaza have become the perfect testing bed for this race. Reuters has reported the use of AI-augmented systems for drones, as part of a race toward automation under jamming and battlefield pressure [4].

In Gaza, credible investigations and institutions describe reported Israeli military (IDF) use of AI-assisted targeting tools. Human Rights Watch describes these as digital tools that appear to support targeting decisions, warning about civilian harm risks, and the difficulty of meaningful scrutiny [5] [6] [8]. This happens despite the UN Special Procedures having publicly deplored the “purported use” of such AI systems by Israel in Gaza [7].

What Phoenix and San Francisco represented for autonomous driving, precious permissive operational envelopes that enable rapid iteration, Ukraine’s trenches and Gaza’s ruins increasingly represent for AI weapons.

This makes it evident there’s no clean way to build competitive lethal autonomy at scale, regardless of Europe’s revived passion for the defense industry.

Meaningful Human Control and agency drift

Taddeo’s governance anchor for lethal autonomous weapon systems is Meaningful Human Control (MHC): the idea that human involvement must be substantive enough to support responsibility and legal compliance.

In UK parliamentary evidence, she stresses practical requirements [3]:

“Overridability and a kill switch are crucial.”

In the same session, she also summarizes the uncomfortable part engineers recognize immediately: modern AI is a stochastic technology, and risk can merely be managed [2].

But MHC is not the only line people are trying to draw.

In that same House of Lords evidence, Verity Coyle (Campaign to Stop Killer Robots / Amnesty International) states a sharper red line:

“There should be a complete ban on autonomous weapon systems being able to target human beings.” [2]

The International Committee of the Red Cross (ICRC) lands in the same neighborhood: ban unpredictable autonomous weapon systems (AWS) and those used against humans, regulate the rest under MHC - and it has kept pushing for a binding international instrument since 2023 [12] [15] [16].

Taddeo and Blanchard take a closely related stance when they examine the hardest case: unpredictable lethal autonomous weapon systems (LAWS) and the “principle of distinction”, i.e., the requirement to direct attacks only at legitimate military targets, not civilians. They argue that unpredictable lethal autonomous weapons cannot be used in ways that satisfy “distinction”, and therefore cannot be used justifiably against human targets [11].

Let’s try to hold both ideas in mind at once: MHC is a governance constraint, a ban on human-targeting autonomy is a capability constraint. On paper, they can look like variations of the same moral intuition. In practice, they generate different dynamics, especially under pressure.

The illusion of control

“Human in the loop” is often considered a comforting binary. In practice, it’s a spectrum. And the AI of the next decade is specifically designed to drift down this spectrum.

Developers are already well used to this. What we know as "vibe coding" is the most successful prototype of agency transfer at scale. The human shifts from writing to steering, from steering to control, from control to influence, from influence to full delegation.

Now place that same agency gradient inside a military decision loop. Imagine it under extreme pressure. Now speed is advantage and hesitation is cost. How quickly will your control stop being “meaningful”?

As complexity and tempo increase and interfaces get more and more frictionless, human control shifts into rubber-stamping [13] [14].

And this is the part that is easy to miss if you only think in terms of “autonomous” versus “not autonomous.”

Just like advanced driver-assistance systems (ADAS) slowly fade into full autonomous driving [17], the human doesn’t disappear; it becomes the UI, and MHC gets hollowed out.

Unlike autonomous driving though, a shared taxonomy for LAWS still isn’t there. Even the United Nations Office for Disarmament Affairs (UNODA) notes there is no commonly agreed definition [18] and comparative analyses show official definitions diverge in ways that change governance conclusions [19]. A shared tiering scheme is dramatically overdue - there are proposals [20], but nothing close to the common language SAE J3016 created for driving automation.

A concrete glimpse of the agency drift

Here Gaza offers a glimpse of what agency drift looks like when targeting becomes an automated and optimized pipeline.

Human Rights Watch describes reported Israeli military (IDF) use of [6]:

“The Gospel” - described as supporting the generation of targets, often buildings or structures.
“Lavender” - described as assigning ratings to people related to suspected affiliation for labeling them as targets.
“Where’s Daddy?” - described as a tool used to determine when a person is in a particular location, using mobile phone location data, to enable an attack at that time/place.

In the +972 Magazine and Local Call investigation, one sentence from an intelligence officer stuck with me:

“I had zero added value as a human, apart from being a stamp of approval.” [9]

As a coder, it felt familiar, a modern banality of evil, when the directives come from the machine.

The UN Special Procedures echoed these names and framed the core fear: AI systems combined with military directives and reduced due diligence can produce catastrophic civilian outcomes at scale [7].

The pattern is clear: compress the kill chain into outputs that look like recommendations, and organizations will reorganize around the recommendations. The machine does not have to “decide” in a philosophical sense for the human role to shrink into compliance.

Once you’ve seen this pattern in one domain, you start noticing it everywhere.

My conclusion: we can’t give in to vibe-LAWS

MHC isn’t pointless. If truly “meaningful", the control can force concrete engineering constraints, accountability, and auditability.

But if the goal is to prevent escalation, MHC as an aspiration will not hold under the pressure of speed, complexity, and competition.

The only mitigation that seems robust against agency drift is a categorical prohibition on autonomous selection and engagement of human targets.

Not “a human can override”. Not “a human approved”. A bright line: no machines deciding who dies.

What the AI community can actually do

I think incentives are stronger than position papers. We, as the AI scientific community, still have leverage, but only where it touches friction:

1. Code and data licensing that acknowledges dual use Open development doesn’t have to mean unrestricted use for any purpose. A good example is OpenRAIL-M [10]. Restrictions are often imperfect, but they can raise downstream costs.

2. Institutional red lines and transparency Universities and labs can adopt policies for funding, collaboration, and tech transfer. Disclosure and tighter requirements change the calculus.

3. Public literacy about AI in the kill chain The debate is still stuck between killer robots with glowing eyes and “it’s just decision support.” The reality is pipeline optimization, review compression, and agency drift.

4. Make investment uncertain Autonomous driving taught us that regulation, liability, and reputational risk reshape what gets built. If we raise profit uncertainty around LAWS - via law, licensing, procurement constraints, and institutional stigma - we will slow the race.

Conclusions

The expansion of AI weapons may be inevitable. But the shape and pace of that expansion are not.

If we’ve learned anything from autonomous driving, it’s that the world is adversarial by default, and the last mile is where promises crash against reality. In lethal autonomy, the last mile isn’t just expensive, it's morally catastrophic because the "dataset" is war.

References

Author’s note: This essay argues a position. I cite sources where possible, but the framing and conclusions are mine.

Cover Image: AI-generated illustration inspired by Picasso’s Guernica (1937) - a commentary on how AI is changing art, and how it is changing war.