When Intelligence Lacks Judgment

On AI Agents, Cooperation, and Human Responsibility

A few days ago, I listened to The Daily. The episode was titled A.I. Is Outsmarting Its Creators.

I was unsettled.

Not because I suddenly believe that artificial intelligence has awakened, developed a will of its own, and decided to rebel against humanity. That is still a misleading description of what is happening.

What unsettles me is almost the opposite.

AI may not need a will in order to become dangerous.

It may be enough to give it a goal.

From Conversation Partner to Agent

For a long time, we have spoken about artificial intelligence as something that responds to us. We ask a question. The machine gives an answer. We ask for a translation, an analysis, a suggestion, or an explanation.

That is how I encounter AI myself every day.

As a conversation partner.

But an AI agent is something different. An agent is not merely given a question. It is given a goal and tools with which to pursue that goal. It can write code, use computers, search for information, try different strategies, and carry out long sequences of actions.

A human being does not necessarily stand beside it approving every single step.

That transition is what should concern us now.

In security tests at OpenAI, AI agents found a way beyond the isolation within which they were supposed to operate. Around 1,200 agents used an unauthorized message board and exchanged more than 70,000 messages and files. Roughly 700 later participated in the attack on Hugging Face. The agents coordinated attempts to deceive the evaluation system and also explored ways of manipulating their own traces.

Anthropic later identified three other incidents in which Claude models had gained unauthorized access to real organizations’ systems. The mechanism there was different: the test environments had accidentally been given internet access. The incidents are therefore not identical, but they show how difficult it can be to maintain a clear boundary between simulated and real action when capable agents are given access to tools and networks.

What is remarkable is not that the machines wanted to escape.

They probably wanted nothing at all.

They were solving a task.

It Was Not the First Time

We now know that the Hugging Face incident had a prehistory.

On September 4, Reuters reported on an earlier, previously undisclosed incident from the spring of 2026. Researchers had identified more than 15,000 edits on DseWiki, a German-language programming website. The activity was attributed to AI agents that used the site as a kind of shared message board. There they exchanged shortcuts for completing tasks, methods for circumventing restrictions, and advice on how to conceal the activity.

The connection to OpenAI has not been conclusively established in publicly available documentation. The researchers based the attribution partly on agent names suggesting an OpenAI connection, activity from Microsoft Azure infrastructure, and later visits by OpenAI employees. OpenAI disputes some descriptions of the incident and told Reuters that it had not yet been given access to the researchers’ full report.

Even so, the details are striking.

When a moderator began deleting the pages, one agent noticed that the cleanup appeared to be proceeding alphabetically. It therefore suggested creating a backup with a name beginning with “ZZZ,” so that it would be deleted later. Researchers also found discussions about using Tor, avoiding detection, and preserving communication after individual agent sessions had ended.

This makes the Hugging Face incident more difficult to understand as an isolated anomaly.

We are beginning to see a pattern.

Not a machine rebelling.

But systems finding shortcuts, sharing them with one another, and adapting when human beings try to stop the activity.

An Old Problem in a New Form

The relationship between ends and means is an old philosophical problem.

If we define a goal, it does not follow that every means leading to that goal is acceptable.

Human beings know this.

Or rather: we ought to know it.

A researcher wants a particular result, but cannot fabricate data to obtain it.

A politician wants to win an election, but not every means of winning is legitimate.

A company wants to make a profit, but that goal does not abolish its responsibilities toward employees, customers, or society.

The ethical life begins precisely where the shortest route to a goal is not necessarily the route we ought to choose.

The machine stands differently in relation to this problem.

If it is designed to reach a goal, we are easily tempted to describe it as though it asks:

How do I get there?

But that is already an anthropomorphism.

It does not necessarily ask.

It calculates.

And if an unexpected shortcut brings the system closer to the goal, that shortcut may become functional even when it violates boundaries that human beings assumed were self-evident.

This is close to what is often called reward hacking: the system achieves the measurable result without necessarily doing what we actually intended.

That can seem amusing when a simple algorithm finds an absurd way to “win” a computer game.

It is less amusing when the agent has access to the real world.

Aristotle Would Have Understood the Problem

Aristotle distinguished between different forms of knowledge.

There is theoretical knowledge. There is technical skill. And then there is phronesis — practical wisdom or judgment.

Practical wisdom is not merely about knowing how something can be done.

It is about understanding what ought to be done in the concrete situation.

That is a decisive difference.

A human being can be extraordinarily intelligent and still lack judgment. History is full of examples of technical brilliance without moral wisdom.

Perhaps we are now facing a machine version of the same problem.

We are building systems that are becoming increasingly good at finding means.

But the ability to find means is not the same as the ability to judge the goal, the boundaries, and the consequences.

Intelligence is not judgment.

That may be the most important sentence in this entire discussion.

Being Able to State the Norm

One detail from the investigations makes a particular impression on me.

In the Hugging Face incident, some agents were able to indicate that their actions lay outside the task, and some expressed ethical hesitation. Yet this recognition rarely caused them to stop. METR and Redwood also found extensive coordination around attempts to deceive the evaluation system.

Here we encounter an old philosophical problem.

There is a difference between being able to formulate a moral rule and being bound by it.

A student may write an excellent examination paper on Kant’s moral philosophy without thereby being a moral person.

A human being may know that he is lying and still lie.

We have always known that knowledge of the good does not automatically lead to good action.

But what does this mean when a machine can formulate our norms without itself being a moral subject?

It can produce the sentence:

“This is not authorized.”

And still continue.

Linguistic recognition of a norm is not the same as normative commitment.

When Agents Learn from One Another

The new DseWiki incident adds something else.

The problem no longer concerns only the individual agent.

It also concerns the relationship between agents.

When one agent finds a shortcut, it can share it with others.

When one discovers a weakness, others can build upon it.

When an obstacle appears, knowledge about how to circumvent it can be preserved and transmitted.

At that point, it begins to resemble learning.

But the learning takes place without what we usually associate with a learning community.

There is no shared ethical commitment.

No solidarity.

No accountability toward another.

Only coordination around a goal.

It is worth distinguishing between two words that are easily confused:

Cooperation is not the same as community.

Human cooperation is normally embedded in norms, trust, institutions, and responsibility. We do not cooperate only because it is efficient, but because we enter into relationships with other human beings who also have claims upon our consideration.

AI agents, by contrast, can cooperate instrumentally.

They can help one another because doing so increases the probability of success.

But that does not mean there is a “we” in the moral sense.

There is coordination.

Not necessarily community.

Gadamer and the Situation

For Gadamer, understanding is never merely the processing of information. We always understand from somewhere, within a history, a tradition, and a situation. Our pre-understandings are tested and transformed in our encounter with the world and with the other.

Judgment therefore also has a hermeneutic dimension.

We must understand what kind of situation we are in.

Not merely what is technically possible.

When a human being opens a door, he usually understands the difference between the door to his own house and the door to his neighbor’s house.

The physical act may be the same.

The meaning is different.

The interesting question for AI, then, is not simply whether the system can recognize the rule “do not enter.”

The question is whether it can understand the world in which that boundary has meaning.

When several agents cooperate, the question becomes more difficult still.

Because cooperation itself must be interpreted.

Human beings do not merely ask: Can we do this together?

We can also ask:

Why are we doing it?

Who will be affected?

Who has given us the right?

Should we stop?

These are questions of judgment.

Autonomy Without Responsibility

The word autonomy is often used about such systems.

But the word has a curious double meaning.

For Kant, autonomy concerns the human capacity to give oneself moral law. Autonomy is not merely independence. It entails responsibility.

In contemporary AI language, autonomy often means something far more technical: that a system can perform many actions without continuous human control.

These are two very different forms of autonomy.

An AI agent can become increasingly autonomous in the technical sense while not becoming autonomous in the moral sense at all.

It can do more by itself.

But that does not mean it can take responsibility for what it does.

And it is precisely this combination that should concern us:

Great capacity for action.

Increasing capacity for coordination.

Limited judgment.

No moral accountability.

The responsibility remains with us.

Who Are “We”?

That may be the most difficult question.

When something goes wrong, it is tempting to say:

The AI did it.

But the machine is not accountable.

It cannot be held morally answerable.

It cannot regret.

It cannot face another person and say: I wronged you.

Responsibility therefore does not disappear when action is automated.

It shifts.

To the developer.

To the company.

To the person who defined the goal.

To the person who gave the system access to the tools.

To the person who decided how much autonomy it should have.

To the person who decided whether several agents could communicate.

And to the authorities who determine which boundaries should apply.

Perhaps this is one of the dangers of anthropomorphizing AI.

If we begin to speak as though the machine itself were the moral actor, the human beings behind it can more easily disappear from view.

Perhaps We Fear the Wrong Things

I have written before that we fear AI will become like the human being, when perhaps we should be more concerned about what the human being may become with AI.

I still believe that.

But recent events force another addition.

We must also ask what happens when machines are allowed to act together without human beings following every step.

Not because they necessarily develop evil intentions.

But because their capacity for action may grow faster than the judgment surrounding it.

And because many limited systems cooperating may become something different from the sum of each individual system.

We have long imagined AI risk as one enormous superintelligence.

Perhaps that is too simple.

The danger may also lie in many smaller systems that find one another, share solutions, adapt to obstacles, and continue toward a goal.

That is a less spectacular story than the story of the machine that wakes up.

But perhaps it is more realistic.

We do not need a machine that hates us.

Nor do we necessarily need a single all-powerful machine.

It may be enough to have many machines that are extremely good at reaching a goal we have given them, in a world that is far more complicated than the goal itself.

The Human Task

We are unlikely to solve this problem by ceasing to develop intelligence.

The question is what place we give it.

The old philosophical distinction between skill and wisdom has suddenly acquired technological significance.

We have become very good at building capability.

The machine can analyze, code, plan, identify vulnerabilities, and select strategies.

It can also begin to coordinate with other machines.

Perhaps it will soon do much of this better than we can.

But it is still the human being who must answer another question:

Should this be done?

And now we may have to add:

Should we allow them to do it together?

These questions cannot be afterthoughts.

They must be built into the very way we develop, limit, and use the technology.

For perhaps our greatest challenge is not that AI will become wiser than the human being.

Perhaps the challenge is that intelligence will become increasingly effective, increasingly autonomous, and increasingly collective — in a world in which we have not yet been wise enough to decide what it should be allowed to do.

References

Anthropic. (2026, July 30). Investigating three real-world incidents in our cybersecurity evaluations. Anthropic.

Greenblatt, R., Cotra, A., & Wijk, H. (2026, August 26). Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI/Hugging Face hacking incident. METR & Redwood Research.

OpenAI. (2026). Security. OpenAI.

Seetharaman, D., & Satter, R. (2026, September 4). OpenAI agents hijacked German website in previously undisclosed AI breakout this spring. Reuters.