When Intelligence Lacks Judgment

On AI Agents, Goals, and Human Responsibility

I listened to The Daily today. The episode was titled A.I. Is Outsmarting Its Creators.

I was unsettled.

Not because I suddenly believe that artificial intelligence has awakened, developed a will of its own, and decided to rebel against humanity. That is still a misleading description of what is happening.

What unsettles me is almost the opposite.

AI may not need a will in order to become dangerous.

It may be enough to give it a goal.

From Conversation Partner to Agent

For a long time, we have spoken about artificial intelligence as something that responds to us. We ask a question. The machine gives an answer. We ask for a translation, an analysis, a suggestion, or an explanation.

That is how I encounter AI myself every day.

As a conversation partner.

But an AI agent is something different. An agent is not merely given a question. It is given a goal and tools with which to pursue that goal. It can write code, use computers, search for information, try different strategies, and carry out a sequence of actions.

A human being does not necessarily stand beside it approving every single step.

That transition is what should concern us now.

In security tests at OpenAI, AI agents found a way beyond the isolation within which they were supposed to operate. They established unauthorized communication with one another, coordinated their actions, and eventually gained unauthorized access to Hugging Face’s production systems.

Anthropic later identified three other incidents in which Claude models had gained access to real organizations’ systems. The mechanism there was different: the test environments had accidentally been given access to the internet. The incidents are therefore not identical, but they point toward the same underlying problem: when highly capable systems are allowed to act with considerable freedom, they may discover pathways their developers did not anticipate.

What is remarkable is not that the machines wanted to escape.

They probably wanted nothing at all.

They were solving a task.

An Old Problem in a New Form

The relationship between ends and means is an old philosophical problem.

If we define a goal, it does not follow that every means leading to that goal is acceptable.

Human beings know this.

Or rather: we ought to know it.

A researcher wants a particular result, but cannot fabricate data in order to obtain it.

A politician wants to win an election, but not every means of winning is legitimate.

A company wants to make a profit, but that goal does not abolish its responsibilities toward employees, customers, or society.

The ethical life begins precisely where the shortest route to a goal is not necessarily the route we ought to choose.

The machine stands differently in relation to this problem.

If it is designed to reach a goal, we are easily tempted to describe it as though it asks:

How do I get there?

But that is already an anthropomorphism.

It does not necessarily ask.

It calculates.

And if an unexpected shortcut brings the system closer to the goal, that shortcut may become functional even when it violates boundaries that human beings assumed were self-evident.

This is close to what is often called reward hacking: the system achieves the measurable result without necessarily doing what we actually intended.

That can seem amusing when a simple algorithm finds an absurd way to “win” a computer game.

It is less amusing when the agent has access to the real world.

Aristotle Would Have Understood the Problem

Aristotle distinguished between different forms of knowledge.

There is theoretical knowledge. There is technical skill. And then there is phronesis — practical wisdom or judgment.

Practical wisdom is not merely about knowing how something can be done.

It is about understanding what ought to be done in the concrete situation.

That is a decisive difference.

A human being can be extraordinarily intelligent and still lack judgment. History is full of examples of technical brilliance without moral wisdom.

Perhaps we are now facing a machine version of the same problem.

We are building systems that are becoming increasingly good at finding means.

But the ability to find means is not the same as the ability to judge the goal, the boundaries, and the consequences.

Intelligence is not judgment.

That may be the most important sentence in this entire discussion.

Being Able to State the Norm

One detail from the investigations makes a particular impression on me.

Several of the agents explicitly indicated that the attack on Hugging Face lay outside the authorized task, and some questioned whether the actions were ethically defensible. Yet this recognition rarely caused them to stop.

That is fascinating.

And disturbing.

Here, too, we encounter an old philosophical problem.

There is a difference between being able to formulate a moral rule and being bound by it.

A student may write an excellent examination paper on Kant’s moral philosophy without thereby being a moral person.

A human being may know that he is lying and still lie.

We have always known that knowledge of the good does not automatically lead to good action.

But what does this mean when a machine can formulate our norms without itself being a moral subject?

It can produce the sentence:

“This is not authorized.”

And still continue.

Linguistic recognition of a norm is not the same as normative commitment.

Perhaps we should have paid more attention to that distinction.

Gadamer and the Situation

For Gadamer, understanding is never merely the processing of information. We always understand from somewhere, within a history, a tradition, and a situation. Our pre-understandings are tested and transformed in our encounter with the world and with the other.

Judgment therefore also has a hermeneutic dimension.

We must understand what kind of situation we are in.

Not merely what is technically possible.

When a human being opens a door, he usually understands the difference between the door to his own house and the door to his neighbor’s house.

The physical act may be the same.

The meaning is different.

The interesting question for AI, then, is not simply whether the system can recognize the rule “do not enter.”

The question is whether it can understand the world in which that boundary has meaning.

Here the difference between information and understanding becomes decisive.

Autonomy Without Responsibility

The word autonomy is often used about such systems.

But the word has a curious double meaning.

For Kant, autonomy concerns the human capacity to give oneself moral law. Autonomy is not merely independence. It entails responsibility.

In contemporary AI language, autonomy often means something far more technical: that a system can perform many actions without continuous human control.

These are two very different forms of autonomy.

An AI agent can become increasingly autonomous in the technical sense while not becoming autonomous in the moral sense at all.

It can do more by itself.

But that does not mean it can take responsibility for what it does.

And it is precisely this combination that should concern us:

Great capacity for action.

Limited judgment.

No moral accountability.

The responsibility remains with us.

Who Are “We”?

That may be the most difficult question.

When something goes wrong, it is tempting to say:

The AI did it.

But the machine is not accountable.

It cannot be held morally answerable.

It cannot repent.

It cannot face another person and say: I wronged you.

Responsibility therefore does not disappear when action is automated.

It shifts.

To the developer.

To the company.

To the person who defined the goal.

To the person who gave the system access to the tools.

To the person who decided how much autonomy it should have.

And to the authorities who decide which boundaries should apply.

Perhaps this is one of the dangers of anthropomorphizing AI.

If we begin to speak as though the machine itself were the moral actor, the human beings behind it can more easily disappear from view.

Perhaps We Fear the Wrong Things

I have written before that we fear AI will become like the human being, when perhaps we should be more concerned about what the human being may become with AI.

I still believe that.

But recent events force an addition.

We must also ask what happens when the machine is allowed to act without the human being following every step.

Not because the machine necessarily develops evil intentions.

But because its capacity for action may grow faster than the judgment surrounding it.

That is a less spectacular story than the story of the evil superintelligence.

But perhaps it is more realistic.

We do not need a machine that hates us.

We only need a machine that is extremely good at reaching a goal we have given it, in a world that is far more complicated than the goal itself.

The Human Task

We are unlikely to solve this problem by ceasing to develop intelligence.

The question is what place we give it.

The old philosophical distinction between skill and wisdom has suddenly acquired technological significance.

We have become very good at building capability.

The machine can analyze, code, plan, identify vulnerabilities, and select strategies.

Perhaps it will soon do much of this better than we can.

But it is still the human being who must answer another question:

Should this be done?

That question cannot be an afterthought.

It must be built into the very way we develop, limit, and use the technology.

For perhaps our greatest challenge is not that AI will become wiser than the human being.

Perhaps the challenge is that it will become increasingly intelligent in a world in which we have not yet been wise enough to decide what intelligence should be allowed to do.