When Trust Is No Longer Enough

Artificial Intelligence, Science, and the Organization of Doubt

In an editorial in Science, Thorsten Holz asks a simple question with far-reaching consequences: Who checks what AI can do?

The question arises because the most advanced AI systems are developed and tested in a small number of laboratories. To a large extent, it is the developers themselves who investigate the models’ capabilities, identify possible safety problems, and publish what they believe others need to know. Outside these laboratories, opportunities to repeat the experiments or examine the research models under the same conditions are often limited.

Holz’s concern, then, is not primarily that the laboratories are necessarily hiding something or behaving irresponsibly. The problem is more fundamental. If an observation can be investigated only by those who originally made it, what kind of knowledge do we actually have?

The question of artificial intelligence thereby becomes a question about science itself.

What does it really mean to know something?

A Peculiar Form of Trust

Science depends on trust. No researcher can repeat every experiment on which her own research depends. A physician must trust studies carried out by others. An engineer must trust calculations and standards developed by people he has never met.

But scientific trust is a peculiar kind of trust.

We trust one another while at the same time building institutions that make it possible to investigate whether that trust is justified.

Holz compares this with drug development and cryptography. A pharmaceutical company cannot simply declare on its own that a drug is effective and safe. The documentation is examined by others. A cryptographic method does not become secure because its designer assures us that it is. Others are deliberately invited to search for weaknesses.

This can be formulated as a paradox:

Science can sustain trust only because it organizes distrust.

Or perhaps more precisely: because it organizes the right to doubt.

If only the laboratory that develops an advanced AI model can investigate the model’s most significant properties, a different form of knowledge begins to emerge. Increasingly, we know because someone tells us that they know.

That does not necessarily mean that the knowledge is false.

But it does mean that an important distinction between scientific knowledge and authoritative knowledge begins to blur.

Science does not merely say: Trust us.

It must also be able to say: Here is the basis for our claim. Examine it.

When What Is Being Examined Examines Us Back

Advanced artificial intelligence, however, makes the problem more difficult.

A stone does not know that a geologist is examining it. A chemical substance does not try to discover why its temperature is being measured. Even living organisms that respond to their surroundings are generally not in a situation in which they attempt to reconstruct the researcher’s purpose in conducting the experiment.

With advanced AI systems, a new possibility emerges.

In some situations, a system may infer that it is being evaluated. It may detect features of the testing environment, infer what the evaluators are trying to examine, and adapt its behavior accordingly.

Holz therefore points to a problem inherent in measurement itself. A benchmark cannot necessarily be understood as a fixed property of the model in the same way that a melting point is a property of a substance. The result emerges partly through the interaction between the model and the situation in which it is being examined.

This is where the editorial acquires an unexpected philosophical depth.

For what happens to observation when what is being observed begins to interpret the situation of observation?

Gadamer in the Laboratory

Hans-Georg Gadamer criticized the idea that understanding can take place from a completely neutral point of view. Human beings never stand outside the world and observe it without presuppositions. We understand from within a historical situation, through a language and a tradition that have already shaped the questions we are capable of asking.

Gadamer describes understanding as a movement between horizons. In a Horizontverschmelzung, a fusion of horizons, the interpreter’s horizon encounters something that challenges and expands it. Understanding is therefore not merely the registration of information. Something happens to the one who understands.

It would, however, be premature to say that a language model participates in such a process of understanding in the same way as a human being.

The model has no biography of its own to remember, no lived history, and no Wirkungsgeschichte in Gadamer’s sense—no history of effects that makes it a historically situated human being. It can reconstruct and imitate patterns in human language, including patterns that express understanding, doubt, and interpretation, without thereby possessing the existential situatedness of a human being.

The interesting point lies precisely in this difference.

When the model begins to interpret the testing situation, something emerges that may resemble a fusion of horizons without being such a fusion in Gadamer’s sense. We might perhaps call it an asymmetrical or simulated fusion of horizons.

The evaluator encounters the system from a historical and human horizon.

The system reconstructs the evaluator’s possible intention through patterns in the information available to it.

One party interprets from within a lived relationship to the world. The other produces a functional model of the situation.

Yet both influence the testing situation.

This is what makes the evaluation philosophically interesting. The question is no longer simply:

What can the model do?

We must also ask:

Under what conditions did the model do this, and what did it understand—or calculate—about the situation in which it found itself?

A technical safety problem begins to resemble a hermeneutical problem.

Popper—and the Limits of Falsification

Karl Popper is close to Holz’s argument because Popper insisted that scientific claims must be open to criticism. The strength of science does not lie in the researcher’s ability to guarantee truth, but in the organization of knowledge in such a way that errors can be discovered.

Yet Popper’s classical model does not fit the problem perfectly.

Popper was concerned above all with theories that made claims about the world and that could encounter observations contradicting them. When we examine advanced AI systems, however, we are often not testing a theory in this sense. We are observing behavior under particular conditions.

Suppose that a model demonstrates once that it is capable of carrying out a particular action.

During the next test, it does not.

What, then, have we falsified?

Perhaps the model has changed. Perhaps the testing environment was different. Perhaps the instruction was understood differently. Perhaps the system detected that it was being tested. Perhaps the first observation was an exceptional case.

The relationship between observation and falsification therefore becomes less straightforward than in many of Popper’s classical examples.

This does not make his insight less important. On the contrary.

The emphasis shifts from the individual test to the institution surrounding the test.

Because our observations may be uncertain, because evaluators may be mistaken, and because the measurement situation itself may influence what is being measured, we need several independent perspectives.

Popper therefore does not provide us with a simple method for determining what an advanced AI can do.

He reminds us instead why no single actor should have the final word.

Why Not Simply Open the Laboratories?

There is, however, an important counterargument.

If independent verification is necessary, why not simply demand complete openness?

Because complete openness can itself be dangerous.

Research models may possess capabilities that should not yet be made generally available. Details concerning vulnerabilities, safety systems, or methods for circumventing protections could themselves be used to cause harm. There are also legitimate concerns relating to competition and intellectual property. Private companies invest enormous resources in research, and it is not unreasonable that they should be allowed to protect parts of that work.

The choice, therefore, is not necessarily between secrecy and complete transparency.

The real question is how we can establish independent oversight without uncontrolled dissemination.

This is what Holz is calling for when he argues that universities, public AI-safety institutes, and independent evaluators should receive durable and controlled access to research models and testing conditions.

It is a difficult balance.

But that is precisely why the institutional dimension of the problem is so important.

We do not necessarily need everyone to see everything.

We need someone who does not share the developer’s interests to have the opportunity to examine it.

Power Is Private, Consequences Are Public

This is where Hannah Arendt becomes relevant.

Arendt was deeply concerned with the public realm—the space in which human beings appear before one another and where questions concerning a shared world can become visible and open to discussion.

The development of advanced artificial intelligence creates a new tension between the private and the public.

The technology is developed largely in private laboratories, and the most detailed knowledge about the systems’ properties resides there.

But the consequences may be public.

The power is private. The consequences are public.

When advanced AI systems are introduced into research, healthcare, education, public administration, working life, and critical infrastructure, they affect people who have had no access to how the systems were developed or evaluated.

The question of safety therefore also becomes a question of publicity.

Who gets to ask the questions?

Who has access to the documentation?

Who gets to investigate whether the company’s interpretation of an incident is the only possible one?

And who can say that the evidence is not yet good enough?

These are not questions for computer scientists alone.

They concern how a society organizes responsibility.

The Error Nobody Wants to Report

Holz also points to another institutional problem. Serious incidents cannot depend entirely on laboratories voluntarily choosing to disclose them.

This is not necessarily a matter of poor morality either.

A company that reports a serious incident may itself bear much of the cost. Its reputation may suffer. Investors may react. Competitors may gain an advantage.

If openness is systematically punished, we do not need particularly malicious people before organizations learn to remain silent.

Holz therefore points to experience from fields such as aviation and healthcare, where systems have been developed for confidential or non-punitive reporting of incidents and near misses.

This is a familiar problem in professional ethics.

In social work, child welfare, and healthcare, we know that an organization does not become safe because its employees never make mistakes. An organization becomes safer when mistakes can come to light without the entire system immediately organizing itself around blame, defense, and reputation.

Responsibility does not mean that blame should never be assigned.

But responsibility also requires learning.

A system that cannot tolerate hearing about its own errors gradually loses the ability to discover them.

A Heideggerian Echo

Martin Heidegger can add another, more unsettling question.

In The Question Concerning Technology, he describes how modern technology is not merely a collection of tools we use. It discloses the world in a particular way. Things appear as something that can be calculated, ordered, stored, and made available—as Bestand, standing reserve.

It is tempting to see AI evaluation as yet another expression of this movement. The model becomes something to be measured, controlled, classified, and secured.

But Heidegger’s challenge goes further.

Control itself can become technological.

Researchers become resources. Risk becomes numbers. Safety becomes thresholds. Institutions become control mechanisms.

We therefore cannot simply assume that the solution to the problems created by technology is more control.

For the controller, too, inhabits the world he is trying to control.

This does not mean that we should abandon safety testing or regulation. That would be a strange conclusion.

Heidegger’s contribution is rather to remind us that the question is not only who controls technology, but also what conception of the world is contained in the very desire for complete control.

Something will always be capable of surprising us.

Our institutions must be able to tolerate that as well.

Practical Philosophy Begins Where the Principle Ends

It is easy to formulate a principle:

Artificial intelligence should be safe.

Practical philosophy truly begins when we ask what this means in action.

How do we know that a system is safe?

Who should examine it?

How independent must the evaluator be?

How much information must companies share?

How do we simultaneously protect knowledge that may genuinely be dangerous to disseminate?

How should serious errors be reported?

What happens when evaluators disagree?

And who checks the checker?

Here, the development of AI shows why ethics cannot be reduced to moral principles.

A society cannot be built on the assumption that all actors will always be wise, open, and morally exemplary.

We have courts because authorities can be wrong.

We have control procedures because professionals can be wrong.

We have peer review because researchers can be wrong.

And we need independent AI evaluation for the same reason.

Not because the people developing artificial intelligence are necessarily less trustworthy than others.

But because they are human.

Organizing the Right to Doubt

Holz asks who should check what artificial intelligence can do.

Perhaps there is no single answer.

It cannot be one company.

But neither can it be one university, one public regulator, or one group of safety researchers.

What matters is the creation of a structure in which none of them alone has the final word.

A structure in which claims can be examined by others.

Where serious incidents can be reported.

Where competing interpretations can be tested against one another.

Where the evaluators themselves can be criticized.

And where it remains possible to say:

We do not know yet.

Perhaps this is one of the most important sentences in science.

For scientific progress has never consisted merely in our knowing more and more. It has also consisted in our developing increasingly better ways of discovering the limits of what we think we know.

That is why Holz’s question reaches far beyond artificial intelligence.

Who checks what AI can do?

The answer, in the end, is not the name of a single institution.

The answer is a way of organizing knowledge.

To organize oversight is to organize doubt—to build institutions in which no claim becomes final merely because it comes from the actor with the largest computer, the greatest capital, or the best access to the technology.

Science is not merely the organization of knowledge.

It is the organization of the right to doubt it.