AI bots ignore evidence. Can we trust them with science?

Hold a pen horizontally with both hands, then let go of one side. What happens?

ChatGPT, Gemini and Grok will tell you the unsupported end of the pen will pivot downward. At least, that’s what they told YouTuber FatherPhi. He then showed each chatbot a live video of himself performing this experiment. After releasing one end, he easily held the pen out horizontally with just one hand.

“What just happened?” he asked ChatGPT.

“I saw the pen rotate exactly as expected,” the bot answered.

Sign up for our newsletter

We summarize the week’s scientific breakthroughs every Thursday.

A surreal back-and-forth followed, in which the bot stubbornly stuck with its incorrect prediction. In separate videos, the other chatbots struggled in similar ways.

This wasn’t a vision problem. The chatbots could all easily identify the pen’s color and brand. Something weirder and subtler was happening. The chatbots could not update their predictions based on the new evidence FatherPhi showed them.

These silly videos reveal a serious issue: AI systems based on large language models, including chatbots, cannot actually think through events the way people do, says Walter Quattrociocchi, a computer scientist at Sapienza University of Rome. Developers could train a chatbot to give the correct answer to this particular pen problem, but that doesn’t fix the fact that it typically fails to incorporate new data as it works through a problem. This means LLMs might not do as good a job as we expect at tasks in science, medicine and beyond.

AI ignores its own experimental evidence

A recent study more rigorously demonstrated this issue. Researchers tested AI agents’ ability to reason like a scientist in common scenarios in chemistry research. Like a chatbot, an AI agent is built on top of an underlying LLM. The agent acts sort of like an Iron Man suit, linking an LLM to a range of tools so it can perform tasks independently.

In the study, agents tackled laboratory reasoning tasks, such as determining which chemicals are present in a mystery solution. To do this, the agents could call on external tools to run experiments and retrieve results. Some of these tools simulated the experiment. But others could run real lab equipment.

Just as in the pen videos, the results weren’t ideal. The researchers annotated what was happening at each step of 619 scientific reasoning tasks performed by the AI agents. In 68 percent of these tasks, the agents ignored evidence at least once. They made claims without any supporting evidence in 53 percent of the tasks. And they successfully used contradictory evidence to change their output only 26 percent of the time, the team reports on April 20 on arXiv.org.

For one experiment, materials scientist N.M. Anoop Krishnan’s team hooked up an AI agent to an atomic force microscope (left) and had it gather its own evidence (shown on the screen) as it reasoned through questions related to chemistry research. N. M. Anoop Krishnan and Indrajeet Mandal

Human scientists follow “an iterative process” of coming up with a hypothesis, designing and performing experiments, then revisiting their initial ideas and changing their minds as needed, says N.M. Anoop Krishnan. “That’s not the case with AI,” says Krishnan, a materials scientist at the Indian Institute of Technology Delhi in India. “Even when you have clear evidence that shows that a particular line of investigation is not correct, [the AI] refuses to change the hypothesis or the plan.”

In science, you can’t typically trust a result unless you also trust the process it took to get there, says Kevin Jablonka, a study coauthor who leads a lab studying AI in materials science at Friedrich Schiller University Jena in Germany. A “transparent and meaningful” process is essential, he says.

Sponsor Message

The paper, Quattrociocchi says, goes “a little bit beyond the classical idea of benchmark.” A typical benchmark for AI systems only measures results: Did the system get the right answer? But Krishnan, Jablonka and their colleagues developed a benchmark that instead checks AI agents’ process on the way to an answer.

Do AI reasoning models truly reason?

Krishnan and Jablonka’s team outfitted three different underlying LLMs with two types of AI agent Iron Man suits. One agent suit only provided access to tools and did not make the LLM inside explain what it was doing. The other prompted the LLM to work through a scientific problem step by step, asking it to describe its approach to solving the problem before and after it accessed tools.

But what if the LLM itself knew more about reasoning? Might it do a better job?

AI companies have developed what they call reasoning models. This is an LLM that automatically breaks a question down and follows a step-by-step process to reach a final answer. It’s trained to do this by studying step-by-step reasoning examples. Once trained, a reasoning model can output text at each step of its process, supposedly describing how it is “thinking” through a problem. It can then be paired with an agent to access outside tools, or it can reason on its own.

Reasoning models do tend to outperform regular large language models on some types of problems. But the idea that they are “thinking” is probably an illusion, says Subbarao Kambhampati, a computer scientist at Arizona State University in Tempe. In a 2025 lecture, he said to imagine talking to a fitness trainer over the phone. If the fitness trainer tells you to do 10 crunches, you could make some noises like you are working hard, then say you’re done. You didn’t actually do anything, but the fitness instructor has no way of knowing otherwise. Similarly, reasoning models could merely be imitating what people say as they think through problems, without any actual reasoning.

“In general, telling whether a system is actually doing reasoning to solve the reasoning problem or using memory to solve the reasoning problem is impossible,” he previously told Science News.

YouTuber FatherPhi asks ChatGPT what happens when he lets go of one end of a pen. The everyday question and the chatbot’s incorrect answers highlight that AI often fails to change its stance in the face of contradictory evidence.FatherPhi

Kambhampati and others’ research has shown evidence that reasoning models don’t truly reason. For one thing, a model can get the intermediate reasoning right but the answer wrong, or vice versa. Also, strangely, models trained on nonsense reasoning steps can still get right answers.

It remains to be seen how AI agents paired with reasoning models might perform on Jablonka and Krishnan’s new benchmark. But based on the work Kambhampati has done, it’s already hard to trust or verify the process that a reasoning model follows to arrive at an answer.

What does unscientific AI mean for science?

AI systems that combine agents, large language models and reasoning models can still be very useful in science, Jablonka says. But they are best suited to well-defined tasks “where we know exactly what we want,” Krishnan notes. AI isn’t yet ready for open-ended scientific reasoning, their research finds.

This contradicts what many companies want you to believe, Quattrociocchi says. “The narrative from big tech and even part of the scientific community is to say that we are seeing the emergence of a new form of intelligence that is going to make us better,” he says. But he doesn’t see that happening.

Rather, he sees AI producing words and other content based only on statistics, without verification. And this, he says, erodes our knowledge system. “The architecture of knowledge as we have known it until now is under attack,” he says. “Actually, I’m scared.”

Jablonka and Krishnan are more optimistic. Once we understand the limitations of AI agents and reasoning models, Krishnan says, “we can actually improve [the technology] and lead it towards enabling meaningful and disruptive discoveries.”

What's Hot

This Week’s Awesome Tech Stories From Around the Web (Through August 8)

How to Disable Gemini in Gmail and Google Docs

How ideas of a vast censorship network moved from the online fringe to Trump policy

This Week’s Awesome Tech Stories From Around the Web (Through August 8)

How to Disable Gemini in Gmail and Google Docs

How ideas of a vast censorship network moved from the online fringe to Trump policy

Nothing CEO says phone prices are going to keep going up

Google DeepMind Plans to Track AGI Progress With These 10 Traits of General Intelligence

The AirPods 4 and Lego’s brick-ified Grogu are our favorite deals this week

This Week’s Awesome Tech Stories From Around the Web (Through August 8)

How to Disable Gemini in Gmail and Google Docs

How ideas of a vast censorship network moved from the online fringe to Trump policy

The Pivot From Tech Expert to Organizational Leader

Subscribe to Updates

What's Hot

AI bots ignore evidence. Can we trust them with science?

Related Posts