AI models can produce convincing descriptions of their reasoning, but there’s little way to tell whether those explanations reflect what’s actually happening inside. An emerging strand of research, however, suggests some models can detect when their internal activity has been tampered with, potentially suggesting some degree of self-reflection.
Whether AI can introspect has become a controversial question in both AI safety and cognitive science. If models could reliably describe what’s going on inside their own neural networks, it would give researchers a powerful new way to check what they’re really up to.
But there is widespread skepticism, and the idea is hard to test. Large language models are trained on vast amounts of text, much of which contains descriptions of human introspection. So AI has learned to talk convincingly about its inner life, regardless of whether those descriptions are grounded in anything.
However, a growing body of research now suggests models can tell when their neural activity has been altered by researchers. While it’s less clear whether they can detect exactly what’s changed, some researchers say the findings indicate at least a limited ability to reflect on their internal states.
“The fact that [an AI model] can notice the anomaly is insane,” Harvey Lederman, a philosopher at New York University and Anthropic, who co-authored a study investigating the phenomenon, told Quanta Magazine. “That is a mind-bending fact. It is wild, and it does seem robust to our various controls.”
Lederman’s study was published on the arXiv preprint server in March, while he was still at the University of Texas. The work is a follow-up to a 2025 report by Anthropic researcher Jack Lindsey, which pioneered the idea of injecting activity patterns associated with specific concepts into a model’s network.
In his experiments, Lindsey had found the best performing models could both detect injections and name the concept in question in about 20 percent of cases. Lederman and his colleague Kyle Mahowald, a linguist at the University of Texas, were skeptical. So they decided to test the idea on two of the largest openly available models, Alibaba’s Qwen3-235B and Meta’s Llama 3.1 405B.
They told the models that a researcher might be injecting “thoughts” into its mind and to let them know if it noticed anything. In trials where they artificially amplified patterns related to 821 concepts, Qwen flagged injections up to 53.9 percent of the time and Llama in up to 31.7 percent. In trials where they didn’t make any changes, neither model claimed a single injection.
The models sometimes named the injected concept correctly too—in up to 13.9 percent of trials for Qwen and 12.9 percent for Llama. But the pair found evidence this had little to do with the models knowing what had been injected.
For a start, wrong guesses bore no relation to the injected concept. Almost three-quarters of Qwen’s wrong guesses were simply “apple.” And when the researchers stopped amplifying a concept before the model began answering, correct guesses dropped sharply while detection rates held steady. That suggests the right answers came from a known side effect of the technique: Injecting a concept tends to make a model talk about it whatever the question.
The researchers concluded that the ability to detect tampering is distinct from whatever enables models to guess the concept. This idea was backed by a follow-up paper from Lindsey and colleagues published the same month that mapped out how anomaly detection works inside Google’s open Gemma 3 model. They suggested the mechanism emerges as raw models are trained into chatbots, and that the ability to identify concepts appears to rely on separate processes.
Not everyone is convinced the findings amount to introspection though. To probe the results, researchers at New York University revamped the experiment by simply telling a model to become obsessed with a particular concept rather than tampering with its network.
They then subjected Meta’s Llama and Google’s Gemma models to both concept injection and their new prompting trick and asked the models whether their internals had been interfered with. The models reported “injected thoughts” more often following the prompt than after a genuine injection.
When the researchers gave the models a third option to report that the prompt had been manipulated, they were unable to reliably differentiate between the two interventions. A model genuinely inspecting its own internal states should be able to spot that difference, the researchers argue.
The entire strand of research also raises thorny philosophical questions. Shauli Ravfogel, one of the authors of the New York University paper, points out that all a model’s workings run on its internal states, so almost anything it does could be dressed up as self-reflection. “If we say that this is introspection, the term ‘introspection’ becomes kind of meaningless,” he told Quanta.
Others argue the work cuts both ways. Steve Fleming, a University College London neuroscientist who studies introspection in humans, told Quanta the research suggests cognitive scientists may have oversold how clear the distinction between thinking and thinking about thinking really was.
Either way, the work is turning once purely philosophical questions about self-awareness into experimental ones. But we’re likely still a long way from settling the debate about what’s really going on inside the heads of today’s leading AI models.
