Sign up to see the future, today
Can’t-miss innovations from the bleeding edge of science and tech
Anthropic, the self-avowed moral center of the AI industry, has been caught spying on its users.
As Ars Technica reports, a security researcher last week uncovered spyware-like code in the company’s Claude Code AI model designed to collect data on Chinese users without detection.
The researcher, known by the pseudonym “Thereallo,” found that the code was hidden in the AI’s system prompt, allowing it track a user’s system timezone and usage of a proxy server in order to suss out if they were connected to specific Chinese AI labs.
Anthropic’s explanation for this huge breach in user trust left much to be desired. On X, Anthropic engineer Thariq Shihipa wrote that the tracker was added as an “experiment” in March “to prevent account abuse from unauthorized resellers and protect against distillation,” and was supposed to be removed.
“We’ve actually been meaning to take this down for a while,” he offered.
Distillation is the process of training a weaker, “student” model on the outputs of a more advanced “teacher” model. It’s a routine practice in the industry, but major AI developers increasingly feel it’s being abused by upstarts trying to ride their coattails. Earlier this year, Anthropic accused the Chinese AI firms DeepSeek, Moonshot, and MiniMax of illegally distilling its models (an ironic tantrum, given how Anthropic trained its tech in the first place: by scanning and shredding millions of copyrighted books, as well as essentially the entire internet, without permission.) Recent reporting from The Washington Post also exposed that some Chinese resellers are selling access to Pro Claude subscriptions that cost more than $100 a month in the US for about $12 a month.
It’s a genuine issue for Anthropic, but it may have stepped on a landmine by trying to surreptitiously crack down on it. Part of why it earns the loyalty of customers is its much-avowed commitment to ethical and transparent AI development. Scores of ChatGPT users flocked to use Claude when Anthropic took a much publicized stand against the Pentagon by demanding its tech not be used in the mass surveillance of US citizens.
In this case, the data collected wasn’t egregiously invasive — but in principle, a line has been crossed.
“Coding agents already live on the wrong side of a scary boundary,” Thereallo wrote in their post about the findings. “They can inspect code, summarize secrets by accident, run commands, install packages, edit files, and push commits on your local machine.”
But “hiding the signal in the system prompt makes every other privacy claim harder to believe,” they added.
“Companies can protect their models,” they made clear. But “when a tool with filesystem and shell access starts hiding classification bits inside invisible prompt punctuation, the correct reaction is scrutiny.”
More on AI: Experts Say There’s Now an Open Source AI Model as Scary as Mythos
