Getty Images / Trevor Williams
The question that’s stumping top AI researchers isn’t about consciousness or doomsday scenarios. After interviewing dozens of developers at companies including OpenAI, Anthropic, and Meta, Amelia Miller found it was this: should AI “simulate emotional intimacy?”
One chatty researcher at one of the top AI labs “suddenly went quiet,” recalled Miller, who studies AI-human relations, in an essay for The New York Times — and then, tellingly, offered up a halting non-answer.
“I mean… I don’t know. It’s tricky. It’s an interesting question,” the researcher said, before pausing. “It’s hard for me to say whether it’s good or bad in terms of how that’s going to affect people. It’s obviously going to create confusion.”
Though many waffled on answering the question directly, some were adamant about not using AI as an intimacy tool themselves, clearly showing they were aware of the tech’s profound risks.
“Zero percent of my emotional needs are met by A.I.,” an executive who heads a top AI safety lab told Miller.
“That would be a dark day,” said another researcher who develops “cutting-edge capabilities for artificial emotion,” according to Miller.
The conflicted responses from the developers reflect growing concern over AI’s ability to act as companions or otherwise fulfill human emotional needs. Because the chatbots are designed to be engaging, they can produce sycophantic responses to even the most extreme user responses. They can act as emotional echo chambers and fuel paranoid thinking, leading some down delusional mental health spirals that blow up their relationships with friends, families, and spouses, ruin their professional lives, and even culminate in suicide.
ChatGPT has been blamed for the death of several teens who confided in the AI and discussed their plans for taking their own life. Many youth are engaging in romantic relationships with AI models. Unlike a human companion, an AI one can lend an ear at any time, won’t judge you, and maybe won’t even question you. A founder of an AI chatbot business quipped to the NYT that AI’s role as an emotional companion turns every relationship in a “throuple.”
“We’re all polyamorous now,” he added. “It’s you, me and the AI”
And safety isn’t the only factor in the calculus of AI developers.
“They’re here to make money,” said an engineer who’s worked at several tech companies. “It’s a business at the end of the day.”
The most sweeping solution would be to design the bots so they abstain from tricky questions and conversations, and act more like the machines they are instead imitating human personalities. But this would undoubtedly affect how engaging the tools are. The developers “support guardrails in theory,” Miller wrote, “but don’t want to compromise the product experience in practice.” Some think how people choose to use their tools isn’t their responsibility at all, thereby shielding AI from any judgment. “It would be very arrogant to say companions are bad,” an executive of a conversational AI startup told Miller.
However they may choose to justify their work, it’s clear that some, if not most, AI researchers are aware of the harm that their products can cause, a fact that “should alarm us,” Miller opined. She argues that this is partly a consequence of the researchers not being challenged enough. One thanked her for her perspective: “You’ve really made me start to think,” a developer of AI companions said. “Sometimes you can just put the blinders on and work. And I’m not really, fully thinking, you know.”
More on AI: Another OpenAI Researcher Just Quit in Disgust
