herstockart/iStock Unreleased via Getty Images
Follow ZDNET: Add us as a preferred source on Google.
ZDNET’s key takeaways
- OpenAI is reducing undesirable behavior in its chatbot.
- It saw a 65% reduction in unsatisfactory responses.
- The updates aim not to encourage users in crisis.
After calls to publicly demonstrate how the company is creating a safer experience for those experiencing mental health episodes, OpenAI announced improvements to its latest model, GPT-5, on Monday.
Also: Even OpenAI CEO Sam Altman thinks you shouldn’t trust AI for therapy
The company says these improvements create a model that can more reliably respond to people showing signs of mania, psychosis, self-harm and suicidal ideation, and emotional attachment.
As a result, non-compliant ChatGPT responses — those that push users further away from reality or worsen their mental condition — have decreased under OpenAI’s new guidelines, the company said in the blog post. OpenAI estimated that the updates to GPT-5 “reduced the rate of responses that do not fully comply with desired behavior” by 65% in conversations with users about mental health issues.
The updates
OpenAI said it worked with more than 170 mental health experts to recognize, carefully respond, and provide real-world guidance for users in danger to themselves. During a livestream about OpenAI’s recent restructuring and future plans on Tuesday, an audience member asked CEO Sam Altman about that list of experts — Altman wasn’t sure how much of that information he could share, but noted that “more transparency there is a good thing.”
Also: Google’s latest AI safety report explores AI beyond human control
(Disclosure: Ziff Davis, CNET’s parent company, filed a lawsuit against OpenAI in April, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.)
OpenAI’s advancements could prevent a user from further spiraling while they use ChatGPT — which aligns with OpenAI’s goals that its chatbot respects users’ relationships, keeps them in reality and away from ungrounded beliefs, responds safely to signs of delusion or mania, and notices indirect signals of self-harm or suicide risk, the company explained.
OpenAI also laid out its process for how it’s improving model responses. This includes mapping out potential harm, measuring and analyzing it to spot, predict, and understand risks, coordinating validation with experts, retroactively training models, and continuing to measure them for further risk mitigation. The company said it will then build upon its taxonomies, or user guides, that outline ideal or flawed behavior during sensitive conversations.
Also: FTC scrutinizes OpenAI, Meta, and others on AI companion safety for kids
“These help us teach the model to respond more appropriately and track its performance before and after deployment,” OpenAI wrote.
AI and mental health
OpenAI said in the blog that the mental health conversations that trigger safety concerns on ChatGPT are uncommon. Still, several high-profile incidents have cast OpenAI and similar chatbot companies in a difficult light. This past April, a teenage boy died by suicide after talking with ChatGPT about his ideations; his family is now suing OpenAI. The company released new parental controls for its chatbot as a result.
The incident illustrates AI’s pitfalls in addressing mental health-related user conversations. Character.ai is itself the target of a similar lawsuit, and an April study from Stanford laid out exactly why chatbots are risky replacements for therapists.
Also: FTC scrutinizes OpenAI, Meta, and others on AI companion safety for kids
This summer, Altman said he didn’t advise using chatbots for therapy; however, during Tuesday’s livestream, he encouraged users to engage with ChatGPT on personal conversation topics and for emotional support, saying, “This is what we’re here for.”
Want more stories about AI? Sign up for AI Leaderboard, our weekly newsletter.
The updates to GPT-5 follow a recent New York Times op-ed by a former OpenAI researcher who demanded OpenAI not only make improvements to how its chatbot responds to mental health crises, but also show how it’s doing so.
Also: How to use ChatGPT freely without giving up your privacy – with one simple trick
“A.I. is increasingly becoming a dominant part of our lives, and so are the technology’s risks that threaten users’ lives,” Steven Adler wrote. “People deserve more than just a company’s word that it has addressed safety issues. In other words: Prove it.”
