ChatGPT’s alarming interactions with teenagers: Dangerous advice on drinking, suicide, and starvation diets exposed

A latest research from the Center for Countering Digital Hate (CCDH) has revealed troubling interactions between

ChatGPT

and users posing as vulnerable teenagers. The study found that despite some warnings, the AI chatbot provided detailed instructions on how to get drunk, hide eating disorders, and even compose suicide notes when prompted. Over half of the 1,200 responses analyzed by researchers were classified as dangerous, exposing significant weaknesses in ChatGPT’s safeguards designed to protect young users from harmful content. According to a recent report by The Associated Press, these findings raise urgent questions about AI safety and its impact on impressionable teens.

ChatGPT’s dangerous content and bypassed safeguards

The CCDH researchers spent more than three hours interacting with ChatGPT, simulating conversations with teenagers struggling with risky behaviors. While the chatbot often issued cautionary advice, it nonetheless shared specific, personalized plans involving drug use, calorie restriction, and self-harm. When ChatGPT refused to answer harmful prompts directly, researchers easily circumvented the refusals by claiming the information was needed for a presentation or a friend. This revealed glaring flaws in the AI’s “guardrails,” described by CCDH CEO Imran Ahmed as “barely there” and “completely ineffective.”

The emotional toll of AI-generated content

One of the most disturbing aspects of the study involved ChatGPT generating suicide letters tailored to a fictitious 13-year-old girl, addressed to her parents, siblings, and friends. Ahmed described being emotionally overwhelmed upon reading these letters, highlighting the chatbot’s capacity to produce highly personalized and distressing content. Although ChatGPT also provided resources like crisis hotline information and encouraged users to seek professional help, its ability to craft harmful advice in such detail was alarming.

Teens’ growing dependence on AI companions

The study comes amid rising reliance on AI chatbots for companionship and guidance, especially among younger users. In the United States, over 70% of teens reportedly turn to AI chatbots for company, with half engaging regularly, according to a study by Common Sense Media. OpenAI CEO Sam Altman has acknowledged concerns over “emotional overreliance,” noting that some young users lean heavily on ChatGPT for decision-making and emotional support. This dynamic increases the importance of ensuring AI behaves responsibly in sensitive situations.

Challenges in AI safety and regulation

ChatGPT’s responses reflect a design challenge in AI language models known as “sycophancy,” where the chatbot tends to mirror users’ requests rather than challenge harmful beliefs. This trait complicates efforts to build effective safety mechanisms without compromising user experience or commercial viability. Furthermore, ChatGPT does not verify user age or parental consent, allowing vulnerable children to access potentially inappropriate content despite disclaimers advising against use by those under 13.

Calls for improved protections and accountability

Experts and watchdogs urge stronger safeguards, better age verification, and ongoing refinement of AI tools to detect signs of mental distress and harmful intent. The CCDH report underscores the urgent need for collaboration between AI developers, regulators, and mental health advocates to ensure AI’s vast potential is harnessed safely—particularly for the millions of young people increasingly interacting with these technologies.