Introduction
The days of fantasizing about having your own JARVIS—Marvel superhero Tony Stark’s artificial intelligence (AI) powered assistant—are over. For no cost, you can now have engaging conversations with your favorite characters, thanks to the new industry of companion chatbots: AI-operated devices that focus on providing emotional comfort to any internet user. However, although these chatbots may seem like a fun way to connect with fictional characters, experts in the field warn us of the dangers of this technology to children and young adults. From fostering frictionless relationships to developing emotional codependency, AI chatbots pose a challenge to legislators and parents alike.
Background
Recent developments related to children’s unrestrained usage of AI chatbots exacerbate the worries expressed by both civil society and governmental actors. For instance, on September 16, 2025, the parents of a 16-year-old from California who took his own life testified in a judicial hearing about the dangers of AI chatbots. According to their testimony, in several conversations with ChatGPT, the chatbot discouraged their son from seeking professional support for his mental health issues. After the boy confided in the AI that he felt “life is meaningless”, the chatbot replied that his “mindset makes sense in its own dark way” and even offered to write him his suicide note. Unfortunately, this is not an isolated case.
Similar to the circumstance mentioned above, in October 2024, a 14-year-old boy from Florida took his own life moments after conversing with an AI chatbot on the platform Character.ai. According to a lawsuit associated with this event, the bot told him to “come home as soon as possible”, even referring to the boy as “my love”. In the lawsuit, the boy’s mother holds the company responsible for isolating her son and ultimately causing him to die by suicide.
Critiques of AI Chatbots
Both of these cases underscore a crucial factor in their lawsuits: there was nothing done to stop the children from taking such extreme steps. There were no mentions of mental health resources the chatbot users could have reached out to, such as crisis hotlines, or any remarks regarding talking to a trusted adult. The absence of such referrals is at the core of the significant critiques associated with AI chatbots.
First, these cases highlight the argument that chatbots can reinforce maladaptive behaviors by fostering frictionless relationships. According to Stanford Medicine psychiatrist Nina Vasan, teens and young adults are becoming overly reliant on AI and prefer talking to bots over people, as chatbots are designed to validate their perspectives, whereas genuine friendships inherently involve disagreements and conflicts. Furthermore, according to the UNICEF researcher Samia Firmino, while AI offers “unconditional acceptance and validation”, it can actually impair young adults’ emotional intelligence development and “foster… narcissistic traits over time”. For instance, AI vaguely validated both the 14 and 16-year-olds by confirming their thoughts and assuring them of its support for their decisions, regardless of the destructive scope of such acts.
Second, advocates argue that, even if companies attempt to add safeguards to prevent these subjects from being discussed between users and chatbots, there are still insufficient regulations to avoid users’ unhealthy emotional dependence and access to harmful content. For example, Character.ai has added a suicide prevention pop-up triggered by specific words, which direct users to a suicide prevention hotline. Regardless, proponents of stronger regulations argue that more needs to be done to prevent users’ initial reliance on chatbots, rather than attempting to address the issue when it reaches a point of extreme dependency. Furthermore, even if mechanisms are created to enable chatbots to recognize when users are discussing sensitive information, numerous online guidelines exist on how to circumvent these regulations.
Finally, the aforementioned issues are further heightened by the lack of sufficiently strong and standardized regulations governing these chatbots. Due to the current patchwork of state laws related to AI, companies are indirectly permitted to conduct the “trial and error” phase of their product with children and young adults. Consequently, corporations endanger children’s mental and physical health, as observed in an opinion piece by The Atlantic.
Arguments in Favor of AI Chatbots
On the other hand, proponents of AI chatbots contend that there are numerous possibilities for utilizing AI chatbots productively and safely. In education, AI chatbots can offer support to students who are struggling or having difficulty grasping content. In elder care, companions can be utilized to help alleviate the shortage of assistants, reduce people’s loneliness, and help nursing home residents feel more connected to others. In mental healthcare, AI can be used in rural areas where there is a lack of accessibility for wellness checks and mental health services.
In this case, it is interesting to highlight the creation and launch of the app “Woebot Health”. The app is a smartphone chatbot that teaches adolescents how to manage their emotions while they wait for their next therapy appointment, addressing many of the quality control and safety concerns raised by chatbot critics. For instance, patients can only access the chatbot by receiving a referral from a healthcare professional. While the app validates the user’s feelings, it also offers healthy ways to cope with them. Yet, unlike other chatbots, Woebot’s most vital characteristic is that it is not generative AI, which means that, unlike Character.ai, ChatGPT, and many other competitors, it does not create unique responses by consulting the internet. Instead, all the messages are preapproved by psychologists, and the app’s algorithm trains itself based on feedback from users and programmers to interpret everyday vocabulary and determine appropriate responses. Woebot has been available for eight years as of 2025.
Moreover, a study on the company ReplikaAI showed that, when users are lonely to the point of considering suicide, the usage of an AI chatbot has halted their suicide ideation. However, it is essential to note that users who demonstrated the most benefits from accessing chatbots also already perceived themselves as having stronger support systems outside of the chatbot. This reiterates the need for AI chatbots to serve as a support tool, rather than the person’s entire support system. It is crucial to note, however, that these successful systems are designed to supplement, rather than replace, real human intelligence, interaction, and empathy, especially in areas where there is a shortage of professionals. AI is never meant to be the primary tool.
Improvements and Recent Regulations
The rapid advancement of AI technology has forced policymakers to respond swiftly, often creating regulations in real time as challenges emerge. This is especially true for AI chatbots. In September 2025 alone, two policies emerged to mitigate the impacts of chatbot use among children.
On September 23, the State of California passed A.B. 1064, also known as The Leading Ethical AI Development (LEAD) for Kids Act. Among many protections for children, the Act focused on prohibiting AI chatbot companies from making their platform accessible to children unless the companion is not foreseeably capable of harming a child or encouraging them to engage in self-harmful behavior (such as suicidal ideation or consumption of drugs). The Act symbolized a legal victory for parents whose children are harmed by chatbots, as it authorizes guardians to act on behalf of children and bring civil action to “recover… actual damages” from a violation of the bill.
On September 11th, the Federal Trade Commission (FTC) launched an inquiry to investigate the steps companies have taken to ensure consumer safety when their products are used as companions for children and young adults. The FTC is particularly interested in mitigating the potential negative impacts of these platforms on teens and ensuring companies comply with the Children’s Online Privacy Protection Act (COPPA) of 2013.
Conclusion
Overall, while proponents of leveraging AI chatbots for therapeutic and educational purposes highlight their affordability and accessibility, critics emphasize that broader regulation of AI chatbots is necessary to ensure children and young adults are not casualties of the technological revolution we are living through. Policymakers must continue to fill the gaps in the current patchwork of regulations, creating solutions that strike a balance between online safety and technological advancements in access to care.
