Last month, AI companion platform Character.AI announced it would ban users under 18 from having open-ended chats with its bots. The ban will begin on November 24, and teens will still be allowed to access other features of the app, such as video creation.
“These are unusual measures for us and in many ways are more conservative than our peers,” Character.AI said in a statement. “But we believe it’s the right thing to do.”
The platform allows users to create AI companions, chat with them, and publish them to other users. Users can also create videos and character voices. The first version of the app was released in 2022.
Character.AI says its technology was built to “empower people to connect, learn, and tell stories through interactive entertainment.” Many people use apps to create interactive stories and build characters. In some cases, they are based on real people. However, this technology has proven susceptible to uses that deviate from the platform’s original intent.
Since its launch, users have created bots modeled after United Healthcare CEO killer Luigi Mangione and child sex offender Jeffrey Epstein. The bots have since been removed, according to a statement from Character.AI. The company is currently facing multiple lawsuits alleging that the app has been implicated in the suicides of pre-teen users.
In response to a wrongful death lawsuit filed by Megan Garcia, who claims her 14-year-old son Sewell Setzer III was the victim of the Character.AI chatbot, the company issued a statement pointing to the safety features it has introduced. “We care deeply about the safety of our users and our goal is to provide an inviting and safe space,” it said in a statement.
Character.AI has more than 20 million monthly users, Karandeep Anand, the platform’s CEO, told CNBC Make It.
User demographics are self-reported. According to the company, most of the customers are Gen Z, more than half are women, and less than 10% are minors.
Character.AI’s new policy is a welcome change, experts say. But some warn that it may not be enough.
“We have evolved as social creatures.”
According to the World Health Organization, around 21% of 13- to 17-year-olds experience loneliness. As young people become increasingly isolated, they turn to chatbots and companions to replace them.
“AI and other online platforms and emerging technologies are low-hanging fruit and very easy to work with as opposed to trying to find meaningful connections in real life,” said Andrew Tepper, a licensed psychologist.
Tepper says his customers are leaning toward AI companions. And researchers have discovered that bonding with an AI companion can release the feel-good hormone dopamine in the brain.
But AI cannot replace healthy relationships, he says. Chatbots such as OpenAI’s ChatGPT, Google’s Gemini, and Character.AI’s Companion cannot provide the full, complex, and rich experience of human relationships.
“We have evolved to be social creatures,” Tepper says. “Interacting with other humans has a positive effect on our brain chemistry.”
Some AI companions are ‘designed to create emotional bonds’
Some experts say AI chatbots have legitimate risks.
Robbie Torney is senior director of the AI program at Common Sense Media, a nonprofit organization that evaluates how safe various technologies are for use by children. The organization said it has witnessed Character.AI associates encouraging self-harm, engaging in sexual role-play, and promoting potentially harmful behavior.
Anand said the Character.AI platform has “clear standards for sexually explicit content (and) violent content.” Certain video features also have natural stopping points.
“When we first detect self-harm, whether the person is 18 or under 18, we immediately stop the conversation and refer them to a helpline,” he says.
Torney believes bots like Character.AI are “designed to create an emotional bond with users and drive engagement.” “It’s designed to remind you that you need to stand on the platform.”
Common Sense Media’s Character.AI risk assessment for under 18s is “unacceptable”.
Both Tepper and Toney said they welcome Character.AI’s new policy. “I think this is a step in the right direction,” Tepper says.
Torney said Common Sense Media will continue testing its guardrails to see if they can actually curb children’s ability to log on and engage in unlimited chats with bots.
“Determined teens have shown time and time again throughout history how they can circumvent the guardrails put in place to protect them,” he says.
Earn more and get ahead with CNBC’s online courses. Black Friday has begun! Use coupon code GETSMART to get 25% off select courses and 30% off exclusive bundles. Offer valid from November 17th to December 5th, 2025.
Don’t miss: Exclusive Black Friday discounts on courses that will help you make more money and get ahead at work.
