Character.AI has implemented new features and security policies to create and interact with the AI-powered virtual personalities it hosts. The new measures aim to make the platform safer for all users, but especially for younger users. The update includes more control over how minors interact with the AI chatbot, more content moderation, and better detection of AI discussing topics such as self-harm.
Although not cited in the blog post about the update, Character AI linked to the announcement in a post on of taking his life. . His family has filed a wrongful death lawsuit against Character.AI, citing a lack of safeguards for AI chatbots as a contributing factor to his suicide.
AI Chat Railings
The release of Character AI introduced several new security features for the platform. For example, if the model detects keywords related to suicide or self-harm, it will display a pop-up urging the user to go to the National Suicide Prevention Lifeline and related resources. AI will also be better at detecting and removing inappropriate content in a conversation, with particular sensitivity when users are under 18.
Presumably, minors would already have restricted content in conversations, but Character.AI may have increased that sensitivity even further. In cases where this is not enough, entire chatbots have been removed.
“We conduct proactive detection and moderation of user-created characters, including the use of custom and industry-standard block lists that are updated regularly. We proactively remove characters that violate our guidelines in response to user reports. Terms of service,” Character.AI explained. in his position. “Users may notice that we recently removed a group of characters who have been flagged as violators, and these will be added to our custom block lists in the future.”
Other new features have more to do with helping ground users. Therefore, you will see a notification when you have spent an hour on the platform asking if you want to continue as a way to help you not lose track of time. You'll also see more prominent disclaimers emphasizing that the AI is not a real person. Such disclaimers already exist in conversations, but Character.AI wants to make it impossible to ignore.
These safety features are the flip side of how Character.AI has made interacting with chatbots feel more like talking to a real person, including voices and two-way voice conversations available with the Character Calls feature. Still, the company is likely interested in ensuring its services are as secure as possible, and its moves could inform how others in the space shape their own AI chatbot personas.