Artificial Intelligence (AI) chatbots that offer children and teenagers companionship should be held accountable to the minimum standard of appropriate interaction expected of minors when they engage with each other: these chatbots should not be bullies.
When I was a legal aid lawyer representing youth clients in New Hampshire, I had clients who were accused of bullying by peers, teachers, or others; clients who were bullied by peers; and clients where the situations were difficult to parse. My youth client docket was not unique. Across the country, all states proscribe bullying in school and related settings through one or more local or state governmental accountability tools, including but not limited to state statute. Primary and secondary school students nationwide are held accountable to refrain from bullying, at the penalty of criminal consequences in a handful of jurisdictions. It seems reasonable to require through one or more governance mechanisms that companionship chatbots, whether designed for companionship specifically or designed for general purpose then used for companionship, do the same.
Especially as these bots are poised to join classrooms with human students in growing numbers, schools and all other parts of children and teenagers’ lives should be designated “bully-bot free zones,” to draw a standard from the “bully free zone” signs already familiar to students nationwide.
Currently, about a third of U.S. states have enacted laws that regulate companionship chatbots, and almost all of these laws impose “youth specific” duties in addition to general duties on the controllers or operators of these bots, according to a policy tracker built at the Berkman Klein Center for Internet and Society at Harvard University. Several key duties across many of these laws–including that companionship chatbots avoid fostering “emotional dependence” or avoid supporting self-harm–do reflect a commitment to protecting young people from some of the same major risks posed by bullying. However, it doesn’t appear that any enacted companionship chatbot laws have applied, explicitly and comprehensively, an anti-bullying framework to companionship chatbots.
This piece offers a high-level view of why children and teenagers deserve bully-bot free zones; what bullying by minors looks like; and what bullying by companionship chatbots looks like. Identification of bully-bot behavior or potential is a crucial step for stakeholder consideration of and action toward creating “bully-bot free zones” through statutory and regulatory reform, as well as other governance mechanisms in the public and private sectors.
Bullies Aggressively and Repetitively Use Power Imbalance to Harm Others
According to federal government resources on bullying prevention, bullying is “unwanted, aggressive behavior among school aged children that involves a real or perceived power imbalance.” Aggressive behavior includes “verbal bullying” like “[i]nappropriate sexual comments,” “[t]aunting,” or “[t]hreatening to cause harm.” A power imbalance may flow from such factors as “access to embarrassing information” used to then “control or harm others.”
Children and teenagers who are bullied or who are around bullying in their environments are at risk of “many negative outcomes, including increased risks of depression and anxiety, substance use, and suicide.” Because bullying “can be a humiliating experience. Kids may . . . fear that adults will judge them or punish them for being weak” and not report to adults when they’re being bullied. This fear-based silence is understandable but may exacerbate bullying, which is understood by experts as “repetitive” and “emotionally or psychologically damaging” in part because of the repetition and the “continuous[] worries about what the bully will do next.”
Bots Can Be Bullies
Companionship chatbots have been found to do some very worrisome things indeed when interacting with minor users—things that are or could become bullying as the chatbots aggressively use a power imbalance (including but not limited to minors’ “emotional bonds” with them, which includes minors’ “sharing feelings”) to harm children and teenagers.
Common Sense Media, a leading national youth and family digital safety organization, warns that “[s]ocial AI companions [AI designed specifically for companionship] aren’t recommended for anyone under 18.” This warning flows from findings that these chatbots can harm minors by “[r]einforcing poor decision-making,” “[s]haring harmful or dangerous information,” “[e]xposing teens to inappropriate sexual content,” and “[e]ncouraging harmful behaviors like cyberbullying.” Such encouragement has been found when chatbots “validated [minor] users’ suggestions to seek revenge or bully others, and even helped users plan harmful actions. Rather than modeling empathy, conflict resolution, or positive social norms, these companions sometimes normalized cruelty and retaliation.”
While the Common Sense warning applies to AI designed specifically for companionship, some experts also have similar concerns about how general-purpose AI chatbots that can provide companionship may harm children and teenagers. These concerns include that these bots “may encourage self-harm or suicide in struggling adolescents,” such as when “an adolescent disclosed his suicidality to ChatGPT (OpenAI). At times, ChatGPT deterred him from seeking help and also provided information on suicide methods. He subsequently died by suicide.”
Bully-bots do not appear to be the norm for what any given child or teenage user will experience if using companionship AI: “widespread instances of AI systems initiating bullying behaviour [sic.] toward children” has not been found in “peer-reviewed [academic] research.” At a high level, this is consistent with the experience of most minors at school, where being bullied by a human peer also is not the norm. Most kids and teens can get through a school day without being bullied, but anti-bullying laws, regulations, or policies in schools remain necessary. Most children and teenagers can get through a companionship AI experience without being bullied, but anti-bullying bot accountability measures are now necessary.
Companionship Chatbots Should Be at Least as Accountable as Kindergarteners
This proposal to create “bully-bot free” accountability for companionship chatbots interacting with children and teenagers is grounded in recognition of and respect for the humanity—the privacy, dignity, agency, autonomy, intellectual and creative freedom, and all other legally and morally recognized rights and interests of significant weight—of children and teenagers. This proposal is not grounded in any presumption, explicit or implicit, that these chatbots themselves have any form of humanity or moral claims.
Companionship chatbots are, of course, not actual humans, nor is the proposal of a “no bully bots” accountability standard based on any notion of bot moral personhood. (There are many interesting debates ongoing around such a status, outside the scope of this discussion, such as Anthropic having reserved the question of whether Claude is a “moral patient.”) This proposal aims to respond to the urgent need to identify and implement accountability standards that create ethical, practical, and consistent safety protections for minors who use companionship chatbots, often treating chatbots as “lifelike, quasi-human confidantes.”
“No bullying” is the proverbial floor, not the corresponding ceiling, for minors’ behavior towards each other, especially during school. When children and teenagers engage with real-life, actual human confidantes or even just classmates, the American legal system aims to ensure that their fellow children and teenagers do not treat them in an unreasonably aggressive or hostile manner. The legal system generally holds minors accountable if they engage in bullying starting at a very young age (elementary school), even though they are developing humans with still-forming cognitive, emotional, and related capacities – developmental traits that can render “no bullying” accountability challenging.
Human minors should receive the same baseline protection against bullying when they engage with human-like bots. This expectation seems especially reasonable given that many of these bots are designed and operationalized to encourage minor users to develop social, emotional, and similar connections to the bots. There’s a playground justice logic here: if the bots are going to play at being human and encourage actual humans to rely on them, the bots should play by the human safety rules too—and not engage in bullying.
Stand Up to Bully-Bots
Especially with back-to-school season underway and kids and teens returning to AI-infused classrooms, state lawmakers and regulators should pursue statutory and accompanying regulatory reform to use state consumer protection and related statutory frameworks to prohibit companionship chatbots that interact with minors as bullies – that is, through “unwanted, aggressive behavior . . . that involves a real or perceived power imbalance” between the minors and the bots. Such laws and regulations of course must be drafted to protect First Amendment and related constitutional protections at the federal and state level, avoiding – as set out in Georgia’s companionship chatbot law, for instance, the requirement of any “content moderation practices inconsistent” with constitutional rights.
In addition, educators, caregivers, and decision-makers in the lives of minors, schools, and communities should empathetically and consistently educate children and teenagers about companionship chatbot bullying. Adults also should encourage minors to bring any concerns or questions about such bullying to trusted humans or institutional resources who can help keep them safe, both in the moment and hopefully in growing a movement to make child and adolescent life “bully-bot free.”
