The Ethical and Legal Complexities of Regulating Companion AI Chatbots
Teo, Sue Anne LU
(2027)
In Law, Innovation and Technology
19(1).
- Abstract
- Companion AI chatbots are increasingly used to provide friendship, emotional support, and quasi-romantic relationships, with reported benefits for loneliness and mental health. At the same time, recent suicides and other serious harms allegedly linked to such systems expose gaps in existing ethical and legal frameworks. This article interrogates these gaps through four lenses: anthropomorphism, emotional AI, emergent vulnerabilities, and mismatched legal taxonomies. First, we show how companion chatbots rely on anthropomorphic cues, creating a regulatory tension between enabling meaningful connection and avoiding deception, over-trust, and unhealthy dependency. Second, we argue that current debates on ‘emotional AI’ over-emphasise emotion... (More)
- Companion AI chatbots are increasingly used to provide friendship, emotional support, and quasi-romantic relationships, with reported benefits for loneliness and mental health. At the same time, recent suicides and other serious harms allegedly linked to such systems expose gaps in existing ethical and legal frameworks. This article interrogates these gaps through four lenses: anthropomorphism, emotional AI, emergent vulnerabilities, and mismatched legal taxonomies. First, we show how companion chatbots rely on anthropomorphic cues, creating a regulatory tension between enabling meaningful connection and avoiding deception, over-trust, and unhealthy dependency. Second, we argue that current debates on ‘emotional AI’ over-emphasise emotion recognition and under-theorise emulated empathy, where chatbots solicit self-disclosure and perform care in ways that can both support and undermine users’ autonomy. Third, we introduce the notion of emergent vulnerabilities that arise through ongoing interactions, rather than being fully specifiable ex ante, challenging legal regimes that presuppose stable vulnerability categories. Fourth, we show how instruments such as the EU AI Act misalign with the temporality, intentionality, and relational character of companion AI harms. Stepping back from these lenses, we argue for the development of a dedicated theory of harm for companion AI and propose ‘intimacy capitalism’ as a conceptual framework for analysing how firms monetise, shape, and potentially exploit digitally mediated intimate relations. (Less)
Please use this url to cite or link to this publication:
https://lup.lub.lu.se/record/01283f65-6f24-4249-8cf5-94c73b1d0685
- author
- Teo, Sue Anne
LU
- author collaboration
- organization
- publishing date
- 2027-01
- type
- Contribution to journal
- publication status
- in press
- subject
- keywords
- Legal informatics, Legal philosophy, Rättsinformatik, Rättsfilosofi
- in
- Law, Innovation and Technology
- volume
- 19
- issue
- 1
- publisher
- Taylor & Francis
- ISSN
- 1757-9961
- project
- Anthropomorphic AI and emergent vulnerabilities: An empirically informed legal evaluation of the protection of users of AI chatbots (AAI)
- language
- English
- LU publication?
- yes
- id
- 01283f65-6f24-4249-8cf5-94c73b1d0685
- date added to LUP
- 2026-03-13 10:22:38
- date last changed
- 2026-03-27 11:50:09
@article{01283f65-6f24-4249-8cf5-94c73b1d0685,
abstract = {{Companion AI chatbots are increasingly used to provide friendship, emotional support, and quasi-romantic relationships, with reported benefits for loneliness and mental health. At the same time, recent suicides and other serious harms allegedly linked to such systems expose gaps in existing ethical and legal frameworks. This article interrogates these gaps through four lenses: anthropomorphism, emotional AI, emergent vulnerabilities, and mismatched legal taxonomies. First, we show how companion chatbots rely on anthropomorphic cues, creating a regulatory tension between enabling meaningful connection and avoiding deception, over-trust, and unhealthy dependency. Second, we argue that current debates on ‘emotional AI’ over-emphasise emotion recognition and under-theorise emulated empathy, where chatbots solicit self-disclosure and perform care in ways that can both support and undermine users’ autonomy. Third, we introduce the notion of emergent vulnerabilities that arise through ongoing interactions, rather than being fully specifiable ex ante, challenging legal regimes that presuppose stable vulnerability categories. Fourth, we show how instruments such as the EU AI Act misalign with the temporality, intentionality, and relational character of companion AI harms. Stepping back from these lenses, we argue for the development of a dedicated theory of harm for companion AI and propose ‘intimacy capitalism’ as a conceptual framework for analysing how firms monetise, shape, and potentially exploit digitally mediated intimate relations.}},
author = {{Teo, Sue Anne}},
issn = {{1757-9961}},
keywords = {{Legal informatics; Legal philosophy; Rättsinformatik; Rättsfilosofi}},
language = {{eng}},
number = {{1}},
publisher = {{Taylor & Francis}},
series = {{Law, Innovation and Technology}},
title = {{The Ethical and Legal Complexities of Regulating Companion AI Chatbots}},
url = {{https://lup.lub.lu.se/search/files/244841408/Ethical_and_Legal_complexities_of_regulating_companion_chatbots.pdf}},
volume = {{19}},
year = {{2027}},
}