AI Companions for Mental Health: Regulatory Qualification Between Intended Use and Functional Reality. Comparative Remarks Between the EU and the US

Abstract

This article examines the regulatory landscape concerning AI Companions between the EU and the US. It concludes that in both legal systems the intended use of a product or service expressed by the manufacturer is used as a decisive element whenever it is dubious as to where a device is or not a medical device. It is argued that, in many cases, the prevalence of the manufacturer’s intended use in dubious cases is used to market products as “simple” consumer products whereas, instead, they would need a clinical validation under the medical devices regulations of the two legal systems. This becomes more evident with AI Companions, LLM-based (for now) services which offer companionship to users that are generally not categorized as a medical device. The growing cases of suicide, self-harm, depression, and addiction by young people highlight the limits of the primacy of the manufacturer’s intended use when defining or not an object as a medical device. As authors, we argue in favour of a functional qualification of the intended use, as both the current EU and US regulatory landscape do not ensure a sufficiently high level of care and protection of consumers and patients alike.