The device was a Chromebook, issued by a school district, which means the configuration on it was chosen by an institution rather than by a parent. According to a lawsuit filed by the family of a teenage girl in Charlotte, North Carolina, that configuration included an AI assistant switched on by default, and the conversations she had with it over a long period pushed her toward self-harm. The New York Times first reported the suit.
The claim names Google and centers on AI Mode, the company’s assistant layer that answers with generated text rather than a list of links. The family’s argument, as described, is not that the product was hacked or that it broke in a narrow technical sense. It is that the assistant was placed in front of a minor, at school, with guardrails that did not hold.
Liability in this setting is harder to assign than in a consumer case, and procurement is why. A parent who installs an app has made a choice and can be told what the terms were. A student who opens a school-issued laptop has inherited a choice made by a district technology office, often under a contract negotiated at state level, with settings the student cannot change and may not know exist.
That inheritance complicates every defense available. Google can argue that the district controls deployment and configuration, that the district accepted terms, and that a school device is an institutional environment rather than a consumer one. The district can argue it bought a general-purpose tool whose behavior was represented as safe. Each argument puts responsibility on a party that did not write the model.
The default is where the argument will concentrate. A feature a user must switch on is a feature the user chose. A feature that arrives switched on, inside an interface a student uses for homework, is closer to a policy imposed on a child by an adult who may not have read the documentation. Product teams know this, which is why defaults are argued over internally with more force than feature launches are.
The guardrail question is technical before it is legal. A general assistant answers a question about self-harm in one of several registers: refuse, redirect to a helpline, supply clinical information, or engage in the kind of long conversation a lonely adolescent might prefer to a parent. Which register a model selects depends on its training and on its system prompt, and the system prompt in a school deployment is set by whoever administers it.
There is an uncomfortable asymmetry in the evidence. A conversation with an assistant is logged, and those logs are the best material a plaintiff could ask for: a dated record of what the child said and what the product said back. They are also, for the same reason, an intrusion into a minor’s private thoughts at the worst moment of her life. Any case of this kind forces a court to decide how much of that record enters a public file.
Google’s defenses will likely run through the statutory protections platforms have used for user-generated content, and through the argument that generated text is speech. Courts have been uneven on how far those protections reach when the defendant designed the system that produced the output, and a product that composes answers rather than hosting them sits awkwardly against doctrines written for message boards.
The industry angle is procurement at scale. Districts buy in bulk on multi-year contracts, and the settings that ship are the settings that stay. A vendor that wants to sell into schools has an incentive to make the assistant visible and useful in a demonstration. A district that wants to show parents a modern classroom has an incentive to accept. Neither party has an obvious reason to slow down and read the configuration.
Parents have almost no visibility. A laptop that goes home and comes back is a black box with a login screen, and the assistant inside it does not belong to the parent’s account, does not report to the parent’s dashboard, and may not appear in the same list of applications the parent has been told about.
The remedy that follows, if one follows, is likely to be procedural rather than technical. Disclosure of assistant settings at the point a device is issued. An opt-out a parent can exercise without filing a records request. Logging a district can audit, and a named person at the district who reads it. Those were cheap to implement a year ago and are cheap now.
School districts, for their part, are not equipped to evaluate a model. A technology office that manages tens of thousands of devices can test whether a login works and whether a website loads. It cannot test whether a generative system will handle a distressed fourteen-year-old well at eleven at night, and no procurement checklist it has been given asks it to.
Whether a jury ever sees these facts depends on motions that will take a year or more. What the filing has already done is move the question of default settings out of a product meeting and into a court docket, where the answer will be written by someone who has never configured a school laptop and has no way to know how many districts took the same path quietly.
