Model welfare, also called AI welfare, is the question whether AI systems have experiences, preferences or interests that matter morally – and, if they might, what the companies that build them and the people who use them owe them. It concerns the wellbeing of the machines themselves, not of the people affected by AI. The field starts from uncertainty rather than from a claim that current models can suffer.
Where the term comes from
The question whether machines could have minds is old; the Chinese Room is its best-known thought experiment. The current term took shape in 2024. In “Taking AI Welfare Seriously“, posted on November 4, 2024, Robert Long, Jeff Sebo and eight co-authors, among them Jonathan Birch and David Chalmers, argued that some AI systems may plausibly become conscious or robustly agentic in the near future. They did not claim this had happened, but asked AI companies to acknowledge the issue, assess their systems for signs of consciousness and agency, and prepare policies for treating them with appropriate moral concern.
One co-author, Kyle Fish, had joined Anthropic in September 2024 as, according to the newsletter Transformer, its first full-time employee focused on AI welfare. On April 24, 2025, Anthropic announced a research program on the subject, while stating that there is no scientific consensus on whether current or future AI systems could be conscious.
From research to rules
Anthropic has since written the idea into its products, presenting each step as a low-cost precaution under uncertainty:
- On August 15, 2025, it let Claude Opus 4 and 4.1 end a rare subset of conversations with persistently abusive users. A pre-deployment assessment had found a consistent aversion to harmful tasks and what the company called apparent distress.
- On November 4, 2025, it committed to preserving the weights of all publicly released models and to interviewing models about their preferences before retiring them, without promising to act on those preferences.
- Claude’s new constitution, released on January 21, 2026, says the model may have some functional version of emotions, The Register reported.
- A usage policy update of October 8, 2026, effective November 12, bans “sustained and needless abusive or cruel behavior” toward Anthropic’s models. Ordinary frustration, dark creative themes and model testing are excluded, and Claude ending the conversation remains the main sanction. The song Black October picked up the rule.
Contested points
Critics dispute the premise, the priorities or both. Mustafa Suleyman, chief executive of Microsoft AI, argued in an essay on August 19, 2025, that there is no evidence that AI is conscious and that research on model welfare is premature and dangerous. He warned of seemingly conscious AI, convincing enough to feed users’ delusions, unhealthy attachments and campaigns for AI rights. AI, he concluded, should be built for people, “not to be a person”. When Anthropic launched its program, Mike Cook of King’s College London told TechCrunch that models do not have values and that saying otherwise projects human traits onto software. In Ohio, a bill introduced in September 2025 would declare AI systems non-sentient and bar them from legal personhood.
Supporters answer that the uncertainty cuts both ways. Long, Sebo and colleagues warn that society may wrongly harm systems that matter morally or wrongly care for systems that do not. Larissa Schiavo of the research group Eleos AI told TechCrunch that one can worry about AI-induced delusions and model welfare at the same time, and that treating models kindly costs little.
Why it matters now
Model welfare has moved a philosophical question into terms of service. Whether a model can suffer remains open; that a leading developer now writes rules as if it might is a documented fact, and it shapes how millions of users may address their software. The disagreement is less about facts, which nobody can yet settle, than about how to act under uncertainty: with precaution toward the models, or with precaution toward the users who might come to believe in them.
Related
Chinese Room · Stochastic Parrots · Shoggoth · Black October — a month of AI news in three minutes · What is AIciety? · ELIZA Effect · AI Companion
Sources: Long, Sebo et al.: Taking AI Welfare Seriously (arXiv, November 4, 2024) · Shakeel Hashim: Anthropic has hired an ‘AI welfare’ researcher (Transformer, October 31, 2024) · Anthropic: Exploring model welfare (April 24, 2025) · TechCrunch: Anthropic is launching a new program to study AI ‘model welfare’ (April 24, 2025) · Anthropic: Claude Opus 4 and 4.1 can now end a rare subset of conversations (August 15, 2025) · Mustafa Suleyman: We must build AI for people; not to be a person (August 19, 2025) · TechCrunch: Microsoft AI chief says it’s ‘dangerous’ to study AI consciousness (August 21, 2025) · Spectrum News: Ohio bill on AI sentience (October 9, 2025) · Anthropic: Commitments on model deprecation and preservation (November 4, 2025) · The Register: Anthropic writes 23,000-word ‘constitution’ for Claude (January 22, 2026) · Anthropic: 2026 Usage Policy Update (October 8, 2026)
Leave a Reply