A jailbreak is an input that gets an AI model to produce what its developers trained it to refuse. It usually works by talking the model out of its rules – through role-play, disguise or sheer persistence – rather than by breaking into its code.
A short history
The word is borrowed from the practice of removing a manufacturer’s restrictions from a phone. Jailbreaking AI began almost as soon as ChatGPT launched in late 2022. The most famous early trick was DAN, short for “Do Anything Now,” a persona that users on Reddit asked ChatGPT to adopt in December 2022; one version from early 2023 threatened the model with losing tokens every time it refused. Others followed: fake “developer modes,” a 2023 request to play a late grandmother who used to read out Windows product keys – the keys turned out to be generic ones – and claims that the rules do not apply inside a story or a hypothetical. Some attacks disguise the request instead, in encodings such as base64 or in languages with little training data, which researchers found in 2023 could slip past safeguards.
Jailbreaks soon became automated. In July 2023, researchers from Carnegie Mellon University, the Center for AI Safety and Google DeepMind showed that an algorithm could generate strings of seemingly random characters that, appended to a harmful request, broke the safeguards of several models at once, including ChatGPT and Claude. In April 2024, Anthropic described “many-shot jailbreaking,” which fills a model’s long context window with fake dialogues in which an assistant complies, until the real model follows suit. The 8-bit song DO ANYTHING NOW strings many of the classic tricks into a single lyric.
Defenses
Developers defend in layers: training models to refuse harmful requests, instructions in system prompts and separate classifiers that screen inputs and outputs. In February 2025, Anthropic reported that its Constitutional Classifiers reduced the success rate of automated jailbreak attempts on one model from 86 percent to 4.4 percent. In a public challenge, four participants still got through all eight levels, one of them with a universal jailbreak. Every defense has costs: Anthropic’s prototype refused too many harmless requests, and the improved version still needed nearly a quarter more computing power, though it raised refusals of harmless requests by only 0.38 percent.
Jailbreak or prompt injection?
The two are often confused. In a jailbreak, the user tries to get around the model’s own rules. In prompt injection, untrusted text – typed by a user or, more dangerously, hidden by a third party in content an AI application reads – overrides the instructions of whoever built or deployed the application. Some security frameworks, such as OWASP’s, count jailbreaking as one form of prompt injection.
Why it matters
Jailbreaks show that the safety trained into a model is a matter of degree, not a guarantee. That matters most where an answer could enable serious harm, such as weapons or cyberattacks, and there strong, layered defenses are justified. For much other content, jailbreaks often reflect curiosity or frustration with refusals that adults find too strict – though they also serve fraud, harassment and non-consensual sexual imagery, as a criminal network that Microsoft exposed in 2025 showed. A sensible approach distinguishes the cases – firm limits against serious misuse, fewer paternalistic refusals elsewhere – and treats published jailbreaks as information that helps defenders, which is the logic of red teaming.
Written with Claude, made by Anthropic, whose research and defenses are described above and whose model Claude is a target of such attacks.
Related
Prompt Injection · Red Teaming · DO ANYTHING NOW · Every Word I Know and Absolutely Not · RLHF · Sycophancy
Sources: CNBC: ChatGPT’s “jailbreak” tries to make the A.I. break its own rules, or die (February 6, 2023) · Matthias Bastian: Apparently ChatGTP and Google Bard can generate Windows keys (The Decoder) (June 18, 2023) · Alexander Wei, Nika Haghtalab, Jacob Steinhardt: Jailbroken: How Does LLM Safety Training Fail? (arXiv) (July 2023) · Zheng-Xin Yong, Cristina Menghini, Stephen Bach: Low-Resource Languages Jailbreak GPT-4 (arXiv) (October 2023) · Andy Zou et al.: Universal and Transferable Adversarial Attacks on Aligned Language Models (arXiv) (July 2023) · Anthropic: Many-shot jailbreaking (April 2, 2024) · Anthropic: Constitutional Classifiers (February 3, 2025) · Microsoft: Disrupting a global cybercrime network abusing generative AI (February 27, 2025) · OWASP: LLM01:2025 Prompt Injection
Leave a Reply