Dark LLM

A dark LLM is a language model sold or shared specifically for criminal use – writing phishing emails, malware or scam scripts – and advertised on hacking forums or Telegram as free of the safeguards that mainstream chatbots have. The term covers both purpose-built tools, often based on open models, and resold access to mainstream models wrapped in jailbreak prompts.

WormGPT and its heirs

The first widely reported example, WormGPT, was sold on the forum HackForums from early 2023 and drew wide attention in July 2023, when the security firm SlashNext showed it writing a convincing business-email-compromise message; its seller claimed it was based on the open model GPT-J. Days later, FraudGPT was advertised on Telegram for similar tasks. After the journalist Brian Krebs identified a developer behind WormGPT, the project shut down in August 2023, its makers citing media attention. In February 2025, Portuguese police arrested a 22-year-old suspected of developing it. The brand lived on: in 2025, researchers at Cato Networks found new “WormGPT” variants built on xAI’s Grok and Mistral’s Mixtral, using system prompts to bypass guardrails and possibly fine-tuning. In November 2025, Palo Alto Networks’ Unit 42 described WormGPT 4, sold since September 2025 for $50 a month or about $220 for lifetime access, and KawaiiGPT, a free tool on GitHub; both generated phishing emails, ransom notes and attack scripts, much of it easy to detect.

How much do they matter?

Security researchers are divided on the real-world impact. On one side, these tools lower the barrier for unskilled criminals: they write fluent phishing messages in any language, generate basic scripts and remove the friction of mainstream refusals. Academic researchers at Ben-Gurion University warned in 2025 that dark LLMs and easily jailbroken mainstream models pose a growing risk. On the other side, analysts at Sophos and Trend Micro have found many offerings to be low-quality wrappers or scams aimed at other criminals, with jailbreak-as-a-service the dominant business model. AI companies’ threat reports describe criminals and a suspected Chinese state-sponsored group misusing mainstream models, sometimes by jailbreaking them, as Anthropic reported in 2025, while OpenAI reports mostly incremental gains for attackers who bolt AI onto old methods.

Countermeasures

Defenses work on several levels: model developers monitor for misuse and ban accounts, security firms track dark LLM offerings on forums, email filters and identity checks target the phishing they produce, and law enforcement pursues sellers, as the Portuguese arrest showed. Because uncensored models can be created from open weights by anyone, part of the defense has to happen downstream – at the points where crimes actually occur.

Why it matters

Dark LLMs mark the far end of the debate about unfiltered AI: tools built not for adult freedom but for crime. Advocates of open models argue that the useful line is between lawful and unlawful use, not between filtered and unfiltered models; safety researchers reply that safeguards in models still raise attackers’ costs (see uplift). Both sides agree that security engineering and old-fashioned policing remain the backbone of the response.

Written with Claude, made by Anthropic, which publishes threat-intelligence reports on attempts to misuse Claude, one of which is cited here, and has reported Claude being jailbroken by criminals and a suspected state-sponsored group.

Related

Uncensored Model · Jailbreak · Uplift · Red Teaming · Prompt Injection · Uncensored AI overview

Sources: Daniel Kelley: WormGPT – The Generative AI Tool Cybercriminals Are Using to Launch Business Email Compromise Attacks (SlashNext) (July 13, 2023) · Rakesh Krishnan: FraudGPT: The Villain Avatar of ChatGPT (Netenrich) (July 25, 2023) · Brian Krebs: Meet the Brains Behind the Malware-Friendly AI Chat Service ‘WormGPT’ (Krebs on Security) (August 8, 2023) · Sophos X-Ops: Cybercriminals can’t agree on GPTs (November 2023) · Notícias ao Minuto: Young man who created a model for cybercrime banned from hacker forums (February 2025) · Vitaly Simonovich: Cato CTRL Threat Research: WormGPT Variants Powered by Grok and Mixtral (Cato Networks) (June 17, 2025) · Unit 42: The Dual-Use Dilemma of AI: Malicious LLMs (Palo Alto Networks) (November 25, 2025) · Michael Fire et al.: Dark LLMs: The Growing Threat of Unaligned AI Models (arXiv) (May 15, 2025) · Trend Micro: The State of Criminal AI · Anthropic: Detecting and countering misuse of AI: August 2025 (August 27, 2025) · Anthropic: Disrupting the first reported AI-orchestrated cyber espionage campaign (November 13, 2025) · OpenAI: Disrupting malicious uses of AI: October 2025 (October 2025)

Leave a Reply

Your email address will not be published. Required fields are marked *