LoRA

LoRA, short for low-rank adaptation, is a technique for fine-tuning large AI models cheaply. Instead of changing all of a model’s billions of weights, it keeps them frozen and trains a small set of additional parameters that adjust the model’s behavior. The result – often megabytes rather than the gigabytes of the full model – can be stored, shared and switched on or off like a plug-in. LoRA is one of the most widely used methods of parameter-efficient fine-tuning and a standard way hobbyists customize open models.

How it works

The method was introduced in 2021 by researchers at Microsoft led by Edward Hu. Its premise is that the change fine-tuning makes to a model’s weight matrices can be approximated by the product of two much smaller matrices – a “low-rank” update. Only those small matrices are trained. For GPT-3, the authors trained about one ten-thousandth as many parameters and needed about a third of the GPU memory, while matching full fine-tuning on the tasks they tested. Because the original model is untouched, one base model can serve many LoRA adapters, each for a different task or customer.

In 2023 researchers at the University of Washington introduced QLoRA, which combines LoRA with quantization of the frozen model. It made it possible to fine-tune a 65-billion-parameter model on a single 48-gigabyte professional graphics card (see GPU), putting customization of large models within reach of individuals and small labs.

Uses

In language models, LoRA is used to teach a model a company’s style, a specialized domain, a language or a task format, and Apple uses LoRA adapters to specialize the on-device model behind Apple Intelligence for different tasks. In image generation it became a phenomenon: users of the open model Stable Diffusion train LoRAs on a few dozen pictures to capture an art style, a character, an object or a person’s face, and share them in large numbers on sites such as Civitai (see Text-to-Image). The same ease raised problems. LoRAs that reproduce a living artist’s style or a real person’s likeness fed disputes over consent and copyright, and some were used to make sexual deepfakes (see Non-Consensual Intimate Imagery). In May 2025, citing payment processors and new laws, Civitai removed models of real people, though many reappeared elsewhere.

Limits

LoRA can learn less than full fine-tuning, especially in demanding domains such as code and math. A 2024 study summarized the trade-off in its title, “LoRA Learns Less and Forgets Less”: adapters change a model’s abilities less, but also damage what it already knew less (see Catastrophic Forgetting). Later work, including a 2025 analysis by Thinking Machines Lab, which sells a LoRA-based fine-tuning service, found that with careful settings LoRA can match full fine-tuning on small and medium datasets and in reinforcement learning. Because adapters are cheap to make, they can also strip safety training from open models: a 2023 study undid the safeguards of Meta’s Llama 2-Chat for under $200 on a single GPU. A related method that needs no training at all is abliteration.

Why it matters

LoRA turned fine-tuning from an industrial process into something hobbyists can do. It is one reason the ecosystem around open weights is so large and inventive – and why questions about consent, likeness and safety can no longer be settled only by the companies that train the base models.

Written with Claude, made by Anthropic, which does not release model weights and currently offers no public fine-tuning of Claude; its one such offering, Claude 3 Haiku on Amazon Bedrock, ended when that model was retired in 2026.

Related

Fine-tuning · Quantization · Open Weights · Model Merging · Text-to-Image · Catastrophic Forgetting · Abliteration

Sources: Edward J. Hu et al.: LoRA: Low-Rank Adaptation of Large Language Models (arXiv) (2021) · Tim Dettmers et al.: QLoRA: Efficient Finetuning of Quantized LLMs (arXiv) (2023) · Dan Biderman et al.: LoRA Learns Less and Forgets Less (arXiv) (2024) · John Schulman and Thinking Machines Lab: LoRA Without Regret (September 29, 2025) · Simon Lermen, Charlie Rogers-Smith, Jeffrey Ladish: LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B (arXiv) (2023) · Apple: Introducing Apple’s On-Device and Server Foundation Models (June 10, 2024) · 404 Media: Civitai Ban of Real People Content Deals Major Blow to the Nonconsensual AI Porn Ecosystem (May 27, 2025) · 404 Media: Hugging Face Is Hosting 5,000 Nonconsensual AI Models of Real People (2025) · Claude Docs: Model deprecations

Leave a Reply

Your email address will not be published. Required fields are marked *