Interpretability

Interpretability is the research field that tries to understand what happens inside AI models: which concepts a neural network represents, how it combines them and why it produces a particular answer. Its most ambitious branch, mechanistic interpretability, aims to reverse-engineer a trained model much as an engineer might reverse-engineer a compiled program.

Why models are opaque

Large language models are not written line by line; they are trained. Billions of numerical weights are adjusted until the model’s predictions improve, and the result works without explaining itself. That is the origin of the “black box” complaint and of memes such as the shoggoth, the alien creature behind a friendly mask. A further complication is that individual artificial neurons rarely stand for one clear concept. Models appear to store more features than they have neurons by overlapping them, a phenomenon researchers call superposition.

Tools and findings

Early work used probes to test what a model represents. In a 2022 study, a model trained only on lists of moves in the board game Othello turned out to keep an internal picture of the board – evidence that is often cited against the view that language models are mere stochastic parrots.

A second approach, dictionary learning with sparse autoencoders, breaks a model’s internal activity into more interpretable features. In May 2024, Anthropic reported extracting millions of such features from its model Claude 3 Sonnet, from cities and famous people to bugs in code and concepts related to deception. Amplifying one feature produced “Golden Gate Claude,” a short-lived demo that brought up the Golden Gate Bridge in nearly every answer. In June 2024, OpenAI researchers trained a sparse autoencoder with 16 million features on GPT-4.

A third approach traces circuits, the paths along which a model computes an answer. In March 2025, Anthropic described how its model Claude 3.5 Haiku plans a rhyme before it writes the line that ends in it, works through mental arithmetic along parallel paths and uses shared concepts across languages. A cheaper complement is to read what reasoning models write down before they answer – chain-of-thought monitoring, a window that researchers have called fragile and that the song Mind Reading fears is closing.

In April 2025, Anthropic’s chief executive Dario Amodei wrote that the company aims for interpretability to reliably detect most model problems by 2027, as a step toward an “MRI for AI.”

Limits

The features found so far explain only part of what models compute, and explanations can be incomplete or misleading. The methods are expensive, and capabilities are advancing faster than understanding. In March 2025, Google DeepMind’s interpretability team reported that sparse autoencoders did worse than simple linear probes on a practical safety task and deprioritized the method; critics such as Dan Hendrycks argue that the bottom-up approach may not scale at all. Many researchers therefore see interpretability as one layer of safety among several, alongside red teaming, evaluations and monitoring in deployment, rather than a solution on its own.

Why it matters

Interpretability is a constructive answer to the black box: learn to understand the technology rather than only test it from outside. Even its champions pair it with caution; Amodei wants export controls to buy it time. If it succeeds, audits could look beyond behavior in tests and detect hidden goals or scheming before a model is deployed, and developers could debug models as they debug software. It is also a scientific frontier in its own right: studying an artificial mind may teach us something about minds in general. A society that relies on AI has good reason to invest in being able to look inside it.

Written with Claude, made by Anthropic, whose interpretability research is described above.

Related

Shoggoth · Stochastic Parrots · Scheming · Red Teaming · Alignment · Automated Artist

Sources: Nelson Elhage et al.: Toy Models of Superposition (Transformer Circuits) (September 2022) · Kenneth Li et al.: Emergent World Representations (arXiv) (October 2022) · Adly Templeton et al.: Scaling Monosemanticity (Transformer Circuits) (May 2024) · Anthropic: Golden Gate Claude (May 23, 2024) · Leo Gao et al.: Scaling and evaluating sparse autoencoders (arXiv) (June 2024) · Anthropic: Tracing the thoughts of a large language model (March 27, 2025) · Google DeepMind mechanistic interpretability team: Negative Results for SAEs On Downstream Tasks (AI Alignment Forum) (March 26, 2025) · Dario Amodei: The Urgency of Interpretability (April 2025) · Dan Hendrycks: The Misguided Quest for Mechanistic AI Interpretability (AI Frontiers) (May 15, 2025) · Tomek Korbak et al.: Chain of Thought Monitorability (arXiv) (July 2025)

Leave a Reply

Your email address will not be published. Required fields are marked *