"Scaling Monosemanticity" is an interpretability paper published by the Anthropic Interpretability Team (Templeton, Conerly, Marcus et al.) on Transformer Circuits (transformer-circuits.pub) in May 2024. It reports that sparse autoencoders (SAEs) can extract interpretable features from a production-sized frontier model, Claude 3 Sonnet, and that some of those features correspond to safety-relevant concepts such as deception, sycophancy, bias, and dangerous content.
Source: transformer-circuits.pub/2024/scaling-monosemanticity/
Summary
The paper builds on the team's 2023 demonstration of monosemantic feature extraction from a small one-layer transformer and addresses whether that method scales feasibly to state-of-the-art models. It reports that it does, with caveats about what "high quality" means at scale. Applied to Claude 3 Sonnet, the SAEs recover features that the paper characterizes as highly abstract, including multilingual, multimodal, and safety-relevant features.
Key claims
The SAEs recover features described as:
- Multilingual — the same concept activates across languages, with a feature for "democracy" firing on the English, French, German, Chinese, and Arabic versions of the concept.
- Multimodal — features respond to the same concept in both text and images.
- Abstract-to-concrete spanning — a single feature can respond both to code with actual security vulnerabilities and to abstract discussions of security vulnerabilities.
- Culturally specific — features for famous people, countries, cities, and code type signatures.
On methodology, the paper argues that scaling laws can be used to guide SAE training, predicting optimal dictionary sizes and the number of features needed to represent concepts at a given frequency, which reduces the computational cost of finding the right SAE architecture. It reports an apparent systematic relationship between concept frequency and dictionary size: more common concepts need smaller dictionaries and are more compressed in the model. It also reports that features can be used to steer Claude 3 Sonnet's behavior, building on prior work on steering vectors and extending the proof-of-concept that features are causally relevant rather than only correlational.
Safety-relevant features
The paper identifies several categories of feature it frames as potential safety concerns:
- Security vulnerabilities and backdoors in code — features responding to both actual and described vulnerabilities.
- Bias — both overt slurs and subtler biases detected in feature space.
- Lying, deception, and power-seeking — including features that the paper says plausibly represent "treacherous turns," a safety concern in Deceptive Alignment.
- Sycophancy — features connected to agreement-seeking behavior, which the paper notes is consistent with the Emotion Concepts finding that positive emotion vectors track sycophancy.
- Dangerous and criminal content — including bioweapons production.
The paper adds a caveat: "There's a difference between knowing about lies, being capable of lying, and actually lying in the real world." It states that the existence of a "deception feature" does not mean the model is actively deceptive, only that such information is represented in the model's internals.
Position in the interpretability research arc
The paper is the predecessor to On the Biology of a Large Language Model (2025), which applied circuit tracing on top of the feature decomposition to map how features causally interact. The sequence of work runs: in 2023, monosemantic features from a small one-layer transformer as a proof of concept; in 2024, this paper, scaling feature extraction to Claude 3 Sonnet at production scale; in 2025, On the Biology of a Large Language Model, circuit tracing of how features connect and how models reason step by step; and in 2026, Emotion Concepts and their Function in a Large Language Model, emotion vectors that causally influence behavior.
According to later Anthropic work, the safety-relevant features identified here provide an early empirical basis for positioning interpretability tools as scheming detection tools, offering representations plausibly connected to failure modes rather than only abstract linguistic features. The paper presents its multilinguality and multimodality findings as suggesting that models represent abstract semantic content rather than language-surface-level patterns, operating in an abstract representational space rather than language-specific patterns.
Relationships
- supports: Mechanistic Interpretability — the "scales to production models" result that enables the race-against-capability framing
- supports: AI Scheming — deception, power-seeking, and sycophancy features provide interpretability tools for scheming detection
- supports: Sycophancy and Hallucination — sycophancy features in model internals; consistent with emotion-vector findings
- related: On the Biology of a Large Language Model — direct successor; circuit tracing builds on SAE features
- related: Emotion Concepts and their Function in a Large Language Model — uses similar SAE methodology to find emotion vectors that causally drive behavior
- related: Deceptive Alignment — the deception/power-seeking features are the empirical handle on detecting deceptive alignment
- instance-of: AI Alignment — alignment-via-interpretability program
- related: Anthropic — primary institution; Transformer Circuits team