acceptodds
Under review as a conference paper at ICLR 2027

A Testable Theory of Atomic Features

Abstract

We develop and test a theory of language model representations in which there exist "atomic" features. Our main theoretical insight is that in such a model, sparse dictionaries (e.g., SAEs) of increasing size recover an increasing prefix of atomic features most prevalent in the training distribution. This "recovery principle" yields three testable predictions: many features in small SAEs are shared by all larger SAEs, SAEs trained on different distributions share features prevalent in both, and sufficiently large SAEs recover both parent and child features. In contrast to conventional wisdom that SAEs features are unstable and "split" as size increases, we find that these predictions hold on SAEs of sizes ranging from 512 to 131,072 trained on two large embedding models. From a theoretical perspective, our results suggest the promise of a scientific theory of representations based on atomic features. Practically, our results suggest the promise of scaling SAEs.

Primary area:
interpretability and explainable AI
Keywords:
linear representation hypothesis, sparse autoencoders, atomic hypothesis, scientific theory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wvJjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes an atomic theory of language model representations. The authors show that the linear representation hypothesis implies the existence of atomic features. They then explore the implications of this for language model representations and SAEs trained on them. They make and validate 3 predictions based on their theory about the results of training SAEs on different data and with different scales.

Critical strengths

The paper is clear, well written and provides a particularly clean account of the consequences of the linear representation hypothesis that could be useful to the field.

This work also has implications for the promise of scaling SAEs or what would not be true of language model representations if scaling up SAEs does not work.

Critical weaknesses

I think the paper could do a better job acknowledging that the idea of discrete/atomic features encoded in superposition in transformer residual streams has been a common idea in the field and the main motivation for using SAEs and other dictionary leaning approaches. As such atomic features and their connection to the LRH are not new even if they have not been combined into a concrete theory with clean testable predictions before this work.

While this work provides a clear and useful theory, it provides more of a formalization and combination of existing ideas than a set of novel ideas. I nevertheless think this is a useful contribution to the field.

Questions for authors

Beyond my point under weaknesses, I would also like to see a bit more discussion of the point made in paragraph 3 of the introduction. After introducing Theorem 2, what is your view of the plausibility of atomic theory? What observations does your theory predict that, if found false in residual stream representations would undermine the LRH? Are there ways to soften the LRH that would make it more likely to fit with observations in real models?

2 comments
AuthorsOpenReview ↗

New experiments with null model (enabling more falsifiability)

Thank you for the thoughtful review.

We agree that the idea of the idea of discrete features is not new, and that our main contribution is a way to make this precise and testable (which we see as a core step in resolving disagreements and building scientific theory).

Regarding your questions, we implemented an additional experiment that shows a comparison to a null model (Gaussian-distributed embeddings with matching mean and covariance). If the stability measures were the same or worse than in this null model, we would see that as evidence against the hypothesis. In experiments on this null model, we found 0% matches on the wiki vs no_wiki training data setting, suggesting that the amount of stability we found (26%) is substantial evidence in favor of the LRH.

We will add this discussion of falsifiability using a null model prominently after Theorem 2.

If you think this new addition helps address your concerns, we would appreciate it if you considered raising your score. Thanks again.

Thank you for your response.

While I believe this improves the paper, my previous concern about novelty still prevents me from raising to 4/4. I still think this is a useful contribution to the field and recommend acceptance.

Reviewer_gf2NOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper develops a theoretical framework for understanding sparse representations in language models through the hypothesis that representations consist of identifiable “atomic” features by three main theoretical results. Experiment suggests the promise of a scientific theory of representations based on atomic features

Critical strengths
  1. experiments and results look good to me
  2. some idea is interesting. for example, theorem 2 provides a connection between feature prevalence and recoverability under restricted dictionary capacity. This is a valuable perspective on why features may be stable across SAE widths despite previously documented feature splitting and instability
Critical weaknesses

While P1, P2, and P3 follow from the proposed framework under its assumptions, it is not clear that these observations are sufficiently specific to atomic representations. For example, persistent SAE directions across widths could arise from stable, frequently occurring semantic directions even if the underlying representation does not possess a unique atomic decomposition. Similarly, cross-distribution stability could arise because different distributions share substantial semantic or geometric structure.

Questions for authors
  1. Can the authors identify a plausible non-atomic representation model under which P1–P3 would fail? How would the proposed tests distinguish the atomic hypothesis from other models exhibiting stable semantic directions?
  2. How sensitive are the main P1 and P3 results to a consistent sparsity parameter, optimization budget, and auxiliary-loss coefficient across widths?
1 comment
AuthorsOpenReview ↗

New results with non-atomic representation model under which predictions fail

Thank you for the thoughtful review, and are glad that you found the ideas interesting and the paper to give a valuable perspective. We answer your questions below.

  1. Yes, we implemented Reviewer CM6f's suggested Gaussian surrogate model (where we generate from a Gaussian with matched mean and covariance as our original embeddings). For example, we find 0 matches in this model in the wiki vs. no_wiki setting. We also find substantially fewer matches in the P1 synthetic setting (3-29% vs 95-97%) and 0 matches in P2 synthetic setting (vs. 44%). We are running further experiments in this null model (which will take a couple of weeks to finish), and update the paper with this discussion. We agree this is a valuable addition.

  2. Section D.1 shows that results carry over in the fixed sparsity setting. Regarding optimization budget, we find that the number of matches increases with more training. We are running an experiment with variable aux-loss for the size 16K SAEs, and will add those experiments when they are complete. (We did find that for too small aux loss, dead neurons did not go to 0).

Thanks again, we agree that this was an important point, and believe that our new experiments strengthen our paper. If you think we have addressed your main concern (since we have identified a non-atomic model under which the predictions do not hold), we would appreciate it if you could raise your score.

Reviewer_CM6fOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper formalizes the linear representation hypothesis as a postulate in which representations are exact sparse nonnegative combinations of "atomic" feature directions. It proves that this decomposition is rigid and that any sufficiently good sparse dictionary smaller than the true one still recovers the most prevalent atoms, which yields three predictions for SAEs. Features of small SAEs should persist in larger ones, SAEs trained on different data should share features, and sufficiently large SAEs should recover both parent and child features, so that parents lost to feature splitting eventually reappear. Experiments with TopK SAEs of up to 131K latents on two embedding models broadly support all three predictions.

Critical strengths
  1. The theory is clean, non-trivial and, as far as I can tell, correct. The Lean formalization adds confidence to the correctness. The recovery principle extends classical uniqueness results for dictionary learning, which compare dictionaries of equal size, to smaller dictionaries trained on idiosyncratic data, and it offers a principled way to reconcile the instability of SAEs across widths and seeds with the LRH.

  2. The experiments are extensive and carefully executed, with several matching criteria, controls for seed and sparsity, and released code. The prediction of mesoscale feature splitting is particularly interesting (Figs. 3a and 17).

Critical weaknesses
  1. The experiments do not distinguish the atomic theory from alternatives. The guarantees are only qualitative (the constant in Theorem 2 exceeds 10^2900 for K = 32), so the predictions reduce to the statement that many features persist or are shared. No null model is considered under which the predictions would fail, and PCA is a weak comparison. SAEs trained in the same way on Gaussian surrogate data with matched covariance would show how much of the persistence comes from SAE training and the geometry of the embeddings rather than from atomic structure. The paper also does not cite Leask et al. (ICLR 2025, Sparse Autoencoders Do Not Find Canonical Units of Analysis), which studies the same cross-width question and reaches the opposite conclusion.

  2. The data depart from both the assumptions and the paper's own simulations, yet every outcome is described as consistent with the theory. The theory assumes exact sparsity matched by the SAE, but k increases from 32 to 128 with width. In the ground-truth simulations persistence exceeds 90%, whereas on real data it peaks around 70% at 4K, drops to about one third at 64K, and at 32K is close to the agreement between two seeds of the same width (Table 4). The hierarchy results also depend strongly on the model (Nemotron recovers fewer than 20% of taxonomy categories at 131K in Fig. 15, which the averaged Fig. 2c obscures), and the U-shaped curves come from single-seed SAEs. The paper should state which outcomes would have counted against the predictions.

Questions for authors
  1. What levels of persistence and sharing do SAEs trained with the same protocol on Gaussian surrogate embeddings, with matched mean and covariance, exhibit? This is the experiment most likely to change my assessment.
  2. Does the theory make any quantitative prediction, for example a usable bound on C or the decline in persistence beyond 4K, and do the trained decoders approximately satisfy the stability assumption?
2 comments
AuthorsOpenReview ↗

New experiments with Gaussian surrogate data

Thank you for the thoughtful review, and for describing our theory as clean and non-trivial, and the experiments as extensive and carefully executed. We're also glad you found the mesoscale feature splitting particularly interesting.

Regarding your main question, we implemented your suggested experiment:

  • On gemini wiki vs. no_wiki, SAEs trained on Gaussian surrogate embeddings with matched mean and covariance had 0 matches at threshold 0.7 (compared to 26 percent on original embeddings). There were 4/16384 matches at threshold 0.3.
  • In our synthetic setting, we found 0 matches on different distributions for the Gaussian surrogate (compared to 45% original), and between 3-29% on across-width SAEs (compared to 95-97% original).

On your second question, the theory does not make a quantitative prediction. We agree that it would be exciting to extend the theory to give these more precise predictions, and will add that to the discussion.

In summary, our new experiments demonstrate that Gaussian surrogates as a strong null model in which our predictions do not hold. We are currently training a more complete set of experiments for Gaussian surrogates (which will require a couple of more weeks to finish), and will include all of those results.

We very much appreciate your suggested experiment, and believe our implementation makes the paper stronger. If you agree, we would appreciate you raising your score. Thanks again.

I thank the authors for their response and for the added and ongoing experiments which I believe will improve the paper. The absence of quantitative predictions to come out of the theory however limits the paper's contribution. I therefore maintain my current score of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.