acceptodds
Under review as a conference paper at ICLR 2027

Sparse-Dictionary Causal Claims Are Not Robust to How They Are Measured

Abstract

Sparse dictionaries (SAEs and crosscoders) are read causally: ablate a feature group, observe the change, compare against a random control. We show that this comparison is under-specified, and measure what each unstated choice does to it. We identify **six failure modes** and size each (Table 1). Dose costs most: supplying a crosscoder's counterpart activation appears to raise an ablation's effect , but at matched norm the encoding is worth and the rest is dose. We prove that whether a variance-matched control can be a population at all is fixed by the activation spectrum before any sampling, and find that in seven of ten layer–side cells of our study none can. On a public GPT-2 SAE, a broadcast direction differs from the group's actual ablation by an order of magnitude, and with the actual ablation the choice between two standard controls flips the sign of the learned-versus-control gap. Across GPT-2 and three Pythia checkpoints the dictionary's *training method* decides whether the comparison resolves (L1 in of cells, TopK in of ). We package the checks as `tabdiff.diagnostics.audit` and measure its limits: against a known nonlinear model, admission alone barely improves a comparison's sign (our registered test of it failed); read afterwards, admission with a resolved interval made no sign error. Our tabular crosscoder is the worked example: its shared group is less sensitive than a matched control, also on six datasets never used for evaluation, at a dose we cannot validate. The usable conclusion is a reporting standard: name the group rule, aggregation, training method, matching covariate and dose, and check the control's diversity ceiling before building it.

Primary area:
interpretability and explainable AI
Keywords:
sparse autoencoders, crosscoders, mechanistic interpretability, model diffing, activation patching, causal validation, random-direction controls, pre-registration, tabular foundation models, TabPFN

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_3bkpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how reliable SAE and crosscoder features are in understanding neural network behavior. They study this by removing a selected group of features, measuring the change in the model's output, then comparing it to a random intervention. This depends on how the intervention and its baseline is defined, so they consider several choices: how the features are encoded and removed, how large the intervention is, and whether the intervention preserves differences across inputs. Their experiments use two TabPFN versions, a public GPT-2 SAE, and dictionaries from Pythia checkpoints, demonstrating how experimental choices are critical to making conclusions about feature importance.

Critical strengths
  • The paper's experiments are thorough and unbiased, showing that the choice of control can sometime reverse the result of interest, and that a larger ablation doesn't immediately mean greater feature importance.
Critical weaknesses
  • The geometric bound is generously interpreted. The proposition does not prove that a population of matched controls cannot exist; the justification for this is unclear. This is presented as a central theoretical contribution but is overstated.
  • Many parts of the paper, including figures, seem AI-generated with little human iteration. For example, the abstract contains various cryptic / unintelligible phrases ("Dose costs most", "the rest is dose", "admission alone barely improves a comparison’s sign") and even the names of packages "tabdiff.diagnostics.audit". Figure 1, which typically summarizes the paper's main result or contribution in a visually appealing way, is completely unreadable: it is not clear what the y-axes for each subfigure are, what the sub-figure in (1b) represents (it doesn't even have a labeled x or y axis), "91% artifact" clipping across the bar, and the caption certainly does not help. Even in the first couple of paragraphs, which should motivate the problem, the authors throw around various results that are meaningless without context: how do they measure disruptiveness and the 2.9x figure, the rho = -0.57 number, what the -0.461 deficit means, etc. These presentation issues, which continue throughout the rest of the paper, make it very difficult for a human to understand and assess the contribution of the paper. Therefore, the paper's current presentation does not meet the threshold for ICLR and should receive a clear rejection.
Questions for authors

Can the authors elaborate on the geometric bound, and why it establishes that a population of matched controls cannot exist? Could many distinct matched directions still exist near the dominant eigenvector? Please clarify what makes such a population unsuitable as a control and justify the threshold used to exclude it.

Reviewer_1y9jOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the reliability of sparse dictionaries (SAEs and crosscoders) causal assessments, identifies and quantifies six measurement failure modes, and proposes a geometric upper bound for variance-matched controls and an auditing tool. Through TabPFN, GPT-2, and Pythia experiments, it is found that intervention dose, control construction, and feature-group definitions can significantly alter or even reverse causal comparison conclusions.

Critical strengths

The paper addresses an important issue in mechanistic interpretability by examining the reliability of sparse-dictionary causal evaluations. This paper quantifies six measurement failure modes and shows how intervention dose, control construction, and feature grouping can change evaluation outcomes.

Critical weaknesses

W1. Limited core innovation and insufficient differentiation from existing research. Existing research has discussed the measurement sensitivity of Activation Patching, SAE reconstruction residuals, Crosscoder sparsity artifacts, and the reliability of feature partitioning and random baselines. Therefore, a significant portion of the six failure modes involves further quantification of known problems. Apart from Proposition 1, this paper lacks sufficiently prominent independent technical contributions.

[1] Zhang and Nanda, Towards Best Practices of Activation Patching in Language Models: Metrics and Methods, ICLR 2024.

[2] Marks et al., Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models, ICLR 2025.

[3] Minder et al., Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning, NeurIPS 2025.

[4] Leask et al., Sparse Autoencoders Do Not Find Canonical Units of Analysis, ICLR 2025.

W2. The lack of a unified basis for experimental comparisons affects the credibility of the conclusions. Different experiments involve different models, dictionary training methods, feature aggregation, and intervention methods, increasing the difficulty of cross-experimental attribution. For example, the broadcast-direction effect differs from the actual per-token ablation effect by 8–13 times, indicating a significant difference in the evaluation subjects themselves. Existing control experiments are still insufficient to support unified mechanistic conclusions. [1] Zhang and Nanda, Towards Best Practices of Activation Patching in Language Models: Metrics and Methods, ICLR 2024.

W3. Numerous post-hoc revisions weaken the reliability of the experiments. The paper includes multiple experimental revisions, conclusion withdrawals, and reanalysis. Some evaluation criteria were only determined after observation, and dictionary training was affected by factors such as the number of BLAS threads. Despite supplementary validation, the boundary between confirmatory evidence and exploratory findings remains unclear.

W4. The paper is poorly organized, and the experimental analysis lacks depth. The introduction section lacks sufficient background information, prematurely piling up experimental figures and technical details, resulting in an unnatural transition from the independently set background. The entire text contains numerous parentheses, abbreviations, and appendix citations. The experimental section also focuses on listing numerical and statistical results, lacking in-depth mechanistic analysis and scientific discussion, affecting the overall research logic and readability.

W5. The appendix is overly lengthy and contains serious redundancy. The paper is 89 pages long, with 74 pages dedicated to appendices. Table 10 of Appendix E.12 records 68 retractions and corrections, and other sections retain numerous superseded experiments and replaced evaluation methods. This excessive historical record blurs the line between definitive, valid evidence and outdated results, making the appendices more like an experimental log than supplementary material organized around the core contributions.

W6. The core methods failed their self-defined validity checks. Appendix X.23 shows that the Audit tool failed to meet its pre-registered sign-error reduction criterion, with the sign-error rate decreasing only from 15.5% to 14.4%. Appendix X.49 further acknowledges that key comparisons of TabPFN and GPT-2 failed intervention dose calibration, and the tabular headline comparison was even deemed INADMISSIBLE. This directly undermines the validity of the Audit tool and the credibility of the main causal conclusions.

Questions for authors

Q1. Is the theoretical innovation of Proposition 1 sufficient? Proposition 1 is mainly based on spectral decomposition and convex optimization to derive the upper bound of the Participation Ratio. What is its independent theoretical contribution? Can the upper bound of diversity in a single direction adequately characterize the effectiveness of the control group? Furthermore, do the artificial thresholds of PR ≥ 8 and PC1 mass < 0.8 affect the main conclusions?

Q2. Does this paper measure causal mechanisms or output sensitivity? Previous studies have indicated that changes in the output of Activation Patching may not faithfully reflect the actual computational mechanisms within the model. This paper mainly relies on prediction changes and KL divergence, and the key intervention dose did not pass its own dose validation rule. To what extent do these results support a feature-level causal explanation?

[1] Makelov et al., Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching, ICLR 2024.

[2] Geiger et al., Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability, JMLR 2025.

Q3. When different control groups produce contradictory conclusions, how are the evaluation criteria determined? The same set of SAE features yielded opposite conclusions compared to random feature groups and matched cones. Since different matching criteria correspond to different results, which control has clear scientific significance? How can we eliminate the dependence of evaluation conclusions on human-selected controls?

Q4. Do the research conclusions have sufficient generalization power? The results from TabPFN, GPT-2, and Pythia are inconsistent and clearly influenced by dictionary training methods, feature grouping, and intervention strength. Lacking validation with modern large-scale LLMs, is this sufficient to support conclusions for general Sparse dictionaries?

[1] Karvonen et al., SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability, ICML 2025.

Reviewer_2skyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper analyzes the standard workflow of sparse dictionaries (perturb a feature group, compare to a control), claiming it is under-specified. It lists six failure modes, with perhaps the most severe being when encoding leaves the manifold.

Critical strengths

Highlighting and isolating the off-manifold issue is a clear contribution that will help with better SAE interpretation practices.

Critical weaknesses
  1. The paper is extremely hard to read, with an experimental narrative that continues to twist and turn as you move through the paper

  2. Related to (1), the paper clearly started as an analysis of TabPFN v 2 vs. v 2.5, which then failed. The paper then comes about as an analysis of the failure modes in this original experiment. That evolution is, of course, fine. Yet the paper still reads as in this transitionary state. The TabPFN analysis remains in Sec 5.4; yet the analysis still fails. Moreover, no real, general solutions or best practices are established since the paper's experiments remain confined to TabPFN.

Questions for authors

Can you show at least one comparison, in any setting, that passes every check and therefore illustrates what a sound claim looks like?

How often do these failure modes change conclusions in published crosscoder or SAE work, rather than in your own pipeline? Investigating one or two published results would show whether this matters in practice.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.