acceptodds
Withdrawn from ICLR 2027

What Pooled Explained Variance Can Hide: Evaluating Crosscoder Fidelity in Vision Encoders

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Crosscoders, a variant of sparse autoencoders, were initially developed to compare representations across models and to analyze features across layers. However, their use for comparing blocks within a single vision encoder remains underexplored. We ask whether raw explained variance reflects typical-token reconstruction and predicts VLM behavior after a reconstruction is placed back into the encoder. We study this limitation using crosscoders, which reconstruct representations across layers and models. Our analysis shows that pooled explained variance weights each token by its squared distance from the activation mean, allowing a small high-norm population to dominate the score and obscure weaker reconstruction elsewhere. We evaluate crosscoders across five vision encoders using masked, token-balanced, and norm-stratified fidelity, separately accounting for population recentering. Controlled sparsity sweeps reveal that typical-token reconstruction deteriorates faster than raw explained variance as crosscoders become sparser. We then test these reconstructions inside four VLMs on the SpatiaLab spatial-reasoning benchmark. In Qwen2.5-VL, reconstructions explain over 99% of pooled activation variance yet reduce accuracy by up to 6.9 percentage points. Additional encoders and a separate stress test support the broader discrepancy between pooled fidelity and preserved behavior. Interventions identify model-specific multilayer perceptron contributions necessary for the measured high-norm contrasts, which later blocks often inherit, while Gemma provides a control without a persistent high-norm tail. Together, these findings establish the need to evaluate reconstruction across token populations and test its behavioral consequences. We recommend reporting raw explained variance alongside token-balanced and norm-stratified fidelity, followed by block replacement to assess whether reconstruction errors affect the behavior being studied.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic interpretability, crosscoders, sparse autoencoders, explained variance, vision–language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_u69tOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper questions the common practice of evaluating sparse dictionary models, such as sparse autoencoders and crosscoders, with pooled explained variance (EV). The authors show that pooled EV can be rewritten as a weighted average of per-token reconstruction scores, where each token's weight is proportional to its squared distance from the activation mean. As a result, a small population of high-norm tokens, which several vision encoders develop, can dominate the score and hide poor reconstruction of the remaining tokens. To expose this, the authors introduce masked, token-balanced, and norm-decile EV and use them to evaluate crosscoders trained on five VLM vision encoders. They further show that reconstructions with near-perfect pooled EV can still degrade SpatiaLab accuracy when substituted back into the encoder (by up to 6.9 percentage points in Qwen2.5-VL), and they analyze the MLP contributions that produce the high-norm tokens.

Critical strengths

S1. The paper pinpoints a practically important issue in how sparse representations are currently evaluated. Figures 1 and 2 convincingly show that the issue arises in practice: in Qwen and LLaVA, the highest-norm decile accounts for over 96% of the variance at late blocks, and raw EV diverges sharply from the masked scores once these tokens emerge.

S2. The experiments cover six vision encoders from different VLM families, which seems to be sufficiently thorough.

Critical weaknesses

W1. Presentation and clarity.

(a) Abstract framing. The sentence "However, their use for comparing blocks within a single vision encoder remains underexplored" does not reflect the paper's main contribution. As I understand it, the paper examines the EV-based evaluation scheme rather than extending crosscoders to blocks within a single vision encoder. The sentence also does not connect to the next one, and "this limitation" in the following sentence has no clear referent. The abstract would benefit from a careful revision so that the motivation leads directly to the research question.

(b) Introduction of EV. The introduction turns to pooled EV without explaining why EV is relevant. I assume this is because sparse representations are commonly evaluated with EV, but this should be stated explicitly, together with what "pooled" means, before the weighting issue is raised.

(c) Description of the crosscoder. Section 2.1 is too brief to understand how the crosscoder is trained and how it is used in the experiments. In particular, the training objective (currently Eq. 5 in the appendix) and the fact that block replacement uses the full clean activation stack, including later blocks, should be described in the main text.

(d) Reliance on the appendix. Sections 4.4 and 5.2 draw conclusions from results that appear only in the appendix (Appendix A.4 and Table A18, respectively). If these results are discussed in the main text, they should also be presented there.

W2. Unclear recommendation among the proposed metrics. The paper introduces three alternative metrics but does not clarify which one future work should adopt. Moreover, the metrics can disagree with each other: for Qwen in Figure 2, token-balanced EV remains high while masked EV is low. The authors explain that the metrics answer different questions, but practitioners still need guidance on which metric to rely on when they conflict, ideally supported by evidence of which metric best predicts behavioral degradation.

W3. Connection to the main claim and novelty of Section 5. Since the main claim is that pooled EV can misrepresent reconstruction quality, the role of the mechanism analysis in Section 5 is unclear. The authors themselves note that these interventions measure norm contrast and do not test their effect on crosscoder fidelity. In addition, the role of MLPs in producing high-norm tokens has already been studied by Jiang et al. (2025) [1], who localize it to a sparse set of neurons. The additional contribution of this analysis is therefore unclear.

[1] : Nick Jiang, Amil Dravid, Alexei A. Efros, and Yossi Gandelsman. Vision transformers don’t need trained registers.

Questions for authors

Q1. Have the authors tried removing the high-norm tokens with the test-time registers proposed by Jiang et al. (2025) [1] and then re-evaluating the crosscoders? It would be informative to see (i) whether the discrepancy between raw EV and the proposed metrics shrinks, and (ii) whether, at the same sparsity, typical-token fidelity and downstream VLM accuracy improve. Such an experiment would also connect the mechanism analysis in Section 5 to the paper's main claim.

[1] : Nick Jiang, Amil Dravid, Alexei A. Efros, and Yossi Gandelsman. Vision transformers don’t need trained registers.

Reviewer_W1pFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Pooled explained variance (EV) for corsscoders on vision encoders is dominated by a small set of high norm tokens, because each token is weighted by its squared distance from the activation mean. Reporting masked, token-balanced, and norm decile EV alongside the raw EV, papers shows via sparsity sweeps on four VLM encoders.

Critical strengths
  • Decomposition in eq.1,2 and the separation of weighting from recentering (eq4) is clean and does makes a point that people training ision SAEs, crosscoders routinely get wrong in practice.
  • Behavioral test is done properly.
Critical weaknesses
  • The core "metric" contribution is thin for a main track paper. Eq. 1 is an algebraic identity, and the paper itself credits Gao et al. (2024) with the high-norm token observation. Per population reconstruction reporting is also not new in vision, as Ro et al. (CVPR 2026 beyond semantics) already report FVU separately for outliers vs all tokens. And ViSAEBench (ICML 2026 MI workshop) already reports that reconstruction error and downstream preservation dissociate for vision SAEs. Latter two should be cited, and the paper does not show its proposed socres add anything beyond that kind of per-population reporting.
  • Paper never shows that typical-token EV is a better predictor of behavior that raw EV, which is what the recommendation needs. By the authors' own admission all metrics give the same disctionary ordering, so the evidence reduces to descriptive correlations over 18 non-independent conditions in the appendix. The headline "raw EV > .995 yet acc drops" is really a satement about one encoder (Qwen) plus LLaVA; in Phi and InternVL raw EV falls alongside accuracy.
  • The dictionaries look undertrained (600 images, one seed, typical-token EV 0.2 - 0.5), and they’re evaluated out of distribution (CV-Bench -> SpatiaLab). The sparsity-sweep and behavioral results may reflect that rather than anything intrinsic to sparse dictionaries.
  • Section 5 is largely disconnected: MLP origin of high-norm tokens is already known, the interventions don’t touch crosscoder fidelity, and the Qwen block-16 result is left unexplained.
Questions for authors
  • Does typical-token EV predict SpatiaLab accuracy beyond raw EV (e.g., partial correlation across the 23 cells)? If not, what shows the new scores improve behavioral prediction rather than just lowering the number?
  • How much of the sparse-setting behavioral loss is distribution shift?
  • Why does Qwen block-16 MLP replacement suppress contrast locally but increase it at block 31, and what does “necessary” mean given that?
Reviewer_CaeEOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work explores the use of crosscoders for comparison blocks within a single vision encoder. Crosscoders are evaluated with five vision encoders using masked, token-balanced, and norm-stratified fidelity, accounting for population recentering. Results show the need to evaluate reconstruction across token populations and test its behaviroal consequences.

Critical strengths

The work focuses on separate pooled explained variance token weighting from population recentering and identity model-specific MLP contributions necessary for the measured norm contrasts. Sparsity sweeps show that typical-token fidelity deteriorates faster than raw EV.

Critical weaknesses
  1. The logic of the overall manuscript is very confusing. It seems they suddenly go into an important question about a specific model. I think it's necessary to mention the background and challenges clearly.
  2. The evaluation in the manuscript is empirical and specific. There are no carefully desgins of the overall evaluation, but just use several model for demonstration.
Questions for authors

Is the work totally from AI?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.