- Rating
- 2(−1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1(−1)
- Soundness
- 2
Read review
Paper summary
The paper evaluates multimodal sparse autoencoders through preservation of compositional decisions rather than reconstruction alone. It identifies pair-conditioned retrieval leakage and ambiguous product-TopK selection, proposes a shared SAE with modality-local inference, and decomposes reconstructed decision margins into feature contributions. Experiments cover three VLM families, sparsity controls, and held-out feature interventions.
Critical strengths
- The evaluation critique is concrete and consequential: random dictionaries can achieve near-perfect retrieval when pair identity enters support construction. The distinction between reconstruction quality and decision preservation is also supported by explicit measurements rather than examples alone.
- The experimental controls are unusually extensive. Modality-local inference, matched-support comparisons, disjoint circuit discovery and evaluation, and energy-matched random interventions address several obvious alternative explanations. The negative result on symbolic relation binding is informative.
Critical weaknesses
- The advantage of the proposed training objective is less clear than the value of the evaluation framework. Appendix H.7 reports that ordinary TopK and reconstruction-only variants preserve decisions better, and H.8 does not isolate Group-Hoyer against ordinary Hoyer. The claimed benefit is therefore a fidelity--shared-feature trade-off, not better preservation alone. A common comparison including behavioral fidelity and shared-concept coverage is needed to establish that trade-off. The theoretical results are useful diagnostics, but mostly elementary identities and bounds.
- The causal evidence concerns SAE-reconstructed decisions, as Section 4.7 correctly states. It does not directly establish the backbone's internal compositional mechanism. Cross-seed evidence is also limited to rank recovery: Section 5.5 reports 31% top-32 recovery with decoder cosine 0.22, while causal transfer remains untested. This supports a useful intervention basis more strongly than a stable mechanistic decomposition.
Questions for authors
- Under a common support budget and semantic evaluator, does the full objective improve the joint trade-off between decision preservation and shared-concept coverage over ordinary TopK and the simpler cross-decoding variants?
- Do circuits matched across seeds retain their held-out intervention effects? If this has not been tested, can the reproducibility claim be restricted explicitly to rank recovery?
Post-rebuttal update: I read the revised PDF and the authors' replies. Please see my comment below for details on the new experiments. Fixing the base-image split leak is a good step, and the cross-seed transfer results on SugarCrepe help. Still, the new ablations show that Group-Hoyer adds very little over plain L1 or Group-L2,1, and the coverage gain over JumpReLU disappears on SigLIP. The paper also needed a major rewrite. I am lowering my score from 3 to 2.
2 comments
Review
Thank you for the detailed response and the new experiments. The rerun using the image-disjoint split answers my question about cross-seed transfer on SugarCrepe. Activation exchange transfers to all four target seeds under both matching rules. However, the ablation evidence is relatively weak, and the ARO results depend heavily on how the circuit is selected and matched.
My main concern remains the complete training objective. Table 2 shows a clear trade-off on SigLIP2: the proposed model finds more strict shared units, while the reconstruction-only baselines better preserve the model's original decisions. Table 44 shows that the fidelity gain brought by Group-Hoyer is very limited; the vast majority of the coverage gain in Table 52 actually comes from cross-decoding and latent masking. Since this experiment uses different hyperparameter weights from the main comparison, I am still unclear about what this complete recipe adds in terms of substantial extra benefits over the simpler "cross-decoding plus masking" variant.
Overall, the new ablations and baselines indicate that the proposed regularizer and training recipe do not offer clear advantages over simpler regularizers or standard SAE baselines. Moreover, to support these claims, the paper has essentially undergone a near-complete rewrite. Combined with the missing literature on causal interventions in VLM SAEs pointed out by Reviewer fMsY, I believe this paper still requires a major overhaul and re-evaluation. Therefore, I am lowering my score from 3 to 2 .
Response to Reviewer bZTY
Thank you for the careful review. Both questions led to new experiments, and we revised the manuscript to reflect what those experiments establish.
W1 / Q1. How does the full objective change the fidelity–coverage trade-off?
We compared methods at a common SigLIP2 support target of 256 features (seeds 17–19), using the same semantic evaluator and decision-preservation protocol. The variable-support baselines were matched on mean activity (Table 2, Figure 2, §5.3).
The selected configuration recovers the most strict shared units in this comparison. Ordinary TopK and the AuxK + cross-decoding variant preserve more SugarCrepe decisions. In the objective ladder, AuxK leaves coverage at 39.0 units, and adding cross-decoding raises it to 56.3 with decision preservation (DP) unchanged relative to ordinary TopK at the reported precision. Adding shared latent masking to the preceding ladder variant increases coverage from 59.7 to 72.7 units and reduces SugarCrepe DP from 0.906 to 0.886 (Figure 2, Appendix N, Table 52). The three added reconstruction-only baselines preserve 0.5–2.5 percentage points more decisions than our selected model across SugarCrepe and SugarCrepe++. We revised the paper to describe this fidelity–coverage trade-off explicitly.
Group-Hoyer ablation. We replaced Group-Hoyer with plain Hoyer, ℓ1, and Group-L2,1, each calibrated to the same initial loss contribution (§4.4, Appendix M).
At this operating point, Group-Hoyer modestly improves reconstruction and decision preservation over plain Hoyer. The largest increases in coverage along the objective ladder occur when cross-decoding and shared latent masking are added. We also clarified in §3.1 that the elementary identities and bounds serve as diagnostics that motivate our measurements.
W2 / Q2. Do matched circuits retain their held-out intervention effects?
Yes, for activation exchange on SugarCrepe. We score each edit on the frozen model's final pooled caption embedding by restoring the reconstruction residual, xedit=x^edit+(x−x^). The resulting change to the dense embedding is exactly the change in the SAE reconstruction induced by editing the selected activations (§4.7).
During the rerun, we found that the original example-level split allowed base images to appear in both discovery and evaluation. The updated causal results in Table 3 and §5.5 use an image-disjoint split. We mapped the seed-17 top-32 circuit into seeds 18–21 by one-to-one Hungarian matching, separately using decoder cosine similarity and activation correlation, without reranking in the target seeds.
The excess drop is measured relative to eight activity-matched random feature sets. All eight transfer sign-flip tests remain significant after Holm correction. Under decoder matching, an average of 32.0% of mapped features on SugarCrepe and 25.8% on ARO also appear in the target seed's own top 32.
For ablation of each seed's own circuit, the paired 95% interval was above zero in three of five seeds. ARO exchange results depended on how the circuit was selected. We therefore scope the cross-seed claim to activation exchange on SugarCrepe at the final pooled caption embedding. The encoder's internal compositional mechanism remains an open question.
We would be grateful if you could look at the revised PDF, particularly Table 2, Figure 2, and Table 3, and let us know whether it addresses your concerns.