- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper constructs an invertible mixing transformation that changes an SAE's feature basis while preserving its reconstruction. It compares mixing with a clone-split control and measures probing and steering outcomes. A steering-strength analysis finds that the sign of the mixing effect changes with intervention magnitude, motivating calibration diagnostics.
Critical strengths
-
The reconstruction-preserving construction provides a concrete test of what reconstruction metrics cannot identify. The nonnegative codes and normalized decoder rows make the intervention more informative than an unrestricted change-of-basis identity alone.
-
Figure 3 and Table 10 show that intervention strength can reverse the comparison. This supports reporting answer-position KL and intervention magnitude. The paper also reports failed checks and withdrawn analyses rather than presenting only favorable results.
Critical weaknesses
-
The primary outcome does not establish useful steering. Section 4 measures a raw target logit rather than a margin or probability. A common offset to all output logits can change this quantity without changing predictions or KL. Appendix D.3 reports negative absolute SAE effects and compares them with a worse random-direction baseline, but that comparison does not establish improved target preference. The strongest calibrated result resolves on irony, while several replications remain uncertain. The observed sensitivity of a selected steering direction therefore supports a narrower conclusion than degradation of basis quality or steering utility.
-
The support and calibration evidence do not fully match the main analysis. Corollary 2 gives constant support only at positive mixing doses, while the fitted slope includes zero, allowing a support step to contribute to the slope. Removing zero leaves negative point estimates in Table 6, so this does not explain away the effect, but corresponding intervals are needed. Table 8 also reports clone support increases of 12.798 and 12.495, exceeding Stage B's ten-pair maximum. Finally, Table 11 extends calibration labels beyond the single pairing measured in Table 10 and includes BoolQ despite stating that it was not separately classified.
Questions for authors
-
Does the effect persist for target-versus-source logit margins or target probabilities at the calibrated coefficient? How does it compare with a random rotation matched for the change in steering direction?
-
Can you report positive-dose-only intervals, identify the experiment behind Table 8, and provide Stage B's support counts? Which cells in the cross-configuration tally have directly measured calibration diagnostics?
Addressing the weaknesses, and answering the questions put up.
Dear Reviewer,
Thank you for your patience. We did not hear back on the question we asked, so we have done our best to address every comment, using the activation-side and decoder-geometry measures we have. The revised PDF is uploaded; all new or changed text is in blue, and page numbers below refer to it.
1. Does the pattern hold with top-k selected by output effect? (Q b)
Addressed. We ran two output-based selection rules, all else at the headline configuration (k=10):
On SNLI the effect holds under both output-based rules; on irony it does not, which is why the claim is now stated for SNLI. Where: App. H.5, Table 16 (pp. 34–35); Table 2 rows "Attribution-selected" and "Measured-effect" (p. 7); Sec. 6 (pp. 9–10).
2. Can cheaper metrics catch the mixing before steering? (Q a)
Addressed. New App. H.8. Some cheap checks do catch the dial:
The appendix also states their limit: in the toy model, decoder coherence and co-firing flag the harmless clone control as strongly as real mixing. On manifold measures: any measure computed on the reconstruction is blind to the dial by Corollary 1, since the reconstruction does not change; only measures on the codes or the decoder can detect it. Where: App. H.8 "Cheaper detectors of mixing" (p. 36); Finding 1 (p. 5); Table 2 FMS row (p. 7).
3. Results are noisy and sensitive to the strength c; the sign flip muddles the takeaway
Addressed. Two fixes.
Section 5 now states the rule: a steering comparison needs its strength, a validity diagnostic and a target-preference outcome; Finding 3 adds recommendation (v), report a margin. The strength-1.0 sign flip (KL 17–31 nats) is kept as the warning it was meant to be. Where: Sec. 4 bootstrap note (p. 7); App. F (p. 31); Sec. 5 and Finding 3 (p. 8); Table 14 (p. 34).
4. The dial is post hoc and does not measure natural polysemanticity from training
Addressed. Agreed for the real model, and the paper says so (Finding 2, Limitations). The new toy model covers training directly, because the true features are known for SAEs trained from scratch. There, training itself produces dictionaries that reconstruct better and are worse:
Where: App. H.7, Table 18 tests 3, 6, 7, 8, 12, 13 (pp. 35–36); Sec. 6 (p. 10); Finding 2 and Limitations (pp. 6, 10).
We would be grateful if you could let us know whether this addresses your questions, and we are happy to clarify anything further.