- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
- The authors propose an "empirical contract" to account for a misconception that a feature tracking a concept doesn't show that the model uses that feature. They argue that because steering a behavior drives the model so far off distribution, we need to test sufficiency and use at values observed on natural inputs.
- They contribute an audit showing that Gemma scope latents steer out of range and under natural contrast are much less effective
Critical strengths
They contribute an "empirical contract" that can be used to validate the extent to which a model actually uses a feature when we steer that feature. The audit of Gemma scope validates their theoretical contribution in a practical setting.
Critical weaknesses
- The evidence is insufficient given the generality of the claim. For example, there are only two published latents and one dense direction per model.
- Contribution is somewhat minimal given the existing large body of work on interchange interventions and denoising activation patching. The main contribution is framing of existing methods, so the primary takeaway is the result of the empirical audit, which (per the above) is limited in scope.
Questions for authors
- To what extent do you think the low results are because you only edit a single token and layer position? (The refusal control shows that full ablation is much more effective than the single-site version, perhaps the same holds for the entity latents?)
- Which update rule should people use? Why?
2 comments
Also, we explained Which update rule (Q2). Test a published SAE claim with writes along the decoder columns under which the claim was made, the direction that steering also uses, with gain correction for individual latents, and report the attained values. Set a dense direction by projection replacement, which keeps the write along the direction: its orthogonal-to-total norm ratio is 0.5% (Llama) and 1.1% (Gemma), against 75.1% and 72.9% for writes through SAE decoders (Appendix C). A closer match to the requested feature values is not by itself a reason to prefer an update. For the agreement set, the encoder-constrained update matches them more closely yet installs 0.80 against 0.95 and rescues 0.56 against 0.88 (Section 5.5); downstream layers read the residual stream, not the encoder's output (Section 6). When the two updates disagree in this way, the disagreement is part of the result: the interpretation holds for the writes under which it was found.
We have made substantial revisions to improve readability, bringing the main findings forward in the abstract and introduction, giving the alias control its own section, and presenting installation, removal and rescue separately. We also added experiments extending the edited tokens and measuring rescue for both published Gemma latents to address concerns about the experimental evidence. These additions strengthen the empirical case for our contribution, while the revised presentation makes the comparisons easier to follow. We respectfully ask you to consider these improvements and the additional evidence when reassessing the paper, and would appreciate your considering an upward revision of the rating.
Thank you for the review. The paper's main results are these. A published SAE latent whose steering raises knowledge abstention by 46 points carries little of that behavior at its natural values. At the known-entity latent's own site, the full residual carries an accuracy gain that the latent does not. The same released feature set gives different answers under two update rules. The review's concerns about the evidence are coverage (W1) and single-site edits (Q1); both are answered below with data. The abstract now reports the dose and alias results and the introduction leads with them, the alias control has its own section (5.3), and Table 3 and Figure 3 report installation, removal and rescue separately.
Evidence versus generality (W1). The audit covers three kinds of representations in three models: two published Gemma latents, known–unknown directions in Gemma and Llama, and a released 147-feature agreement set. It finds small natural-value effects for the published latents, opposite installation–removal asymmetries for the dense directions (the Llama direction installs 0.89 and removes 0.07; the Gemma direction removes 0.61 and installs 0.27), and strong installation, removal and rescue for the agreement set under the decoder update (0.95, 0.95 and 0.88). The tests also detect substantial effects where the latents show little: in the same Gemma model and task, at the latents' own sites, the dense direction removes 0.61 of the natural contrast, the full residual recovers +10.6 of the +11.2-point alias gain, and the same single-token edit at a steering-range value raises abstention by +12.0 points. For the specific claim that steering success does not establish use at natural values, one published case is sufficient, and the latents provide it.
Framing of existing methods (W2). Our contribution combines an empirical contract for feature claims with its application to published representations. The contract separates association and control from sufficiency and use and measures the latter two at natural values; steering-based validation stops at control. Applied to a published latent that steers strongly, it shows that the latent carries little of the behavior at its natural values. Four comparisons make the audit interpretable. Two hold everything else fixed: a same-operator dose comparison (+1.5 versus +12.0 points) and a same-site full-residual control (+10.6 versus −0.2 points). Two vary how the edit is made: the update rule (for the agreement set, installation 0.95 versus 0.80 and rescue 0.88 versus 0.56) and the edited tokens (adding the question mark changes the interchange fractions by at most 0.03; removing the refusal direction at one token lowers refusal by 6 points, against 45 for ablation at every layer and position). Interchange and denoising patching are standard tools; Section 2 says so and positions the work relative to Geiger et al. (2025) and Makelov et al. (2025).
Single token and layer (Q1). In a separate experiment, adding the question mark to the entity token in the natural-value interchange changes each installation and removal fraction by at most 0.03, one flipped outcome (Section 5.5, Appendix C). At the known-entity latent's own site, copying the full residual recovers the alias accuracy gain (+10.6 of +11.2 points) while copying the latent gives −0.2 (Section 5.3). Each latent is tested at its published layer and token (Table 2); editing its direction at other layers would be a different intervention from the natural-value tests reported here. The latents are naturally silent at the end-of-instruction tokens that published steering also edits: zero activation at all five of them on every prompt of the 148 pairs, and on the 400 steering questions only the first newline is active, on 2% of them (Section 5.5, Appendix C). Published steering yields +46 points against +12 for the same value at the entity token alone, but the two also differ in write rule and question set (Section 5.2, Appendix B.1); the paper treats the edited tokens and the update rule as part of the claim (Section 5.5). The refusal gap you cite is on the removal side; the same direction at the same single site installs +29 points when set to the natural class mean (Section 5.2), so a single-site natural-value edit is not inherently weak. The controls show that the tested sites can support substantial behavioral effects: the same single-token edit to the unknown-entity latent moves abstention when the value is in the steering range (Section 5.2), and the Llama direction installs 0.89 of the natural contrast at one token (Section 5.4).