- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
A steering direction defines only one step. Later steps need a rule. The paper calls this rule the execution law. Re-encoding between steps adds interaction error. An exact formula links this error to decode-encode error. The paper proves that three properties are separate: reconstruction, interaction error and reaching targets. The paper tests eight public GPT-2 SAEs. In these SAEs, width improves reconstruction but raises decode-encode error. The paper also tests in-house GPT-2 and Gemma-2-2B SAEs. There, per-order terms usually pick the order with less interaction error. Final error barely changes.
Critical strengths
Figure 5 confirms the exact formula numerically.
In eight public GPT-2 SAEs, decode-encode error grows with width. This finding is a useful warning for state-dependent steering that re-encodes. Decoder conditioning explains only part of this growth (Appendix N).
Critical weaknesses
Interaction error has a narrow scope and a small effect. With a linear SAE decoder, fixed-vector steering has zero interaction error. Fixed-vector steering is the most common kind. The authors found no released method that needs their state-dependent feedback rule (Appendix A). With this rule, the chosen order cuts interaction error by 24% to 38%. Final error changes by about 1% or less. The paper itself stresses this point. Section 6 presents the formula mainly as a code check. The check tests that code follows its stated execution law. The narrow scope and small effect explain my informativeness score and rating.
The abstract supports its order claim with a rank correlation of 0.98 to 0.99. This correlation measures the size of interaction error, not the order. Order accuracy is 93% to 94%. When no feature turns on or off, per-order terms give each interaction error exactly. Section 5.2 and Appendix J therefore treat this match as a code check. The abstract calls the same match a prediction. When a steered feature turns on or off, order accuracy falls to 50% to 78%. The abstract mentions this drop without a number. My presentation score mainly reflects this use of the rank correlation.
Questions for authors
The paper should test whether order matters in a released state-dependent method. Two FLAS fields are one example.
At strength 0.02 times the hidden-state norm, interaction error is 4% to 7% of final error. The paper should report this share at the largest strength in Appendix H.
The authors should report order accuracy in the abstract. The authors should present the rank correlation as a code check.