- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies reward-model preference reversals under input changes intended to preserve meaning and human preference. It uses sparse autoencoders (SAEs) to detect these failures and proposes two mitigation methods: suppressing selected SAE features and learning an additive reward correction. Experiments cover four reward models, two datasets, and three perturbation families. The results show substantial detection gains and strong recovery for pattern injection and backdoors. However, the evidence is less convincing for the proposed causal explanation and broad robustness claims.
Critical strengths
-
Relevant problem and practical approach. Improving a frozen reward model without backbone retraining is a useful objective.
-
Meaningful experimental coverage. Multiple models and perturbation mechanisms provide evidence beyond a single case study.
-
Useful additional validation. The appendix includes raw/PCA correction baselines, random-feature steering, mixed perturbations, and held-out templates and rewriters.
-
Independent utility evaluation. RewardBench 2 helps assess whether attack recovery comes at the expense of other capabilities. Residual correction generally performs well on this measure.
Critical weaknesses
-
Semantic and preference preservation are assumed rather than verified. The defining premise requires human preference to remain unchanged after perturbation. Rewrite instructions alone do not establish this, especially after repeated adversarial edits. Appended warnings and endorsements can also legitimately change response quality. A blind human evaluation of representative examples would strengthen the central claim.
-
The formal feature definition can miss preference reversals. The absolute winning–losing feature difference discards direction. For example, scores changing from (1, 0) to (0, 1) reverse preference while leaving the absolute difference unchanged. Opposite changes across samples can also cancel inside the expectation. The definition therefore needs a clearer connection to preference orientation and causal reward influence.
-
Detection may recognize perturbation artifacts rather than preference failure. The evaluation compares successfully inverted perturbations with benign originals. Without unsuccessful attacks and ordinary benign variations, high classification accuracy does not establish failure detection. Similarly, Table 3’s benign accuracy measures retention on a selected, initially correct subset, rather than accuracy on the complete benign distribution.
-
The controls do not fully isolate the proposed mechanism. Steering begins from the SAE reconstruction, so it changes reconstruction residuals alongside selected features. A reconstruction-only control is needed. Moreover, selected and random feature steering perform similarly on paraphrases: perturbed accuracy is 35.4% versus 34.8% in Table 10. This weakens the claim that the selected features specifically drive paraphrase instability.
-
Reproducibility and uncertainty need clarification. The paper does not clearly specify whether matched variants and repeated prompts remain within the same split, which HH partition trains the SAEs, or how validation governs model selection. Test denominators and seed variation are also missing. Some TruthfulQA subsets appear small, making confidence intervals important.
-
Several headline claims exceed the results. Utility preservation is substantially stronger for residual correction than for feature steering. Qwen3-4B’s RewardBench 2 accuracy falls from 83.0% to 57.9% under one steering configuration. Raw features also achieve near-perfect detection AUC for TruthfulQA injection, qualifying the claim that separation is nearly invisible in raw representations. “Without retraining the backbone” is more accurate than describing the entire framework as training-free.
Questions for authors
Please see weaknesses.
2 comments
Thanks for the response and the extra controls. Narrowing the detection and causal claims brings the paper much closer to what the results support, and Table 12 clearly shows that the steering gains on backdoors and injection do not just come from SAE reconstruction.
Still, the new numbers confirm my main concerns. On paraphrasing, which is the most realistic case here, selected SAE steering reaches 35.4% compared to 36.7% for reconstruction alone, and residual correction recovery on HH stays limited at 20.5% on Llama-3.1-8B and 17.8% on the unseen rewriter. On HH injection, human preference holds in only 14 out of 20 pairs, so a fair share of those flips may be real quality changes rather than instability. A few evaluation issues are also still open: the detection split remains row-level instead of grouped by pair, and the main tables have no confidence intervals or seed variation even though TruthfulQA test sets go as low as 28 pairs.
The revision is clearer, but given the narrow scope where the method works reliably and these remaining evaluation gaps, I will keep my score at 2.
Response to Reviewer KvN3
We thank the reviewer for the thoughtful review.
Preference preservation (W1). We ran the suggested blind evaluation with three independent annotators. Procedure and results are in our response to Reviewer zRbx (W2, Q2) and Appendix A.2.4.
Feature definition (W2). On direction, see our response to Reviewer fUnd (Q1). On cancellation, Definition 2 is a population-level criterion by design. It characterizes instability as a property of a perturbation family δ ∈ Δ, through the shift that δ induces in expectation over all triples (Eq. 10), and therefore targets the dimensions that δ shifts systematically. On causal reward influence, see W4.
Detection and benign accuracy (W3). We agree and aligned the claim with our definitions, so the detector now targets instability-inducing perturbations (see our response to Reviewer zRbx, W1, which also reports a benign-variation check). B in Table 3 is renamed "Benign retention" (initially correct pairs), and RewardBench 2 covers the full benign distribution, including items the base model gets wrong.
Isolating the mechanism (W4). We added a reconstruction-only control, in which the hidden state is replaced by its SAE reconstruction without modifying any feature (Appendix B.7). The table gives perturbed accuracy on Poisoned-7B HH.
On injection and backdoor, the gains are not explained by the reconstruction. On paraphrase, all three coincide, so we narrowed the causal claim to injection and backdoor, although the overall trade-off on paraphrase is still best with SAE Residual Correction (see our response to Reviewer fUnd, W3).
Reproducibility and uncertainty (W5).
Overstated claims (W6). Revised as follows.