- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether benign RLVR separates unsafe-request recognition from refusal behavior. It tracks model checkpoints and identifies Reward Override Components (ROCs) whose influence on verifiable task success increases while their influence on refusal decreases. Bidirectional activation patching tests their causal effects. The proposed Circuit Anchor RLVR constrains selected component activations toward an aligned reference model during training. The full method anchors both ROCs and stable refusal mediators and reports improved jailbreak safety with largely preserved code performance.
Critical strengths
-
The paper addresses an important question and explains its motivation and method clearly. Tracking the same model across training and distinguishing recognition from refusal provides a useful perspective beyond endpoint safety scores.
-
Bidirectional checkpoint patching and matched random-component controls provide convincing evidence that the selected components affect refusal. The mitigation results are also promising: average harmful compliance falls from 29.8% for the strongest adapted baseline to 17.4%, with a reported code average of 62.2.
Critical weaknesses
-
The durability of selective anchoring is not established. The method constrains activations on calibration inputs rather than freezing the selected heads. Other heads, MLPs, or downstream pathways could compensate or bypass the anchored components. The current results show effectiveness within the reported training schedule, but do not explain why functional migration is prevented or whether safety remains protected under longer training.
-
Generality and some measurement details need clarification. Evidence mainly concerns relatively small Qwen-derived models, with limited validation on newer or stronger backbones. Recognition and action use different prompts, so their gap does not directly establish unsafe recognition during harmful generation. Donor matching and the verifier/refusal margins also need precise definitions. The reported capability comparison includes a published Code-R1 reference rather than a fully matched rerun.
Questions for authors
-
Why do other components not compensate for or bypass the anchored components? Can longer training, repeated component localization, and post-training patching show whether the same components still control safety?
-
Does the mechanism and the anchoring benefit persist on newer, stronger models? The reported 14B replication is useful, but broader mitigation results would strengthen the generality claim.
-
How are successful, unsuccessful, harmful, and benign donor states matched? At the final instruction position, identical inputs have identical activations within a fixed checkpoint regardless of later sampled outcomes. Please clarify which inputs differ and define the verifier and refusal margins.
-
Can the authors report uncertainty across the three training seeds and rerun all capability evaluations under the same protocol, including the unmodified RLVR baseline?