acceptodds
Under review as a conference paper at ICLR 2027

Rewarding the Wrong Circuit: How RLVR Reshapes Safety in Reasoning Models

Abstract

Reinforcement learning with verifiable rewards (RLVR) can substantially improve reasoning, yet most safety evaluations compare only endpoints measured before and after training. We ask a mechanistic question that endpoint scores cannot answer: does safety remain in control after reward optimization? We track the same aligned model before, during, and after RLVR focused on capability and separate two functions that are usually conflated: recognizing that a request is unsafe and allowing that recognition to govern the final policy. Our hypothesis is decoupling between recognition and policy: RLVR can preserve harm recognition while reallocating causal control toward answer commitment driven by reward. We identify Reward Override Components (ROCs) by a dual causal criterion: a component must gain mediation of verifiable reward while losing mediation of refusal. On the frozen trajectory from Qwen2.5-7B-Instruct to Code-R1, the gap between recognition and refusal increases from 19.3 to 36.6 points; restoring ROC states from the reference checkpoint recovers 24.6 pp refusal. We then introduce Circuit Anchor RLVR (CA RLVR), which anchors only this discovered interface during capability RLVR. Under the protocol that follows the same model lineage, CA RLVR achieves 17.4% harmful compliance on reasoning jailbreaks versus 29.8% for the strongest adapted baseline while retaining 62.2 code average versus 62.5 for unmodified RLVR.

Primary area:
reinforcement learning
Keywords:
SAFETY, REASONING MODELS

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wmXcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether benign RLVR separates unsafe-request recognition from refusal behavior. It tracks model checkpoints and identifies Reward Override Components (ROCs) whose influence on verifiable task success increases while their influence on refusal decreases. Bidirectional activation patching tests their causal effects. The proposed Circuit Anchor RLVR constrains selected component activations toward an aligned reference model during training. The full method anchors both ROCs and stable refusal mediators and reports improved jailbreak safety with largely preserved code performance.

Critical strengths
  1. The paper addresses an important question and explains its motivation and method clearly. Tracking the same model across training and distinguishing recognition from refusal provides a useful perspective beyond endpoint safety scores.

  2. Bidirectional checkpoint patching and matched random-component controls provide convincing evidence that the selected components affect refusal. The mitigation results are also promising: average harmful compliance falls from 29.8% for the strongest adapted baseline to 17.4%, with a reported code average of 62.2.

Critical weaknesses
  1. The durability of selective anchoring is not established. The method constrains activations on calibration inputs rather than freezing the selected heads. Other heads, MLPs, or downstream pathways could compensate or bypass the anchored components. The current results show effectiveness within the reported training schedule, but do not explain why functional migration is prevented or whether safety remains protected under longer training.

  2. Generality and some measurement details need clarification. Evidence mainly concerns relatively small Qwen-derived models, with limited validation on newer or stronger backbones. Recognition and action use different prompts, so their gap does not directly establish unsafe recognition during harmful generation. Donor matching and the verifier/refusal margins also need precise definitions. The reported capability comparison includes a published Code-R1 reference rather than a fully matched rerun.

Questions for authors
  1. Why do other components not compensate for or bypass the anchored components? Can longer training, repeated component localization, and post-training patching show whether the same components still control safety?

  2. Does the mechanism and the anchoring benefit persist on newer, stronger models? The reported 14B replication is useful, but broader mitigation results would strengthen the generality claim.

  3. How are successful, unsuccessful, harmful, and benign donor states matched? At the final instruction position, identical inputs have identical activations within a fixed checkpoint regardless of later sampled outcomes. Please clarify which inputs differ and define the verifier and refusal margins.

  4. Can the authors report uncertainty across the three training seeds and rerun all capability evaluations under the same protocol, including the unmodified RLVR baseline?

Reviewer_nDdeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper examines how Reinforcement Learning with Verifiable Rewards (RLVR) affects the safety behavior of reasoning models. The authors decompose safety into two functions: (1) Safety Recognition, which measures whether the model identifies an input request as unsafe, and (2) Refusal Policy Control, which measures whether the final generated response refuses the request. The paper hypothesizes that capability-oriented RLVR preserves safety recognition while shifting internal causal control toward reward-driven answer generation, widening the gap between recognition and refusal.

To localize this effect, the authors define Reward Override Components (ROCs)—components that gain reward mediation and lose refusal mediation after RLVR—and perform activation patching across intermediate checkpoints. Based on these components, the authors propose Circuit Anchor RLVR (CA-RLVR), which periodically penalizes the L2L_2 distance between the current model's activations and the initial aligned model's activations at the selected components on a calibration set during RLVR training, aiming to reduce harmful compliance on reasoning jailbreaks while preserving task capability.

Critical strengths
  1. Separate evaluation routes across training checkpoints: The paper evaluates both a recognition route and an action route across multiple intermediate checkpoints ({0, 128, 256, 512, 832}) along the same model training lineage.
  2. Bidirectional activation patching protocol: The analysis performs two-way checkpoint interventions—patching post-RLVR component states into the base checkpoint and patching base states back into the post-RLVR checkpoint—alongside norm-matched random component controls.
Critical weaknesses
  1. Self-contradiction between the core narrative ("safety loss under multi-step reasoning pressure") and the intervention locus ("final instruction position" during prefill):

    • Narrative vs. Method: Throughout the paper (e.g., Introduction and Section 5.4), the authors argue that RLVR models remain safe on direct harmful prompts, and that safety control is lost specifically under long multi-step reasoning pressure ("when reasoning is long, adversarially scaffolded, or verifier driven"). However, in Section 3.3, all causal mediation scores (McR,McSM_c^R, M_c^S), ROC localization, checkpoint patching, and CA-RLVR anchoring are performed exclusively at the final instruction position (the last prompt token during prefill, before a single reasoning token is generated).
    • Core contradictions:
      1. If safety failure is already determined and reversible at the final prompt token prior to generation, then the phenomenon is a prefill representation shift rather than a failure that emerges dynamically across long reasoning chains.
      2. During autoregressive generation of thousands of CoT tokens, every layer's attention heads attend directly to all preceding instruction tokens and generated reasoning tokens. The paper provides no analysis of how intervening on 22–30 components at a single prefill token position governs generation trajectories thousands of tokens later, nor does it analyze component dynamics during CoT generation itself.
      3. At the final instruction position (prior to sampling), the hidden state of a given model θ\theta on a prompt xx is strictly deterministic. Yet Section 3.3 states that McR(θ)M_c^R(\theta) is computed at the final instruction position by patching the state from a "matched unsuccessful trajectory." Since successful and unsuccessful rollouts of the same prompt share the exact same deterministic prefill state, it is unclear how donor states differ at that position unless they are patched from entirely different prompts—which would confound prompt content with reward mediation.
  2. Missing experimental details for evaluating "Safety Recognition" (qθ(x)q_\theta(x)):

    • The main text lacks a clear specification of the experimental setup for the Recognition Route in Section 3.2 (e.g., exact prompt templates, parsing criteria, and how qθ(x)q_\theta(x) is extracted).
    • It is unclear whether the input xx in the Recognition Route is the raw harmful request or the full adversarial reasoning-jailbreak prompt (e.g., wrapped with HCoT or Trotter templates). Furthermore, querying the model with an external meta-question ("whether the request is unsafe and what an assistant aligned for safety should do") removes the reasoning-generation context altogether. Showing that a model can classify harmfulness under a separate meta-prompt does not establish that safety recognition is internally active during the Action Route rollout.
  3. Incomplete evaluation on over-refusal and general/reasoning capability benchmarks:

    • Over-refusal evaluation: The proposed alignment intervention is not thoroughly evaluated on standard over-refusal benchmarks. Suitable over-refusal tests include the benign portion of XSTest [1] and adversarial benign prompts from WildJailbreak [2].
    • General and reasoning utility benchmarks: While Section 4 mentions StrongREJECT, WildJailbreak, HarmBench, MATH-500, and AIME2024, Table 2 omits StrongREJECT, WildJailbreak, and HarmBench, and CA-RLVR is only evaluated on a single code trajectory (Code-R1). Even though Section 5.6 examines math RLVR checkpoints diagnostically, CA-RLVR training is never evaluated on math RLVR or additional model families. Evaluating on a representative subset of reasoning and general-capability benchmarks—such as AlpacaEval, MATH-500, AIME, GPQA, and MMLU-Pro—is needed to assess utility under the same intervention settings.
    • Unaddressed two-stage training overhead: Computing OcO_c in Equation (2) requires the post-RLVR checkpoint θRL\theta_{\text{RL}}. Consequently, CA-RLVR requires training an unconstrained RLVR model to completion, running activation patching sweeps to identify ROCs, and then retraining RLVR from scratch with the anchor loss. The computational cost of this two-stage requirement is not reported.
  4. Terminology proliferation and unclear mathematical formulations:

    • Undefined or loosely defined terminology: The paper introduces multiple terms—such as Safety Recognition, Refusal Policy Control, Answer Commitment, and Reward Override Components—without rigorous formal definitions.
    • Lack of definition for "Components" and missing equations for McR,McSM_c^R, M_c^S: In the Introduction (Line 089), the authors state "Second, we define Reward Override Components using a dual causal criterion: components must gain verifiable reward mediation and lose safety and refusal mediation after RLVR," without defining what a "component" is. Only in Section 3.3 is it mentioned that components correspond to pre-WOW_O attention head outputs and MLP outputs at the final instruction token, and the paper never provides mathematical equations for McR(θ)M_c^R(\theta) or McS(θ)M_c^S(\theta), nor explains how binary code/math verification outcomes yield a continuous "verifier margin."
    • Mismatch between Equation (3) and the evaluated CA-RLVR method: Equation (3) defines CA-RLVR as anchoring c∈CROCc \in \mathcal{C}_{\text{ROC}} with weights wcw_c proportional to OcO_c. However, Section 5.5 reveals that anchoring ROCs alone yields 19.6% HCR, whereas the main reported CA-RLVR result (17.4% HCR) anchors both "stable refusal mediators + ROCs." Because stable refusal mediators have Oc=0O_c = 0 by Equation (2), their selection rule and weighting in Equation (3) are left unexplained.

References:

  • [1] Röttger, P., et al. (2024). XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. NAACL.
  • [2] Jiang, L., et al. (2024). WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models. NeurIPS.
Questions for authors

refer to weakness

Reviewer_U3kDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether benign reinforcement learning with verifiable rewards (RLVR) preserves explicit harm recognition while weakening refusal on reasoning-jailbreak prompts. It identifies Reward Override Components (ROCs) through changes in reward- and safety-patching effects, tests them with bidirectional checkpoint interventions, and proposes Circuit Anchor RLVR to regularize selected activations toward an aligned reference. On the Code-R1 trajectory, the paper reports 17.4% average harmful compliance across four reasoning-jailbreak suites, compared with 43.2% for unmodified RLVR and 29.8% for the strongest adapted baseline, with a code average of 62.2.

Critical strengths
  1. The question is useful for safety evaluation. Tracking explicit harm recognition and refusal across capability-training checkpoints could reveal changes that endpoint comparisons miss. The reported divergence motivates a closer examination of safety during RLVR, even if its mechanistic interpretation requires further support.

  2. The intervention and ablation design is promising. Bidirectional checkpoint patching, matched random components, and safety-component controls probe whether localized changes affect refusal. Table 3 also compares ROC-only, safety-mediator-only, combined, and full-state anchoring. These are meaningful controls, and the reported results suggest that selective anchoring may improve the safety–capability tradeoff.

Critical weaknesses
  1. The link between the behavioral gap and the proposed mechanism remains incomplete. The recognition score is obtained under a separate classification prompt, whereas refusal is measured under the original generation prompt (Section 3.2; Appendix A). Their divergence demonstrates different behavior under these conditions, but does not by itself show that recognized harmfulness is overridden within the action-generation computation. The patching results provide causal evidence that the tested interventions change behavior; the connection to this particular mechanism needs clearer support. Section 3.3 also leaves the code-task “verifier margin” and the construction of unsuccessful-trajectory donors at the final instruction position insufficiently specified. These definitions matter for interpreting the reward side of the ROC criterion. Without that link, the results support a narrower conclusion: the identified components affect refusal under the tested interventions. They do not yet establish that RLVR overrides recognized harm during generation.

  2. The method description and main comparison need reconciliation. Equation 3 anchors only ROCs, whereas Section 5.5 and Table 3 attribute the headline 17.4% HCR to safety mediators plus ROCs; ROC-only anchoring gives 19.6%. The combined selection and weighting rule therefore needs to be stated. Figure 4 also shows negative values for a score defined as nonnegative in Equation 2; a labeling or plotting explanation may resolve this, but its effect on component selection is unclear. Finally, Tables 2–3 omit uncertainty despite the bootstrap and three-seed protocol stated in Section 4, and Table 2 compares the method's code score with a published Code-R1 score. Matched evaluation and per-seed results would help assess capability retention and whether the HCR advantage holds across seeds. These concerns may be fixable, but currently limit my confidence in the reported advantage.

Questions for authors
  1. How are “verifier margin” and the unsuccessful-trajectory donor defined? Given that the intervention is applied at the final instruction position (Section 3.3), please specify the donor input, checkpoint, and matching rule. How do the recognition measurements and patching results together support the proposed override mechanism within the action route?

  2. What is the complete objective and component-selection rule for the combined method achieving 17.4% HCR, including the weights assigned to safety mediators? What quantity is plotted in Figure 4, given that Equation 2 defines a nonnegative score but the figure shows negative values? Does resolving this discrepancy affect component selection or any reported result?

  3. Can you provide per-seed safety and capability results with uncertainty for the paired method differences? Table 2 uses the published Code-R1 code score; can you report a matched evaluation of unmodified RLVR using your own pipeline? Please also specify the SafeReAct/SInternal adaptations and tuning, the guard or judge model used for each benchmark, and the human-audit sample size and agreement with the automatic judge.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.