acceptodds
Under review as a conference paper at ICLR 2027

Belief-Shift Reward: Redistributing RLVR Credit with Policy-Internal Target Support

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) provides a scalable framework for improving reasoning post-training, but binary outcome rewards cannot distinguish among trajectories that an external verifier judges correct. To address this limitation, we propose Belief-Shift Reward (BSR), a lightweight reward transformation that preserves the external verifier as the sole criterion of correctness. Within the verifier-correct set, BSR differentiates trajectories by the change each one induces in the policy's support for the verified target relative to the prompt-only prior. Because this prior is shared within a prompt, it cancels under correct-set normalization. BSR therefore requires at most one detached posterior target-scoring pass per verifier-correct trajectory, without an auxiliary reward model or explicit step-level scoring. Across mathematical reasoning, agentic search, and code-output prediction, BSR yields aggregate improvements over matched RLVR baselines, spanning multiple model families and four group-relative optimization objectives. These results support the practical utility of policy-internal target support for finer-grained within-correct credit assignment when the verifier target can be scored by the policy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
RLVR, Reward Shaping, Reasoning, Policy-Internal Signals, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VP1jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses a limitation of standard RL training for reasoning: Binary reward gives same credit to correct trajectories, but cannot distinguish the quality of reasoning. The method proposed is BSR, which measures the quality of the reasoning trace by the probability the model generates the target output. Experiment is conducted on math reasoning, agentic search and code-output prediction tasks, and BSR out-performs standard binary reward under different RL approaches and models.

Critical strengths

The experiment is abundant, covering three different task types, multiple model sizes and different RL methods. The performance is also significant despite tested on a single seed.

Critical weaknesses
  1. The presentation is unclear and hinders understanding. BSR scoring is simple on its own: Replacing the output text with a unified format, and measure the probability of generating the output. But the main text does not explain this at all, and specifically, how to compute sis_i in Equation (8) is actually computed. The implementation details are all deferred to Section C.1. In addition, the paper tries to build a complex theory around this already simple and easily understood method, for instance, using a telescoping method in Equation (6) and (7) to explain that intermediate reasoning probability does not need to be computed. The paper tries to place it as a motivation, but the final BSR scoring formula (Equation (5)) is better motivated than the derivation.
  2. The paper hides key implementation choice from main text. In appendix C.1, the paper mentions that they truncate the concluding answer-formatting block, and replace it with another answering format ("Therefore, the answer is "), which is not the natural generation of the model. The paper calls it "target-leakage prevention", but this actually changes nothing but the final output format, so it is unnecessary. It does not solve the leakage issue neither. From Table 5 we see that both responses indicate at the end that the answer should be 0, so the probability of the final answer cannot reflect the quality of the intermediate reasoning at all. More seriously, it forces the model to predict a sequence that the model is not naturally trained to generate, so it is an unreliable measure of generation quality. The paper fails to justify why such a replacement is implemented and why this key implementation is not put in the main text.

Overall, I admit the empirical performance gain but I highly question the novelty and motivation of the method.

Questions for authors

Can the author explain why the output truncation and replacement in Appendix C.1 is needed? Can the authors post an example of raw response, along with the processed response?

Reviewer_FogCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Belief Shift Reward, or BSR, to address the limited ability of binary reinforcement learning with verifiable rewards to distinguish among multiple correct trajectories. The method retains an external verifier for assessing answer correctness and redistributes rewards among correct trajectories according to the probability of generating the target answer after conditioning on the reasoning process. BSR does not require an additional reward model and can be integrated with several existing group based reinforcement learning objectives. The experimental results suggest that the method can provide improvements across mathematical reasoning, search, and code related tasks.

Critical strengths
  1. The method is simple and extensible. BSR constructs an additional reward signal directly from the policy probability assigned to the target answer. It does not require a separately trained process reward model and can be incorporated into existing reinforcement learning with verifiable rewards frameworks with relatively limited modification.
  2. The paper addresses an important limitation of binary rewards. In groups where all sampled trajectories are correct, conventional group relative rewards often provide no further learning signal. BSR offers a plausible approach for distinguishing among correct trajectories and continuing to learn from such groups.
Critical weaknesses
  1. The relationship between target support and reasoning quality could be clarified further. A higher probability assigned to the target answer may indicate more reliable reasoning, but it may also reflect greater consistency with the model’s existing preferences or with a particular answer format.
  2. The mechanism through which continued learning from fully correct groups improves generalization remains somewhat unclear. BSR increases the probability of correct trajectories with high target support while decreasing the probability of other correct trajectories. A more detailed explanation of why this confidence sharpening process should improve performance on unseen problems would strengthen the paper.
Questions for authors
  1. BSR uses the probability assigned to the target answer to rank correct trajectories. Could the authors further explain why higher target support should generally correspond to better reasoning, rather than greater familiarity with a particular answer expression or reasoning pattern?
  2. BSR provides a learning signal for fully correct groups, while fully incorrect groups still offer no effective learning signal. Could the authors explain why further reinforcing high confidence trajectories among successful samples is expected to improve generalization?
Reviewer_Bpa2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the inability of binary rewards in RLVR to distinguish among verifier-correct trajectories and the vanishing outcome-based relative advantages in all-correct groups. It proposes Belief-Shift Reward (BSR), which preserves the external verifier as the sole criterion of correctness while allocating bonus rewards among correct trajectories according to the policy’s relative support for the verified target conditioned on each trajectory. BSR requires no auxiliary reward model or explicit step-level scoring. Experiments show aggregate improvements over matched RLVR baselines across mathematical reasoning, agentic search, and code-output prediction.

Critical strengths
  • Clear motivation. The paper addresses the inability of binary RLVR rewards to differentiate among correct trajectories and the vanishing relative advantages in all-correct groups.
  • Simple and lightweight design. BSR integrates easily into existing RLVR frameworks, preserves external correctness verification, and eliminates the shared prior through normalization, requiring neither an auxiliary reward model nor explicit step-level scoring.
Critical weaknesses
  1. Insufficient related-work coverage and limited novelty. Closely related methods, including LaTRO, VeriFree, and RLPR, already use reference-answer likelihoods or probabilities as learning signals but are not discussed in the paper. BSR’s novelty appears primarily in restricting and normalizing this signal within the verifier-correct subset. Clearer positioning and controlled comparisons are needed to establish the value of this specific design.
  2. Experimental limitations.
  • Insufficient baseline comparisons. Table 15 provides only a cross-paper comparison with Skywork-O1-PRM, S-GRPO, and GRPO-VPS, without controlled comparisons under matched training and evaluation settings.
  • Missing key ablation. The paper lacks a controlled comparison between bonuses restricted to verifier-correct trajectories and bonuses applied to all trajectories, while retaining the same binary outcome reward and comparable bonus budgets.
  • Limited model coverage. Experiments primarily use relatively small, earlier-generation models, leaving effectiveness on stronger reasoning models and longer reasoning traces unverified.
  1. Limited relevance of sentence-level diagnostics. Although explicitly presented as diagnostic, the sentence-level analysis provides only indirect support for BSR and does not directly validate reasoning-quality rankings among verifier-correct trajectories.
Questions for authors

please see Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.