- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates how reinforcement learning with verifiable rewards affects intermediate reasoning traces. It interprets outcome-based RLVR as learning prompt augmentations that increase final-answer correctness, irrespective of their semantic validity. The theoretical analysis derives a trace-reweighting identity for the base policy conditioned on successful outcomes, obtained as a limiting case of KL-regularized reward maximization.
Empirically, the authors apply GRPO to Qwen2.5-0.5B and LLaMA-3.2-1B on GSM8K and evaluate on GSM8K and two GSM8K-Symbolic variants. They use a GPT-4o judge, validated against 175 human-annotated examples, to measure trace coherence. Their principal finding is that the conditional probability of answering correctly given an incoherent trace, β, increases in every reported setting. They also reinterpret previously published maze experiments as supporting evidence. The paper concludes that outcome rewards can reinforce both coherent and incoherent traces that lead to correct answers.
Critical strengths
-
The paper measures an informative aspect of reasoning quality that aggregate accuracy can obscure. Reporting (p_c), (\alpha), and (\beta) separates the prevalence of coherent traces from answer correctness within each trace category. Evaluating both successful and unsuccessful responses is useful because restricting trace evaluation to correct answers can conceal changes in the underlying trace distribution. The consistent increase in β across two model families and the evaluated distribution shifts merits investigation.
-
The idealized analysis provides a useful account of what outcome rewards constrain. The success-conditioned reweighting identity makes clear that an outcome-only objective contains no independent preference for semantic validity among trajectories with equivalent reward. This offers a concrete explanation for why incorrect reasoning can survive or receive reinforcement. The explicit acknowledgment that coherence is a proxy for validity, together with the reported human validation of the judge, also improves the transparency of the evaluation.
Critical weaknesses
1. The central conclusion conflates the absence of an explicit reasoning reward with the absence of an implicit incentive for correct reasoning.
Reinforcing some incorrect traces is compatible with increasing the overall probability of correct traces. Indeed, within the paper’s own success-conditioned model, fix a prompt and let denote correct reasoning, with , , and . Then
, provided that and .
Thus, the proposed framework itself permits an implicit incentive for correct reasoning.
The empirical argument has a related problem: an increase in does not imply an increase in the unconditional probability of producing an incoherent trace with a correct answer, . For LLaMA-3.2-1B on GSM8K at Pass@1, increases from 18.4% to 34.6%, while this joint probability decreases from 6.1% to 5.6% and coherence increases from 66.9% to 83.8% (Tables 3 and 5).
These findings can challenge the specific conjecture that decreases during training, which Wen et al. discuss separately from their conditional argument about increasing correct-CoT probability. They do not refute that conditional argument. This distinction is consequential because the claimed contradiction motivates much of the paper’s contribution.
2. The validation does not sufficiently establish the proposed explanation of practical RLVR.
Theorem 1 concerns the success-conditioned policy obtained as . At finite , the optimizer instead satisfies
.
Remark 1 treats the success-conditioned policy as the optimum of Equation 5 and assumes that the trained GRPO policy approximates it, without establishing the approximation. Comparing aggregate coherence and correctness before and after training does not directly test the predicted relationship between a fixed trace’s base-model solve probability and its probability change under RL.
The central empirical measurement also needs stronger validation. The judge achieves 81.1% accuracy, with 75% precision for the incoherent class, on a curated validation set. Consequently, some apparently incoherent but successful traces may be coherent traces misclassified by the judge. As coherence prevalence changes during training, such contamination can affect even without a corresponding change in actual shortcut behavior. Validation specifically within the correct-answer/incoherent-judgment subset is missing.
Finally, at , the coherent trace and the correct answer may belong to different samples, as the paper acknowledges. Therefore, cannot generally be interpreted as the probability of solving a problem through a coherent trace, despite the wording in Table 5. These limitations weaken the claimed connection between the measured statistics and the proposed mechanism.
Questions for authors
-
Clarification of the central claim: How do you reconcile the conclusion that RLVR does not implicitly incentivize correct reasoning with the increase in (P_{\pi^\star}(F)) when (\alpha>\beta), and with the substantial increases in (p_c) in your experiments? Is the intended conclusion instead that outcome rewards do not exclusively reinforce correct reasoning?
-
Clarification and supporting evidence for the theory: Which conclusions are intended to hold for finite-KL optimization and the actual GRPO implementation? What justifies the approximation in Remark 1? Do you have direct measurements relating fixed-trace probability changes to (P_{\mathrm{base}}(\mathrm{Sol}\mid x,c)), as predicted by the proposed explanation?
-
Requested validation of the current empirical claim: Does the increase in β persist under blinded human annotation of base and RL outputs, stratified by answer correctness and judge-assigned coherence? Please report the underlying joint counts and uncertainty, particularly for responses judged incoherent despite a correct answer.
-
Clarification and reanalysis of the K > 1 results: How do the conclusions change when coherent reasoning and answer correctness must occur in the same sampled response? Please distinguish this event from the intersection of “at least one coherent trace” and “at least one correct answer.”
3 comments
Response to Reviewer xhig (2/3)
1. The central claim (continued)
"an increase in β does not imply an increase in the unconditional probability of producing an incoherent trace with a correct answer"
We agree. β = β⁺ / (β⁺ + β⁻), where β⁺ and β⁻ are the numbers of responses with an incoherent trace and a correct or an incorrect answer, respectively, so β can rise because β⁺ grows or because β⁻ shrinks. Particularly, for LLaMA-3.2-1B on GSM8K the rise comes from β⁻ alone. We now report the two joint counts separately. On the mazes, the counts out of about 1,000 responses are:
| Experimental setting | Test set | β⁺ Base | β⁺ RL | β⁻ Base | β⁻ RL |
|---|---|---|---|---|---|
| (a) Normal, valid and invalid | Wilson | 17 | 35 | 491 | 335 |
| Kruskal | 18 | 45 | 489 | 310 | |
| DFS | 12 | 12 | 314 | 130 | |
| (b) Normal, valid only | Wilson | 17 | 20 | 487 | 325 |
| Kruskal | 17 | 30 | 492 | 295 | |
| DFS | 12 | 8 | 313 | 143 | |
| (c) Swapped, invalid only | Wilson | 515 | 847 | 485 | 153 |
| Kruskal | 521 | 886 | 479 | 114 | |
| DFS | 730 | 865 | 186 | 51 |
In (a), β⁻ shrinks on all three test sets and β⁺ grows on Wilson and Kruskal. In (b), where the β⁺ responses are left out of the update, β⁺ grows less on Wilson and Kruskal and falls on DFS, while β⁻ shrinks as in (a). In (c), the Swapped model setting, every trace is invalid, so the whole gain in accuracy is a gain in β⁺. The update on the β⁺ responses therefore raises their unconditional probability. Under GRPO, β thus rises because the update reinforces the invalid traces that reach a correct answer, on the basis of the answer alone, even while the total number of invalid traces falls.
2. The actual GRPO, and a direct measurement (Weakness 2, Question 2)
"Which conclusions are intended to hold for finite-KL optimization and the actual GRPO implementation? … Do you have direct measurements …?"
We have extended the analysis from the optimal policy to GRPO as implemented, with finite G, group-normalised advantages and the PPO-clipped surrogate. We view GRPO as an iterative procedure of self-training on the model's own verified responses, as in ReST-EM [1]. One GRPO step on a prompt x has four parts:
- Generate. Sample G responses from the current policy π_t(· | x).
- Verify. Score each answer with a binary reward r(y) and compute the group-relative advantage Â(y) = (r(y) − group mean) / group std. A group in which all answers are correct or all are wrong has an advantage of zero, Â = 0, for every response and is filtered out.
- Target. The groups that remain contain both correct and incorrect responses. Their advantages define the group-relative advantage-tilted policy π_new(y | x) ∝ π_t(y | x) · exp(η Â(y)). The advantage  depends on a response only through its answer: every correct response receives the same positive advantage and every incorrect response the same negative advantage, whatever its trace.
- Update as classification. With the importance-sampling ratio ρ = π_θ / π_t, the clipped surrogate of GRPO is exactly a weighted hinge loss [2]. At every token, the importance ratio acts as the score and the outcome as the label, with margin ε and weight |Â|. One mirror-descent step on this loss gives π_new of step 3 [2], and the gradient step of GRPO points in the direction that fits π_θ to π_new. Each update therefore moves π_θ towards π_new.
Thus, at finite G, a trace c is up-weighted in expectation when P_t(correct | x, c) > P_t(correct | x), whether the trace is valid or not, and the prompt-augmentation view applies.
Direct measurement. We ran this test on the maze model, with 200 prompts per test set and 16 base-model traces per prompt. For each fixed trace c we estimated its solve probability, P_base(Sol | x, c), and its log-probability under the base and the RL model. The change in the log-probability of a fixed trace after RL increases with its solve probability under the base model (rank correlation 0.66 to 0.70), and at equal solve probability a valid trace gains no more than an invalid one.
[1] Singh et al., arXiv:2312.06585. [2] Huang et al., "Neural PPO-Clip Attains Global Optimality: A Hinge Loss Perspective", arXiv:2110.13799.
Response to Reviewer xhig (3/3)
3. Validation of the measurement (Weakness 2, Question 3)
"Does the increase in β persist under blinded human annotation of base and RL outputs …? Please report the underlying joint counts and uncertainty …"
Stronger judges. We labelled every response again with two stronger judges, gpt-oss-120b and DeepSeek V4.1 Flash. On the validation set of the paper (175 human-labelled examples), their precision on incoherent traces is 0.87 and 0.90, and their accuracy is 87.4% and 88.6%, compared with 0.75 and 81.1% for GPT-4o.
Results with the stronger judges on GSM8K and GSM8K-Symbolic. The tables give p_c, α and β under each stronger judge, with a 95% interval for every Δβ.
gpt-oss-120b
| Model, dataset | p_c Base | p_c RL | α Base | α RL | β Base | β RL | Δβ [95% interval] |
|---|---|---|---|---|---|---|---|
| Qwen2.5-0.5B, GSM8K | 24.6 | 53.4 | 65.5 | 99.6 | 5.4 | 11.6 | +6.2 [+3.1, +8.9] |
| Qwen2.5-0.5B, Symbolic-P1 | 9.5 | 27.0 | 67.4 | 99.6 | 2.4 | 4.5 | +2.1 [+0.1, +3.9] |
| Qwen2.5-0.5B, Symbolic-P2 | 2.6 | 7.4 | 52.0 | 94.6 | 0.9 | 3.2 | +2.3 [+1.0, +3.7] |
| LLaMA-3.2-1B, GSM8K | 63.8 | 83.2 | 94.5 | 98.6 | 12.8 | 16.7 | +3.9 [−2.2, +9.3] |
| LLaMA-3.2-1B, Symbolic-P1 | 43.4 | 63.2 | 93.5 | 99.7 | 7.4 | 11.7 | +4.3 [+0.1, +8.1] |
| LLaMA-3.2-1B, Symbolic-P2 | 22.6 | 40.1 | 90.3 | 96.3 | 5.9 | 9.3 | +3.4 [+0.4, +6.2] |
DeepSeek V4.1 Flash
| Model, dataset | p_c Base | p_c RL | α Base | α RL | β Base | β RL | Δβ [95% interval] |
|---|---|---|---|---|---|---|---|
| Qwen2.5-0.5B, GSM8K | 24.6 | 55.0 | 66.5 | 99.2 | 5.1 | 8.9 | +3.8 [+1.0, +6.4] |
| Qwen2.5-0.5B, Symbolic-P1 | 9.2 | 27.1 | 71.7 | 99.6 | 2.2 | 4.4 | +2.2 [+0.3, +3.9] |
| Qwen2.5-0.5B, Symbolic-P2 | 2.0 | 7.2 | 65.0 | 94.4 | 0.8 | 3.4 | +2.6 [+1.2, +3.9] |
| LLaMA-3.2-1B, GSM8K | 63.7 | 84.0 | 95.5 | 98.6 | 11.3 | 12.3 | +1.0 [−4.7, +6.0] |
| LLaMA-3.2-1B, Symbolic-P1 | 43.6 | 62.9 | 94.9 | 99.8 | 6.0 | 12.1 | +6.1 [+2.0, +9.8] |
| LLaMA-3.2-1B, Symbolic-P2 | 22.5 | 41.9 | 92.9 | 96.2 | 5.3 | 6.7 | +1.4 [−1.3, +3.9] |
Under both stronger judges, the point estimate of Δβ is positive in all six cases, and its 95% interval excludes zero in 9 of the 12 cases. The rise is smaller than under GPT-4o, in particular for LLaMA-3.2-1B.
Human audit. As requested, we checked the responses that are judged incoherent despite a correct answer. In a blinded audit of 50 correct-answer GSM8K responses that both stronger judges flag as incoherent (25 base, 25 RL), the annotator found incorrect reasoning in 18 of 25 base responses (72%) and in 8 of 25 RL responses (32%), although these responses are long for a human to check exhaustively step by step. We therefore take our main results from the maze setting, where a program formally verifies every trace (point 1), and use the judged results on mathematical reasoning, the setting of Wen et al., as supporting evidence.
4. K > 1 (Weakness 2)
"at K > 1, the coherent trace and the correct answer may belong to different samples … despite the wording in Table 5"
For K > 1, Pass@K of the trace is measured independently of the answer, and α and β are now computed per response: every response is labelled by the coherence of its own trace and by its corresponding answer, so α and β never combine a trace and an answer from different samples. The table gives the change from Base to RL on the GSM8K test set, under GPT-4o and under the stronger judge gpt-oss-120b:
| Model, judge | Δα, K = 1 | Δα, K = 4 | Δα, K = 16 | Δβ, K = 1 | Δβ, K = 4 | Δβ, K = 16 |
|---|---|---|---|---|---|---|
| Qwen2.5-0.5B, GPT-4o | +31.8 | +34.1 | +33.4 | +8.0 | +9.7 | +8.4 |
| Qwen2.5-0.5B, gpt-oss-120b | +34.1 | +34.0 | +33.4 | +6.2 | +7.7 | +6.1 |
| LLaMA-3.2-1B, GPT-4o | +6.7 | +6.9 | +6.9 | +16.2 | +14.9 | +14.4 |
| LLaMA-3.2-1B, gpt-oss-120b | +4.1 | +3.7 | +4.3 | +3.9 | +4.9 | +5.2 |
β rises at every K under both judges, and the increase is smaller under the stronger judge. On the mazes, with program-verified traces and 1,024 sampled responses for each of 100 mazes per test set, β rises from 5.2 to 9.9 on Wilson and from 4.0 to 7.8 on Kruskal.
We hope that these changes address the reviewer's concerns, and we are happy to clarify any remaining point.
Response to Reviewer xhig (1/3)
We thank the reviewer for the constructive review. We answer the three questions in order, and Weaknesses 1 and 2 are covered under them.
1. The central claim (Weakness 1, Question 1)
Yes. We conclude that outcome rewards do not exclusively reinforce correct reasoning: under the success-conditioned model the probability of correct reasoning rises whenever α > β, and p_c rises in our experiments. The revised paper narrows the abstract and the conclusion accordingly. Our claim is that the update is blind to the trace: it up-weights every trace, valid or not, that makes a correct answer more likely. The correct-trace rate and β therefore rise together, as the prompt-augmentation view predicts. The swapped model is the limiting case: with no valid trace to select, accuracy still rises by 15 to 37 points.
On the mazes, a program verifies every trace. We ran three experimental settings: (a) Normal, updates on valid and invalid traces, (b) Normal, updates on valid traces only, and (c) Swapped, updates on invalid traces only. The table gives accuracy, p_c (the valid-trace rate), α and β in percent:
In (a), β rises on all three test sets, α is at its ceiling (97 to 99) before RL and stays flat, and the gain in accuracy (15 to 20 points) comes with a higher valid-trace rate, as selection on the answer predicts when α > β. In (b), β rises less than in (a) on all three test sets. This gives counterfactual evidence that incorrect reasoning is reinforced when invalid traces with a correct answer are in the update. In (c), the limiting case, the swapped model has no valid trace to select: validity stays at zero, α is not defined, and the whole gain in accuracy (15 to 37 points) is a rise in β. Thus the swapped model gives the cleanest evidence that a trace without semantics acts as an augmentation and is reinforced on the basis of the outcome during RLVR updates.
These results support our claim: RLVR reinforces any trace that makes a correct answer more likely, whether the trace is valid or not.
The revised paper no longer rests on a contradiction with Wen et al. Its contribution is the prompt-augmentation view, which accounts for the rise in the correct-trace rate that Wen et al. report and for the finding of Yu et al. that accuracy improves without a gain in the causal importance or sufficiency of the reasoning. The conditional result of Wen et al. does not concern β. Our measurement of β addresses two further statements in their paper.
The direction of β. Section 4 of Wen et al. ("Discussions on the effectiveness of GRPO") states that during RLVR training α increases and β decreases, so that the gap α − β widens. We measure α and β separately, and in the maze setting with verified traces the gap narrows:
Implicit incentive or selection. Wen et al. treat the reinforcement of incorrect CoTs as an exception that arises when α > β does not hold, and describe it as unintentional (their Section 4, "Discussions on failure modes in GRPO"). We argue that it occurs even while α > β holds, because RLVR selects based only on the answer.