acceptodds
Under review as a conference paper at ICLR 2027

RLVR as Prompt Augmentation reinforces both Correct and Incorrect Reasoning to the Correct Answer

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR)-based post-training of Large Language Models (LLMs) has been shown to improve accuracy on reasoning tasks and continues to attract significant attention. While RLVR is known to improve solution accuracy, there has been little agreement on its effect on the so-called reasoning traces–with previous investigations coming to differing conclusions. We argue that outcome rewards cannot distinguish between reasoning from any other scaffold that leads to final answer correctness. To support our claims, we show that RLVR as learning prompt-augmentation conditioned on solution correctness, where a CoT is reinforced when it raises the probability that the model produces a correct final answer independent of whether CoT is correct or not. The objective is indifferent to every other property of the CoT including semantics, interpretability and faithfulness. We derive testable predictions and evaluate the prior results in a formally verifiable domain and test on GSM8K and two of its variants, GSM8K-Symbolic-P1 and GSM8K-Symbolic-P2. For GSM8K, where trace validity cannot be reliably established, we measure trace coherence, defined by the absence of errors. We compute , the probability of a correct final answer given a coherent trace, and , the probability given an incoherent one, for the base model and after RLVR training. Across all settings rises without exception, contradicting the claim that RLVR implicitly incentivizes correct reasoning. Thus RLVR learns prompt augmentations conditioned on final answer correctness: the objective does not specifically upweight valid reasoning, but any CoT that increases final-answer correctness.

Primary area:
reinforcement learning
Keywords:
RLVR, Reasoning, Traces, Math Domain, Prompt Augmentation, CoT

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xhigOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how reinforcement learning with verifiable rewards affects intermediate reasoning traces. It interprets outcome-based RLVR as learning prompt augmentations that increase final-answer correctness, irrespective of their semantic validity. The theoretical analysis derives a trace-reweighting identity for the base policy conditioned on successful outcomes, obtained as a limiting case of KL-regularized reward maximization.

Empirically, the authors apply GRPO to Qwen2.5-0.5B and LLaMA-3.2-1B on GSM8K and evaluate on GSM8K and two GSM8K-Symbolic variants. They use a GPT-4o judge, validated against 175 human-annotated examples, to measure trace coherence. Their principal finding is that the conditional probability of answering correctly given an incoherent trace, β, increases in every reported setting. They also reinterpret previously published maze experiments as supporting evidence. The paper concludes that outcome rewards can reinforce both coherent and incoherent traces that lead to correct answers.

Critical strengths
  1. The paper measures an informative aspect of reasoning quality that aggregate accuracy can obscure. Reporting (p_c), (\alpha), and (\beta) separates the prevalence of coherent traces from answer correctness within each trace category. Evaluating both successful and unsuccessful responses is useful because restricting trace evaluation to correct answers can conceal changes in the underlying trace distribution. The consistent increase in β across two model families and the evaluated distribution shifts merits investigation.

  2. The idealized analysis provides a useful account of what outcome rewards constrain. The success-conditioned reweighting identity makes clear that an outcome-only objective contains no independent preference for semantic validity among trajectories with equivalent reward. This offers a concrete explanation for why incorrect reasoning can survive or receive reinforcement. The explicit acknowledgment that coherence is a proxy for validity, together with the reported human validation of the judge, also improves the transparency of the evaluation.

Critical weaknesses

1. The central conclusion conflates the absence of an explicit reasoning reward with the absence of an implicit incentive for correct reasoning.

Reinforcing some incorrect traces is compatible with increasing the overall probability of correct traces. Indeed, within the paper’s own success-conditioned model, fix a prompt and let FF denote correct reasoning, with p=Pbase(F)p=P_{\mathrm{base}}(F), α=Pbase(Sol∣F)\alpha=P_{\mathrm{base}}(\mathrm{Sol}\mid F), and β=Pbase(Sol∣¬F)\beta=P_{\mathrm{base}}(\mathrm{Sol}\mid\neg F). Then

Pπ⋆(F)=pαpα+(1−p)β>pP_{\pi^\star}(F)=\frac{p\alpha}{p\alpha+(1-p)\beta}>p, provided that α>β\alpha>\beta and 0<p<10 < p < 1.

Thus, the proposed framework itself permits an implicit incentive for correct reasoning.

The empirical argument has a related problem: an increase in β\beta does not imply an increase in the unconditional probability of producing an incoherent trace with a correct answer, (1−pc)β(1-p_c)\beta. For LLaMA-3.2-1B on GSM8K at Pass@1, β\beta increases from 18.4% to 34.6%, while this joint probability decreases from 6.1% to 5.6% and coherence increases from 66.9% to 83.8% (Tables 3 and 5).

These findings can challenge the specific conjecture that β\beta decreases during training, which Wen et al. discuss separately from their conditional argument about increasing correct-CoT probability. They do not refute that conditional argument. This distinction is consequential because the claimed contradiction motivates much of the paper’s contribution.

2. The validation does not sufficiently establish the proposed explanation of practical RLVR.

Theorem 1 concerns the success-conditioned policy obtained as η→∞\eta\rightarrow\infty. At finite η\eta, the optimizer instead satisfies

πη(y∣x)∝πbase(y∣x)eηR(y)\pi_\eta(y\mid x)\propto\pi_{\mathrm{base}}(y\mid x)e^{\eta R(y)}.

Remark 1 treats the success-conditioned policy as the optimum of Equation 5 and assumes that the trained GRPO policy approximates it, without establishing the approximation. Comparing aggregate coherence and correctness before and after training does not directly test the predicted relationship between a fixed trace’s base-model solve probability and its probability change under RL.

The central empirical measurement also needs stronger validation. The judge achieves 81.1% accuracy, with 75% precision for the incoherent class, on a curated validation set. Consequently, some apparently incoherent but successful traces may be coherent traces misclassified by the judge. As coherence prevalence changes during training, such contamination can affect β\beta even without a corresponding change in actual shortcut behavior. Validation specifically within the correct-answer/incoherent-judgment subset is missing.

Finally, at K>1K>1, the coherent trace and the correct answer may belong to different samples, as the paper acknowledges. Therefore, pcαp_c\alpha cannot generally be interpreted as the probability of solving a problem through a coherent trace, despite the wording in Table 5. These limitations weaken the claimed connection between the measured statistics and the proposed mechanism.

Questions for authors
  1. Clarification of the central claim: How do you reconcile the conclusion that RLVR does not implicitly incentivize correct reasoning with the increase in (P_{\pi^\star}(F)) when (\alpha>\beta), and with the substantial increases in (p_c) in your experiments? Is the intended conclusion instead that outcome rewards do not exclusively reinforce correct reasoning?

  2. Clarification and supporting evidence for the theory: Which conclusions are intended to hold for finite-KL optimization and the actual GRPO implementation? What justifies the approximation in Remark 1? Do you have direct measurements relating fixed-trace probability changes to (P_{\mathrm{base}}(\mathrm{Sol}\mid x,c)), as predicted by the proposed explanation?

  3. Requested validation of the current empirical claim: Does the increase in β persist under blinded human annotation of base and RL outputs, stratified by answer correctness and judge-assigned coherence? Please report the underlying joint counts and uncertainty, particularly for responses judged incoherent despite a correct answer.

  4. Clarification and reanalysis of the K > 1 results: How do the conclusions change when coherent reasoning and answer correctness must occur in the same sampled response? Please distinguish this event from the intersection of “at least one coherent trace” and “at least one correct answer.”

3 comments
AuthorsOpenReview ↗

Response to Reviewer xhig (1/3)

We thank the reviewer for the constructive review. We answer the three questions in order, and Weaknesses 1 and 2 are covered under them.

1. The central claim (Weakness 1, Question 1)

"Is the intended conclusion instead that outcome rewards do not exclusively reinforce correct reasoning?"

Yes. We conclude that outcome rewards do not exclusively reinforce correct reasoning: under the success-conditioned model the probability of correct reasoning rises whenever α > β, and p_c rises in our experiments. The revised paper narrows the abstract and the conclusion accordingly. Our claim is that the update is blind to the trace: it up-weights every trace, valid or not, that makes a correct answer more likely. The correct-trace rate and β therefore rise together, as the prompt-augmentation view predicts. The swapped model is the limiting case: with no valid trace to select, accuracy still rises by 15 to 37 points.

On the mazes, a program verifies every trace. We ran three experimental settings: (a) Normal, updates on valid and invalid traces, (b) Normal, updates on valid traces only, and (c) Swapped, updates on invalid traces only. The table gives accuracy, p_c (the valid-trace rate), α and β in percent:

Experimental settingTest setAcc BaseAcc RLp_c Basep_c RLα Baseα RLβ Baseβ RLΔβ [95% interval]
(a) Normal, valid and invalidWilson49.665.149.263.097.497.83.39.5+6.2 [+3.0, +9.4]
Kruskal49.767.249.364.597.297.23.612.7+9.1 [+5.6, +12.9]
DFS65.085.264.484.598.899.23.78.5+4.8 [+0.7, +9.3]
(b) Normal, valid onlyWilson50.066.549.665.597.498.53.45.8+2.4 [−0.3, +5.4]
Kruskal49.469.149.167.597.197.93.39.2+5.9 [+3.0, +9.2]
DFS65.183.764.583.598.899.23.75.3+1.6 [−1.8, +5.5]
(c) Swapped, invalid onlyWilson51.584.70.00.0n/an/a51.584.7+33.2 [+29.3, +36.9]
Kruskal52.188.60.00.0n/an/a52.188.6+36.5 [+32.8, +40.1]
DFS79.794.40.00.0n/an/a79.794.4+14.7 [+11.7, +17.8]

In (a), β rises on all three test sets, α is at its ceiling (97 to 99) before RL and stays flat, and the gain in accuracy (15 to 20 points) comes with a higher valid-trace rate, as selection on the answer predicts when α > β. In (b), β rises less than in (a) on all three test sets. This gives counterfactual evidence that incorrect reasoning is reinforced when invalid traces with a correct answer are in the update. In (c), the limiting case, the swapped model has no valid trace to select: validity stays at zero, α is not defined, and the whole gain in accuracy (15 to 37 points) is a rise in β. Thus the swapped model gives the cleanest evidence that a trace without semantics acts as an augmentation and is reinforced on the basis of the outcome during RLVR updates.

These results support our claim: RLVR reinforces any trace that makes a correct answer more likely, whether the trace is valid or not.

“These findings can challenge the specific conjecture that β decreases during training, which Wen et al. discuss separately from their conditional argument … They do not refute that conditional argument. This distinction is consequential because the claimed contradiction motivates much of the paper’s contribution.”

The revised paper no longer rests on a contradiction with Wen et al. Its contribution is the prompt-augmentation view, which accounts for the rise in the correct-trace rate that Wen et al. report and for the finding of Yu et al. that accuracy improves without a gain in the causal importance or sufficiency of the reasoning. The conditional result of Wen et al. does not concern β. Our measurement of β addresses two further statements in their paper.

The direction of β. Section 4 of Wen et al. ("Discussions on the effectiveness of GRPO") states that during RLVR training α increases and β decreases, so that the gap α − β widens. We measure α and β separately, and in the maze setting with verified traces the gap narrows:

Test setGap α − β, BaseGap α − β, RL
Wilson94.188.3
Kruskal93.684.5
DFS95.190.7

Implicit incentive or selection. Wen et al. treat the reinforcement of incorrect CoTs as an exception that arises when α > β does not hold, and describe it as unintentional (their Section 4, "Discussions on failure modes in GRPO"). We argue that it occurs even while α > β holds, because RLVR selects based only on the answer.

AuthorsOpenReview ↗

Response to Reviewer xhig (2/3)

1. The central claim (continued)

"an increase in β does not imply an increase in the unconditional probability of producing an incoherent trace with a correct answer"

We agree. β = β⁺ / (β⁺ + β⁻), where β⁺ and β⁻ are the numbers of responses with an incoherent trace and a correct or an incorrect answer, respectively, so β can rise because β⁺ grows or because β⁻ shrinks. Particularly, for LLaMA-3.2-1B on GSM8K the rise comes from β⁻ alone. We now report the two joint counts separately. On the mazes, the counts out of about 1,000 responses are:

Experimental settingTest setβ⁺ Baseβ⁺ RLβ⁻ Baseβ⁻ RL
(a) Normal, valid and invalidWilson1735491335
Kruskal1845489310
DFS1212314130
(b) Normal, valid onlyWilson1720487325
Kruskal1730492295
DFS128313143
(c) Swapped, invalid onlyWilson515847485153
Kruskal521886479114
DFS73086518651

In (a), β⁻ shrinks on all three test sets and β⁺ grows on Wilson and Kruskal. In (b), where the β⁺ responses are left out of the update, β⁺ grows less on Wilson and Kruskal and falls on DFS, while β⁻ shrinks as in (a). In (c), the Swapped model setting, every trace is invalid, so the whole gain in accuracy is a gain in β⁺. The update on the β⁺ responses therefore raises their unconditional probability. Under GRPO, β thus rises because the update reinforces the invalid traces that reach a correct answer, on the basis of the answer alone, even while the total number of invalid traces falls.

2. The actual GRPO, and a direct measurement (Weakness 2, Question 2)

"Which conclusions are intended to hold for finite-KL optimization and the actual GRPO implementation? … Do you have direct measurements …?"

We have extended the analysis from the optimal policy to GRPO as implemented, with finite G, group-normalised advantages and the PPO-clipped surrogate. We view GRPO as an iterative procedure of self-training on the model's own verified responses, as in ReST-EM [1]. One GRPO step on a prompt x has four parts:

  1. Generate. Sample G responses from the current policy π_t(· | x).
  2. Verify. Score each answer with a binary reward r(y) and compute the group-relative advantage Â(y) = (r(y) − group mean) / group std. A group in which all answers are correct or all are wrong has an advantage of zero, Â = 0, for every response and is filtered out.
  3. Target. The groups that remain contain both correct and incorrect responses. Their advantages define the group-relative advantage-tilted policy π_new(y | x) ∝ π_t(y | x) · exp(η Â(y)). The advantage  depends on a response only through its answer: every correct response receives the same positive advantage and every incorrect response the same negative advantage, whatever its trace.
  4. Update as classification. With the importance-sampling ratio ρ = π_θ / π_t, the clipped surrogate of GRPO is exactly a weighted hinge loss [2]. At every token, the importance ratio acts as the score and the outcome as the label, with margin ε and weight |Â|. One mirror-descent step on this loss gives π_new of step 3 [2], and the gradient step of GRPO points in the direction that fits π_θ to π_new. Each update therefore moves π_θ towards π_new.

Thus, at finite G, a trace c is up-weighted in expectation when P_t(correct | x, c) > P_t(correct | x), whether the trace is valid or not, and the prompt-augmentation view applies.

Direct measurement. We ran this test on the maze model, with 200 prompts per test set and 16 base-model traces per prompt. For each fixed trace c we estimated its solve probability, P_base(Sol | x, c), and its log-probability under the base and the RL model. The change in the log-probability of a fixed trace after RL increases with its solve probability under the base model (rank correlation 0.66 to 0.70), and at equal solve probability a valid trace gains no more than an invalid one.

[1] Singh et al., arXiv:2312.06585. [2] Huang et al., "Neural PPO-Clip Attains Global Optimality: A Hinge Loss Perspective", arXiv:2110.13799.

AuthorsOpenReview ↗

Response to Reviewer xhig (3/3)

3. Validation of the measurement (Weakness 2, Question 3)

"Does the increase in β persist under blinded human annotation of base and RL outputs …? Please report the underlying joint counts and uncertainty …"

Stronger judges. We labelled every response again with two stronger judges, gpt-oss-120b and DeepSeek V4.1 Flash. On the validation set of the paper (175 human-labelled examples), their precision on incoherent traces is 0.87 and 0.90, and their accuracy is 87.4% and 88.6%, compared with 0.75 and 81.1% for GPT-4o.

Results with the stronger judges on GSM8K and GSM8K-Symbolic. The tables give p_c, α and β under each stronger judge, with a 95% interval for every Δβ.

gpt-oss-120b

Model, datasetp_c Basep_c RLα Baseα RLβ Baseβ RLΔβ [95% interval]
Qwen2.5-0.5B, GSM8K24.653.465.599.65.411.6+6.2 [+3.1, +8.9]
Qwen2.5-0.5B, Symbolic-P19.527.067.499.62.44.5+2.1 [+0.1, +3.9]
Qwen2.5-0.5B, Symbolic-P22.67.452.094.60.93.2+2.3 [+1.0, +3.7]
LLaMA-3.2-1B, GSM8K63.883.294.598.612.816.7+3.9 [−2.2, +9.3]
LLaMA-3.2-1B, Symbolic-P143.463.293.599.77.411.7+4.3 [+0.1, +8.1]
LLaMA-3.2-1B, Symbolic-P222.640.190.396.35.99.3+3.4 [+0.4, +6.2]

DeepSeek V4.1 Flash

Model, datasetp_c Basep_c RLα Baseα RLβ Baseβ RLΔβ [95% interval]
Qwen2.5-0.5B, GSM8K24.655.066.599.25.18.9+3.8 [+1.0, +6.4]
Qwen2.5-0.5B, Symbolic-P19.227.171.799.62.24.4+2.2 [+0.3, +3.9]
Qwen2.5-0.5B, Symbolic-P22.07.265.094.40.83.4+2.6 [+1.2, +3.9]
LLaMA-3.2-1B, GSM8K63.784.095.598.611.312.3+1.0 [−4.7, +6.0]
LLaMA-3.2-1B, Symbolic-P143.662.994.999.86.012.1+6.1 [+2.0, +9.8]
LLaMA-3.2-1B, Symbolic-P222.541.992.996.25.36.7+1.4 [−1.3, +3.9]

Under both stronger judges, the point estimate of Δβ is positive in all six cases, and its 95% interval excludes zero in 9 of the 12 cases. The rise is smaller than under GPT-4o, in particular for LLaMA-3.2-1B.

Human audit. As requested, we checked the responses that are judged incoherent despite a correct answer. In a blinded audit of 50 correct-answer GSM8K responses that both stronger judges flag as incoherent (25 base, 25 RL), the annotator found incorrect reasoning in 18 of 25 base responses (72%) and in 8 of 25 RL responses (32%), although these responses are long for a human to check exhaustively step by step. We therefore take our main results from the maze setting, where a program formally verifies every trace (point 1), and use the judged results on mathematical reasoning, the setting of Wen et al., as supporting evidence.

4. K > 1 (Weakness 2)

"at K > 1, the coherent trace and the correct answer may belong to different samples … despite the wording in Table 5"

For K > 1, Pass@K of the trace is measured independently of the answer, and α and β are now computed per response: every response is labelled by the coherence of its own trace and by its corresponding answer, so α and β never combine a trace and an answer from different samples. The table gives the change from Base to RL on the GSM8K test set, under GPT-4o and under the stronger judge gpt-oss-120b:

Model, judgeΔα, K = 1Δα, K = 4Δα, K = 16Δβ, K = 1Δβ, K = 4Δβ, K = 16
Qwen2.5-0.5B, GPT-4o+31.8+34.1+33.4+8.0+9.7+8.4
Qwen2.5-0.5B, gpt-oss-120b+34.1+34.0+33.4+6.2+7.7+6.1
LLaMA-3.2-1B, GPT-4o+6.7+6.9+6.9+16.2+14.9+14.4
LLaMA-3.2-1B, gpt-oss-120b+4.1+3.7+4.3+3.9+4.9+5.2

β rises at every K under both judges, and the increase is smaller under the stronger judge. On the mazes, with program-verified traces and 1,024 sampled responses for each of 100 mazes per test set, β rises from 5.2 to 9.9 on Wilson and from 4.0 to 7.8 on Kruskal.

We hope that these changes address the reviewer's concerns, and we are happy to clarify any remaining point.

Reviewer_BTBKOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a debate around the effect of RLVR on reasoning trace quality; while Wen et al. (2025) claims that RLVR implicitly incentivizes correct reasoning traces, recent empirical work (e.g., Yu et al., 2026) shows that RLVR increases answer accuracy without reinforcing legible or causally important reasoning traces. The authors characterize the optimal policy of the KL-regularized objective as the base policy conditioned on success; they then argue that RLVR acts as “prompt augmentation” (Kambhampati et al., 2025), reinforcing any CoT that raises the probability of a correct answer. The authors predict that RLVR should raise answer accuracy without raising either trace validity (verifiable domains) or “trace coherence.” In experiments on Qwen2.5-0.5B and LLaMA-3.2-1B, they show that β\beta – the proportion of incoherent traces with correct answers – indeed rises as a consequence of RLVR on GSM8K. The authors take this as evidence against the claim that RLVR implicitly incentivizes correct reasoning.

Critical strengths
  1. This work is a valuable empirical extension of the work done by Wen et al. (2025), who claim that β\beta should fall over the course of RLVR training but don’t actually measure it. The authors provide a stronger measurement of trace coherence than Wen et al. (2025) (i.e., they define coherence independent of the answer). Overall, this work addresses an important empirical gap in the debate around the impact of RLVR.
Critical weaknesses

(I included a third weakness because it's important but also easy to fix.)

  1. The paper is fairly small in scale and could be made more rigorous. The models tested are very small and are only trained for 100 steps with no mention of the number of seeds used (and no confidence intervals reported). It also only focuses on the GSM8K dataset. It’s also possible that gains in accuracy come from format learning rather than answer correctness; this doesn’t appear to be accounted for in the paper.
  2. The authors claim that RLVR “can increase final-answer accuracy through both correct and incorrect reasoning traces;” however, it isn’t clear that they have actually shown that incoherent reasoning is reinforced. A rise in β\beta doesn’t necessarily indicate that incoherent traces with correct answers are being reinforced (i.e., comprise a larger share of model responses); it can also mean that the overall number of incoherent traces is decreasing (e.g., the incoherent traces with incorrect answers get pruned away, while the ones with correct answers remain). Table 5 in the Appendix indeed seems to show that the proportion of problems solved via an incoherent trace barely changes or even decreases over the course of RL training. This is a weaker result than what is discussed in the main body of the paper; it seems more accurate to say that incoherent traces with correct answers can remain after RL training, but that models in practice do not actually learn to satisfy the objective by increasing the share of incoherent traces with correct answers. Furthermore, the number of incoherent traces may be somewhat inflated; the LLM judge’s precision on the incoherent traces is fairly low (0.75), and 21% of coherent traces are marked as incoherent (Table 2). Trace incoherence is also a fairly weak proxy of trace validity, since it counts traces with harmless errors or mistakes that models correct as incoherent.
  3. The prose needs some editing; there are grammatical errors throughout (misplaced commas, misuse of single vs. plural nouns, wrongly capitalized nouns).
Questions for authors

Clarification requests:

  1. Can the authors clarify whether an increase in β\beta is meant to be interpreted as an increase in incoherent traces with correct answers (as a proportion of total model responses), or as a decrease in incoherent traces with incorrect answers?

Suggestions for future work:

  1. Can the authors provide some experimental results with larger models?
  2. Does GPT-4o’s judgements align with a stronger model’s judgements? Additional judge validation would be valuable.
2 comments
AuthorsOpenReview ↗

Response to Reviewer BTBK (1/2)

We thank the reviewer for the constructive review. We first answer the clarification request and then address the weaknesses. The two suggestions for future work (larger models, a stronger judge) are answered under Weaknesses 1 and 2.

Clarification on the increase in β

"… an increase in incoherent traces with correct answers (as a proportion of total model responses), or … a decrease in incoherent traces with incorrect answers?"

Both parts contribute, and the update itself raises β⁺. Here β = β⁺ / (β⁺ + β⁻), where β⁺ and β⁻ are the numbers of responses with an incoherent trace and a correct or an incorrect answer, respectively, so β can rise either because β⁺ grows or because β⁻ shrinks. For LLaMA-3.2-1B on GSM8K, the rise comes from β⁻ alone.

We now report the two joint counts separately, on the mazes, where a program formally verifies every trace. We ran three experimental settings that differ in which responses receive the update: (a) Normal, updates on valid and invalid traces. (b) Normal, updates on valid traces only: responses with an invalid trace and a correct answer are filtered out. (c) Swapped, updates on invalid traces only. The counts out of about 1,000 responses per test set are:

Experimental settingTest setβ⁺ Baseβ⁺ RLβ⁻ Baseβ⁻ RL
(a) Normal, valid and invalidWilson1735491335
Kruskal1845489310
DFS1212314130
(b) Normal, valid onlyWilson1720487325
Kruskal1730492295
DFS128313143
(c) Swapped, invalid onlyWilson515847485153
Kruskal521886479114
DFS73086518651

In (a), β⁻ shrinks on all three test sets and β⁺ grows on Wilson and Kruskal. In (b), where the β⁺ responses are left out of the update, β⁺ grows less on Wilson and Kruskal and falls on DFS, while β⁻ shrinks as in (a). In (c), every trace is invalid, so the whole gain in accuracy comes from β⁺. The update on the β⁺ responses therefore raises their unconditional probability: they are reinforced on the basis of the correctness of the final answer. β thus rises through both parts, and the growth of β⁺ is an effect of the update.

Weakness 1: scale and rigour

"The models tested are very small … no mention of the number of seeds … no confidence intervals … only focuses on the GSM8K dataset … format learning …"

Larger model and harder math datasets. We trained Qwen3.5-9B-Base with GRPO on DAPO-Math and tested it on MATH-500 and AIME 2024 + 2025, in the setting of Wen et al. and with their CoT-judge prompt. α and β both rise:

Benchmarkp_c Basep_c RLα Baseα RLβ Baseβ RLΔβ [95% interval]
MATH-50091.693.489.394.99.536.4+26.9 [+7.0, +45.7]
AIME 2024+202545.056.763.091.20.03.8+3.8 [0.0, +13.3]

We also added the maze experiments with verified traces, which give a clean measure of α and β and remove the dependence on LLM-as-judge errors. The table gives accuracy, p_c (the valid-trace rate), α and β in percent on about 1,000 mazes per test set, before RL (Base) and after 70 GRPO steps (RL):

Experimental settingTest setAcc BaseAcc RLp_c Basep_c RLα Baseα RLβ Baseβ RLΔβ [95% interval]
(a) Normal, valid and invalidWilson49.665.149.263.097.497.83.39.5+6.2 [+3.0, +9.4]
Kruskal49.767.249.364.597.297.23.612.7+9.1 [+5.6, +12.9]
DFS65.085.264.484.598.899.23.78.5+4.8 [+0.7, +9.3]
(b) Normal, valid onlyWilson50.066.549.665.597.498.53.45.8+2.4 [−0.3, +5.4]
Kruskal49.469.149.167.597.197.93.39.2+5.9 [+3.0, +9.2]
DFS65.183.764.583.598.899.23.75.3+1.6 [−1.8, +5.5]
(c) Swapped, invalid onlyWilson51.584.70.00.0n/an/a51.584.7+33.2 [+29.3, +36.9]
Kruskal52.188.60.00.0n/an/a52.188.6+36.5 [+32.8, +40.1]
DFS79.794.40.00.0n/an/a79.794.4+14.7 [+11.7, +17.8]

In (a), β rises on all three test sets, α is at its ceiling (97 to 99) before RL and stays flat, and the gain in accuracy (15 to 20 points) comes with a higher valid-trace rate, as selection on the answer predicts when α > β. In (b), β rises less than in (a) on all three test sets. This gives counterfactual evidence that incorrect reasoning is reinforced when invalid traces with a correct answer are in the update. In (c), the limiting case, the swapped model has no valid trace to select: validity stays at zero, α is not defined, and the whole gain in accuracy (15 to 37 points) is a rise in β. Thus the swapped model gives the cleanest evidence that a trace without semantics acts as an augmentation and is reinforced on the basis of the outcome during RLVR updates.

AuthorsOpenReview ↗

Response to Reviewer BTBK (2/2)

Weakness 1: scale and rigour (continued)

Pass@K and confidence intervals. We evaluated at K = 1, 4 and 16 sampled responses per problem. Each result is from one training run, and the 95% intervals are over the test problems. At K = 4 and K = 16, β also rises under GPT-4o and gpt-oss-120b.

Model, judgeΔβ, K = 1Δβ, K = 4Δβ, K = 16
Qwen2.5-0.5B, GPT-4o+8.0+9.7+8.4
Qwen2.5-0.5B, gpt-oss-120b+6.2+7.7+6.1
LLaMA-3.2-1B, GPT-4o+16.2+14.9+14.4
LLaMA-3.2-1B, gpt-oss-120b+3.9+4.9+5.2

Format learning. The answer format is a verifiable part of the answer: a response with no final answer in the required format is scored as an incorrect answer, for the base model and for the RL model alike. The trace label does not use the format, so a format error is never counted as an incoherent trace. In the maze setting the base model already writes the answer section in 97.7% of its outputs, so there is little format to learn, and β rises there as well (maze table above).

Weakness 2: judge precision, and coherence as a proxy for validity

"… the LLM judge's precision on the incoherent traces is fairly low (0.75) … Trace incoherence is also a fairly weak proxy of trace validity …"

Stronger judges. As suggested, we labelled every response again with two stronger judges, gpt-oss-120b and DeepSeek V4.1 Flash. On the 175 human-labelled examples their precision on incoherent traces is 0.87 and 0.90, and their accuracy is 87.4% and 88.6%, compared with 0.75 and 81.1% for GPT-4o. The table gives Δβ under each judge on GSM8K and its two Symbolic variants:

Model, datasetΔβ, GPT-4oΔβ, gpt-oss-120b [95% interval]Δβ, DeepSeek V4.1 Flash [95% interval]
Qwen2.5-0.5B, GSM8K+8.0+6.2 [+3.1, +8.9]+3.8 [+1.0, +6.4]
Qwen2.5-0.5B, Symbolic-P1+6.0+2.1 [+0.1, +3.9]+2.2 [+0.3, +3.9]
Qwen2.5-0.5B, Symbolic-P2+4.5+2.3 [+1.0, +3.7]+2.6 [+1.2, +3.9]
LLaMA-3.2-1B, GSM8K+16.2+3.9 [−2.2, +9.3]+1.0 [−4.7, +6.0]
LLaMA-3.2-1B, Symbolic-P1+14.0+4.3 [+0.1, +8.1]+6.1 [+2.0, +9.8]
LLaMA-3.2-1B, Symbolic-P2+9.4+3.4 [+0.4, +6.2]+1.4 [−1.3, +3.9]

The point estimate of Δβ is positive in all six cases under both stronger judges, and its 95% interval excludes zero in 9 of the 12 cases.

Coherence and validity. We agree that trace coherence is a weak proxy, and that trace validity cannot be established for mathematical reasoning. We therefore study two settings: we use trace coherence for mathematical reasoning and trace validity for the mazes, where a program formally verifies every trace. The maze results give the clean test with trace validity.

Weakness 3: prose

We will revise the prose and correct the grammatical errors in the paper.

We hope that these changes address the reviewer's concerns, and we are happy to clarify any remaining point.

Reviewer_izvtOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that outcome-reward RLVR (e.g., GRPO) reinforces any chain of thought that raises the probability of a correct answer, not specifically correct reasoning. Theoretically, a CoT is reweighted only through its effect on solve probability, regardless of its validity. Empirically, on GSM8K and GSM-Symbolic with small models, the correct-answer rate given incoherent traces (β) rises after RL in every setting, contradicting Wen et al.'s prediction that it falls, though most accuracy gains still come through coherent traces.

Critical strengths
  1. The theoretical analysis provides a clear success-conditioned reweighting view of outcome-only RLVR.
  2. Separately reporting coherence, α, and β provides useful diagnostics beyond final-answer accuracy.
Critical weaknesses

1. The theory–GRPO connection needs stronger justification

Theorem 1 characterizes an ideal, per-prompt optimum in the vanishing-KL limit, whereas the experiments use finite-step GRPO with clipping, group-normalized advantages, and a nonzero KL penalty. Remark 1 does not sufficiently justify this approximation, and the per-prompt analysis does not explain the observed behavior on held-out or OOD problems. Directly testing the predicted trace-probability reweighting, accounting for finite KL regularization, would strengthen the connection.

More importantly, success-based reweighting can still implicitly favor valid reasoning if it improves answer correctness. Likewise, increasing β can coexist with increasing the correct-trace rate. Wen et al.’s core result assumes α > β; it does not require β to decline. The observed β increase therefore does not, by itself, contradict that conditional result. I suggest narrowing the abstract’s “contradicting” claim.

2. The robustness of Δβ to judge errors remains unclear

The judge validation uses 175 curated examples, with incoherent-class precision of 0.75 and κ = 0.62. Precision may deteriorate when incoherent traces are uncommon: illustratively, sensitivity 0.84 and specificity 0.79 would imply approximately 31% precision at 10% true incoherence prevalence. This is a sensitivity example, not an estimate of actual test-set precision. For K > 1, response-set labels also require separate validation.

Because β conditions on judge-assigned incoherence, false positives from coherent, correct responses could distort it. There is also no separate judge validation for base and RL outputs, leaving potential differences in classification errors unaddressed. I would like a human audit stratified by model, answer correctness, and judge label, together with uncertainty or sensitivity analysis for Δβ. A second judge would provide an additional check.

3. The scope and comparison with Wen et al. need qualification

The new experiments cover two small models, one GRPO recipe, and the GSM8K family, evaluated after 100 steps—approximately four epochs. This limits conclusions about larger models, longer CoTs, harder tasks, and later training. Different settings do not automatically invalidate a counterexample, but the relevant assumptions must be checked.

P1 relies on previously reported maze experiments. In the fully swapped setting, if valid traces are absent, α is not estimable, so the α > β premise is not demonstrated. For the normal-trace results, the original study describes validity evaluation within correct plans. The authors should clarify whether the quoted 89.0%→52.1% decline concerns P(valid trace | correct answer) or overall trace validity. A more direct comparison would include both trace classes, check α > β, and track the correct-trace rate, α, and β during training.

Questions for authors

Please address the clarification requests in Weaknesses 1–3 above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer izvt (1/2)

We thank the reviewer for the constructive review. We address the weaknesses below.

1. The "contradicting" claim (Weakness 1)

"Wen et al.'s core result assumes α > β; it does not require β to decline …"

We agree that the conditional result of Wen et al. assumes α > β and does not require β to decline, and the revised paper narrows the abstract accordingly. Our results show that the correct-trace rate and β rise together. Our measurement of β addresses two further statements in their paper.

The direction of β. Section 4 of Wen et al. ("Discussions on the effectiveness of GRPO") states that during RLVR training α increases and β decreases, so that the gap α − β widens. We measure α and β separately, and in the maze setting with verified traces the gap narrows:

Test setGap α − β, BaseGap α − β, RL
Wilson94.188.3
Kruskal93.684.5
DFS95.190.7

Implicit incentive or selection. Wen et al. treat the reinforcement of incorrect CoTs as an exception that arises when α > β does not hold, and describe it as unintentional (their Section 4, "Discussions on failure modes in GRPO"). We argue that it occurs even while α > β holds: the update is blind to the trace and up-weights every trace, valid or not, that makes a correct answer more likely. RLVR therefore selects based on the answer, and the prompt-augmentation view predicts the rise in both α and β that we report.

2. Connection between the theorem and GRPO (Weakness 1)

"Theorem 1 characterizes an ideal, per-prompt optimum in the vanishing-KL limit, …"

We have extended the analysis from the optimal policy to GRPO as implemented, with finite G, group-normalised advantages and the PPO-clipped surrogate. We view GRPO as an iterative procedure of self-training on the model's own verified responses, as in ReST-EM [1]. One GRPO step on a prompt x has four parts:

  1. Generate. Sample G responses from the current policy π_t(· | x).
  2. Verify. Score each answer with a binary reward r(y) and compute the group-relative advantage Â(y) = (r(y) − group mean) / group std. A group in which all answers are correct or all are wrong has an advantage of zero, Â = 0, for every response and is filtered out.
  3. Target. The groups that remain contain both correct and incorrect responses. Their advantages define the group-relative advantage-tilted policy π_new(y | x) ∝ π_t(y | x) · exp(η Â(y)). The advantage  depends on a response only through its answer: every correct response receives the same positive advantage and every incorrect response the same negative advantage, whatever its trace.
  4. Update as classification. With the importance-sampling ratio ρ = π_θ / π_t, the clipped surrogate of GRPO is exactly a weighted hinge loss [2]. At every token, the importance ratio acts as the score and the outcome as the label, with margin ε and weight |Â|. One mirror-descent step on this loss gives π_new of step 3 [2], and the gradient step of GRPO points in the direction that fits π_θ to π_new. Each update therefore moves π_θ towards π_new.

Thus, at finite G, a trace c is up-weighted in expectation when P_t(correct | x, c) > P_t(correct | x), whether the trace is valid or not, and the prompt-augmentation view applies.

"the per-prompt analysis does not explain the observed behavior on held-out or OOD problems"

GSM8K-Symbolic keeps the GSM8K problems and changes the names and the numerical values, and P1 and P2 add one and two clauses. These sets test the performance of the RL model under small changes of the problem, and on them α and β rise as on GSM8K.

"Directly testing the predicted trace-probability reweighting … would strengthen the connection."

Following this suggestion, we ran the direct test on the maze model with 200 prompts per test set and 16 base-model traces per prompt. For each fixed trace c we estimated its solve probability, P_base(Sol | x, c), and its log-probability under the base and the RL model. The change in the log-probability of a fixed trace after RL increases with its solve probability under the base model (rank correlation 0.66 to 0.70), and at equal solve probability a valid trace gains no more than an invalid one.

[1] Singh et al., arXiv:2312.06585. [2] Huang et al., "Neural PPO-Clip Attains Global Optimality: A Hinge Loss Perspective", arXiv:2110.13799.

AuthorsOpenReview ↗

Response to Reviewer izvt (2/2)

3. Robustness of Δβ to judge errors (Weakness 2)

"… no separate judge validation for base and RL outputs … a human audit … A second judge would provide an additional check."

Second judge. We added two stronger judges, gpt-oss-120b and DeepSeek V4.1 Flash. On the 175 human-labelled examples their precision on incoherent traces is 0.87 and 0.90, and their accuracy is 87.4% and 88.6%, compared with 0.75 and 81.1% for GPT-4o. The table gives Δβ under each judge, and its point estimate is positive in all six settings.

Model, datasetΔβ, GPT-4oΔβ, gpt-oss-120b [95% interval]Δβ, DeepSeek V4.1 Flash [95% interval]
Qwen2.5-0.5B, GSM8K+8.0+6.2 [+3.1, +8.9]+3.8 [+1.0, +6.4]
Qwen2.5-0.5B, Symbolic-P1+6.0+2.1 [+0.1, +3.9]+2.2 [+0.3, +3.9]
Qwen2.5-0.5B, Symbolic-P2+4.5+2.3 [+1.0, +3.7]+2.6 [+1.2, +3.9]
LLaMA-3.2-1B, GSM8K+16.2+3.9 [−2.2, +9.3]+1.0 [−4.7, +6.0]
LLaMA-3.2-1B, Symbolic-P1+14.0+4.3 [+0.1, +8.1]+6.1 [+2.0, +9.8]
LLaMA-3.2-1B, Symbolic-P2+9.4+3.4 [+0.4, +6.2]+1.4 [−1.3, +3.9]

Human audit, base and RL separately. In a blinded audit of 50 correct-answer GSM8K responses that both stronger judges flag (25 base, 25 RL), the annotator found incorrect reasoning in 18 of 25 base responses (72%) and in 8 of 25 RL responses (32%), although these responses are long for a human to check exhaustively step by step. The judges thus mark some correct reasoning as incorrect, more often after RL, so our main evidence is on verifiable traces (point 4).

For K > 1, every response is now labelled by the coherence of its trace and by its corresponding answer. We used gpt-oss-120b to label all responses at K = 4 and K = 16 again, and β rises under both judges, less for LLaMA-3.2-1B under the stronger judge:

Model, judgeΔβ, K = 1Δβ, K = 4Δβ, K = 16
Qwen2.5-0.5B, GPT-4o+8.0+9.7+8.4
Qwen2.5-0.5B, gpt-oss-120b+6.2+7.7+6.1
LLaMA-3.2-1B, GPT-4o+16.2+14.9+14.4
LLaMA-3.2-1B, gpt-oss-120b+3.9+4.9+5.2

4. Scope and the comparison with Wen et al. (Weakness 3)

"The new experiments cover two small models, one GRPO recipe, and the GSM8K family …"

We extended the scope with a larger model in the setting of Wen et al.: Qwen3.5-9B-Base, trained with GRPO on DAPO-Math and tested on MATH-500 and AIME 2024 + 2025. α and β both rise, and on MATH-500 the rise in β is the larger one:

BenchmarkAcc BaseAcc RLp_c Basep_c RLα Baseα RLβ Baseβ RLΔβ [95% interval]
MATH-50082.691.091.693.489.394.99.536.4+26.9 [+7.0, +45.7]
AIME 2024+202528.353.345.056.763.091.20.03.8+3.8 [0.0, +13.3]

"P1 relies on previously reported maze experiments. In the fully swapped setting …"

P1 now rests on our own maze runs, in three experimental settings: (a) Normal, updates on valid and invalid traces, (b) Normal, updates on valid traces only, and (c) Swapped, updates on invalid traces only. The table gives accuracy, p_c (the valid-trace rate), α and β before RL (Base) and after (RL).

Experimental settingTest setAcc BaseAcc RLp_c Basep_c RLα Baseα RLβ Baseβ RLΔβ [95% interval]
(a) Normal, valid and invalidWilson49.665.149.263.097.497.83.39.5+6.2 [+3.0, +9.4]
Kruskal49.767.249.364.597.297.23.612.7+9.1 [+5.6, +12.9]
DFS65.085.264.484.598.899.23.78.5+4.8 [+0.7, +9.3]
(b) Normal, valid onlyWilson50.066.549.665.597.498.53.45.8+2.4 [−0.3, +5.4]
Kruskal49.469.149.167.597.197.93.39.2+5.9 [+3.0, +9.2]
DFS65.183.764.583.598.899.23.75.3+1.6 [−1.8, +5.5]
(c) Swapped, invalid onlyWilson51.584.70.00.0n/an/a51.584.7+33.2 [+29.3, +36.9]
Kruskal52.188.60.00.0n/an/a52.188.6+36.5 [+32.8, +40.1]
DFS79.794.40.00.0n/an/a79.794.4+14.7 [+11.7, +17.8]

In (a), β rises on all three test sets, α is at its ceiling and stays flat, and the gain in accuracy (15 to 20 points) comes with a higher valid-trace rate, as selection on the answer predicts when α > β. In (b), β rises less than in (a) on all three test sets. This gives counterfactual evidence that incorrect reasoning is reinforced when invalid traces with a correct answer are in the update. In (c), the limiting case, the base model has no valid trace to select: validity stays at zero, α is undefined, and the whole gain in accuracy (15 to 37 points) is a rise in β. Thus the swapped model gives the cleanest evidence that a trace without semantics acts as an augmentation and is reinforced on the basis of the outcome during RLVR updates.

“… the quoted 89.0%→52.1% decline …”

The quoted 89.0% to 52.1% is P(valid trace | correct plan) in Table 5 of Beyond Semantics, for Searchformer-style traces. In our Wilson runs, the same quantity falls from 91.2 to 82.1.

We hope that these changes address the reviewer's concerns, and we are happy to clarify any remaining point.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Reviews, Responses and Revisions for the Area Chair

Dear Area Chair,

We thank you for handling our submission, and the reviewers for their constructive feedback.

Our paper shows that RLVR learns prompt augmentation conditioned on solution correctness, reinforcing a CoT whenever it raises the probability of a correct final answer, whether or not the CoT itself is correct. This view predicts that β, the probability of a correct answer given a trace without the property, rises under RLVR. We test it in maze pathfinding, where a program verifies every trace, and in mathematical reasoning, where an LLM judge labels every trace.

The reviewers appreciated:

  1. The theory as "a clear success-conditioned reweighting view of outcome-only RLVR" [izvt].
  2. The measurement of α and β as "an informative aspect of reasoning quality that aggregate accuracy can obscure" [xhig].
  3. The contribution to the debate as "a valuable empirical extension of the work done by Wen et al." [BTBK].

The reviewers raised concerns about the connection between Theorem 1 and GRPO (izvt, xhig), the scope of the claim relative to Wen et al. (izvt, xhig), the reliance on an LLM judge (izvt, BTBK, xhig) and the scale of the experiments (izvt, BTBK). We were not able to engage further with the reviewers during the discussion period, but we hope that our responses address their concerns, and we believe that the revisions have strengthened the paper. Below we summarize these revisions, which are highlighted in blue in the revised PDF, and how they address the concerns.

1. Connection between Theorem 1 and GRPO as implemented

Theorem 1 shows that, under its assumptions, the optimal policy is the base policy conditioned on success. We extended this to finite-step GRPO by viewing GRPO as iterative self-training on verified responses, as in ReST-EM, with the advantage-tilted exponential policy as the target. Each GRPO update tracks this target through a mirror-descent step with a hinge loss, namely the clipped surrogate, whose label at every token is the outcome (Remark 1 in Section 3) (Reviewers izvt, xhig).

2. Reframing of the claim relative to Wen et al.

We reframed the claim in the abstract and the conclusion around the prompt-augmentation view, which predicts the rise in β. The conditional result of Wen et al. assumes α > β and does not require β to decline, but their implicit-incentive view does not account for the rise in β. It treats the reinforcement of incorrect reasoning as an unintentional exception and expects the gap α − β to widen (Section 4.1). In our results β rises in all settings, and the gap narrows on every maze test set (Section 6.1) (Reviewers izvt, xhig).

3. Stronger evidence from verifiable traces, stronger judges and a larger model

  • Verifiable traces. We ran GRPO on maze models in three settings (Sections 4.2 and 6.1, Table 2). β rises with updates on valid and invalid traces, and rises less when the invalid traces with a correct answer are filtered out of the update, which is counterfactual evidence that these traces are reinforced. The swapped model, whose traces are all invalid, gains 15 to 37 points in accuracy entirely through β, which supports our view. The joint counts β⁺ and β⁻ confirm that RLVR reinforces invalid traces that end in a correct answer (Appendix D, Table 6) (Reviewers izvt, BTBK, xhig).
  • Stronger judges. We labelled every math response again under gpt-oss-120b and DeepSeek V4.1 Flash, whose precision on the human-labelled set is 0.87 and 0.90 against 0.75 for GPT-4o. β rises in all six settings under both judges (Table 1, Table 3 and Appendix C, Table 5) (Reviewers izvt, BTBK, xhig).
  • The setting of Wen et al. on a larger model. We trained Qwen3.5-9B-Base with GRPO on DAPO-Math and tested it on MATH-500 and AIME 2024 and 2025, as in Wen et al. α and β both rise (Sections 5 and 6.2, Table 3) (Reviewers izvt, BTBK).

4. Other revisions and clarifications

  • Fixed-trace test. On the maze model, the probability of a fixed trace changes with its solve probability, not with its validity (Appendix D, Table 7) (Reviewers izvt, xhig).
  • α and β at K > 1. α and β are now computed per response at K = 1, 4 and 16 (Section 5 and Table 4) (Reviewer xhig).
  • Prose. The prose is revised throughout (Reviewer BTBK).

We hope this summary helps your assessment. Thank you again for your time.

Best regards,

Authors of Submission 55631

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.