acceptodds
Withdrawn from ICLR 2027

Progressive Reasoning Reinforcement Learning via Confidence-Aware Reward Reweighting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In this paper, we propose Progressive Reasoning Reinforcement Learning (ProRL) to improve the reasoning capabilities of large language models. Existing RLVR methods, such as Group Relative Policy Optimization (GRPO), rely on binary outcome rewards that treat trajectories with the same outcome indiscriminately, overlooking their varying reliability and learning value. We argue that sampled trajectories exhibit different levels of mastery and should contribute unequally to policy optimization. Based on this insight, we instantiate ProRL with a likelihood-aware reward reweighting strategy that uses the likelihood under the historical policy, detached from gradient computation, as an estimate of trajectory confidence to dynamically adjust trajectory rewards. This enables progressive consolidation of reliable reasoning, refinement of emerging capabilities, and correction of dominant errors while preserving reasoning diversity. Experiments with Qwen2.5-1.5B on MATH-lighteval and DAPO-MATH-17K, together with scaling experiments using Qwen2.5-7B on MATH-lighteval, demonstrate that ProRL consistently outperforms strong RLVR baselines on both Pass@1 and Pass@16, improving reasoning accuracy while preserving diversity in solution exploration.

Primary area:
reinforcement learning
Keywords:
Large Language Models, Reasoning, Reinforcement Learning with Verifiable Rewards, Likelihood-Aware Reward Reweighting, Reasoning Diversity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RnJBOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Progressive Reasoning Reinforcement Learning (ProRL), which reweights trajectory rewards using confidence estimated from the historical policy. The method is motivated by the idea that stable reasoning patterns should be consolidated, while emerging capabilities should be progressively refined. Experiments on mathematical reasoning benchmarks show clear improvements over GRPO.

Critical strengths
  • The motivation is interesting: stable reasoning patterns should be further consolidated, while emerging capabilities should be gradually refined through subsequent optimization.

  • The experimental results show substantial improvements over GRPO.

Critical weaknesses
  • The method is simple and the substantive contribution is limited. Simple yet effective algorithms can be valuable, but the paper does not sufficiently demonstrate why this method is effective. ProRL is essentially confidence-weighted RL, which is highly similar to entropy-aware RL and many existing related methods. The paper does not provide sufficiently deep discussion beyond this. Therefore, although the motivation is appealing, the proposed method does not fully match the claims and appears somewhat overclaimed. In particular, equating stable reasoning patterns and emerging capabilities with confidence seems superficial.

  • The experiments should be broadened. Besides GRPO, only one baseline appears to have been peer-reviewed at a major venue (ICLR 2026), while there are many well-known related methods. The evaluation is also limited to the Qwen2.5 series, which is not a reasoning model. More recent reasoning models such as Qwen3 may exhibit substantially different behavior.

  • Figure 2 does not demonstrate stronger exploration. The higher reward variance of ProRL is directly induced by reward reweighting. At step 0, the exploration behavior of GRPO and ProRL should be the same, while their reward variances are different. Therefore, reward variance is not equivalent to exploration ability.

Questions for authors
  • Are consolidating reliable reasoning patterns and refining emerging capabilities really equivalent to confidence? Why do the authors believe so?

  • There has already been substantial work on confidence-weighted RL and entropy-weighted RL. What is special about this work compared with them?

  • Can the authors provide more empirical evidence to support their claims?

7 comments

We thank the reviewer for the constructive feedback and for recognizing ProRL's simplicity, consistent empirical improvements, and ablation results. We appreciate the important concerns regarding the method's novelty, the interpretation of trajectory confidence, its distinction from existing confidence- and entropy-aware RL methods, and the empirical evidence for exploration. We are glad to address these concerns and provide further clarification and experimental evidence.

[Q1: Broader Baselines and Evaluation on Recent Reasoning Models(w2)]

We additionally compare ProRL with GTPO and GRPO-S (Tan et al., 2026) and ConfClip (Zhang et al., 2025) using Qwen2.5-1.5B-Instruct trained on MATH-lighteval. The results are presented below:

Table : Comparison with additional baselines on Qwen2.5-1.5B-Instruct trained on MATH-lighteval (%).

MethodOlympiadBench P@1OlympiadBench P@16MATH500 P@1MATH500 P@16AMC23 P@1AMC23 P@16AIME24 P@1AIME24 P@16Avg. P@1Avg. P@16
GRPO19.8858.1654.4084.6017.5065.001.6716.6723.3656.11
ConfClip25.6757.7655.4081.7323.3365.831.6710.0026.5253.83
GRPO-S26.2658.4654.6081.6724.1767.502.7812.2226.9554.96
GTPO24.1859.3055.2081.9321.8864.171.1117.7825.5955.80
ProRL (Ours)20.4760.8358.8086.8024.3875.006.6720.0027.5860.66

The results show that our method achieves the best overall performance among the compared methods, reaching 27.58% average Pass@1 and 60.66% average Pass@16. Compared with the strongest baseline, GRPO-S for Pass@1 and GTPO for Pass@16, ProRL improves the two metrics by 0.63 and 4.86 percentage points, respectively. Notably, ProRL achieves the highest Pass@16 across all four benchmarks and the highest Pass@1 on MATH500, AMC23, and AIME24. Although its Pass@1 on OlympiadBench is lower than that of some baselines, its consistent gains in the overall averages demonstrate the effectiveness of confidence-aware reward reweighting. These results provide further evidence that ProRL remains competitive against diverse reward-design baselines.

We further compare our method with standard GRPO, GRPO-S, GTPO, ConfClip, and RL-ZVP which was the best-performing baseline (excluding GRPO and our method) in our Qwen2.5-1.5B-Instruct experiments. All methods are trained on Qwen3-1.7B using MATH-lighteval. The results are presented in the following table:

Table: Performance comparison on Qwen3-1.7B trained on MATH-lighteval (%).

MethodOlympiadBench P@1OlympiadBench P@16MATH500 P@1MATH500 P@16AMC23 P@1AMC23 P@16AIME24 P@1AIME24 P@16Avg. P@1Avg. P@16
GRPO32.9461.0364.6088.6743.7565.005.8322.2236.7859.23
RL-ZVP33.0961.5266.8089.6045.8365.006.1123.3337.9659.86
ConfClip33.2361.5765.8088.8045.4265.835.2822.2237.4359.61
GRPO-S33.9861.4766.0089.6047.5063.334.7220.0038.0558.60
GTPO33.0962.0266.2089.6046.4664.175.0023.3337.6959.78
ProRL (Ours)34.8762.0767.2091.0747.5065.836.3923.3338.9960.58

Overall, these results demonstrate that ProRL remains effective on Qwen3-1.7B, achieving the best average Pass@1 and Pass@16 among all compared methods.

[Q2: Reward Variance Does Not Establish Stronger Exploration(w3)]

For this question, We agree that reward variance is not equivalent to exploration ability. Figure 2 illustrates that ProRL can provide learning signals in groups where GRPO's advantages vanish. When all trajectories receive the same reward, GRPO produces zero relative advantages. ProRL's confidence-based reweighting can yield nonzero advantages when the shared reward is nonzero and confidence weights differ, including all-incorrect groups receiving a format reward.

These additional learning signals potentially support exploration, but do not establish stronger exploration. At initialization, both methods have identical sampling behavior; their different reward variances arise from reweighting. We will revise the caption accordingly. We additionally examine whether ProRL exhibits reasoning-diversity collapse during training during Rebuttal. The specific answer is in the answer to the following question.

[Q3: Reasoning diversity examination(w3)]

Stop-gradient blocks differentiation through confidence estimation, but confidence weights still affect rewards and advantages and can indirectly reinforce preferred trajectories. We therefore empirically examine whether ProRL exhibits reasoning-diversity collapse during training.

Evaluation protocol.

We compare ProRL and GRPO at five training stages on 100 randomly sampled MATH-500 questions, generating 16 responses per question per method. To control for differences in the number of correct responses, we set

K=min⁡(nProRL,nGRPO),K=\min(n_{\mathrm{ProRL}},n_{\mathrm{GRPO}}),

for each question and stage, where nn counts correct responses with non-empty reasoning. For the first three metrics below, we sample KK correct responses per method without replacement, repeat this procedure 50 times, and average the results. Questions with fewer than two eligible responses are excluded from these metrics.

Metrics.

  • Strategy Count: The number of distinct reasoning strategies among the KK sampled correct responses. Each response receives one primary label from 12 predefined strategy categories using Qwen3-4B-Instruct-2507.

  • Strategy Entropy:

    H=−∑spsln⁡ps,H=-\sum_s p_s\ln p_s,

    where psp_s is the fraction of the KK responses assigned to strategy ss. This entropy is not normalized.

  • Correct Distinct-2: Bigram diversity among the KK sampled correct responses:

    {\sum_i|B_i|},$$ where $B_i$ is the set of unique adjacent token bigrams within response $i$.
  • All Distinct-2: The same bigram-diversity measure using all non-empty reasoning responses, including incorrect ones, without matching KK or resampling.

For both Distinct-2 metrics, tokenization covers words, numbers, LaTeX commands, and punctuation; bigrams never cross response boundaries. All results are averaged across questions, with resampling results first averaged within each question.

Table: Reasoning diversity at different training stages. Both methods share the same initial checkpoint and baseline evaluation.

StageMethodStrategy CountStrategy EntropyCorrect D-2All D-2
InitialShared baseline1.39390.19910.26000.2073
25%ProRL1.38890.22940.44660.5699
25%GRPO1.38440.21270.36720.5599
50%ProRL1.42410.23800.49200.7982
50%GRPO1.36360.22800.38600.7792
75%ProRL1.45220.24600.50100.7784
75%GRPO1.37630.22980.39200.7683
100%ProRL1.43710.24100.50100.6781
100%GRPO1.36540.23100.39200.6601

ProRL scores higher on all four metrics at every post-initial stage. These observations provide evidence against reasoning-diversity collapse relative to GRPO.

[Q4: Are Reliable Reasoning Patterns and Emerging Capabilities Reflected in Confidence?(w1)]

We agree that confidence is not equivalent to reasoning maturity. ProRL uses length-normalized likelihood under the historical policy as a proxy for how familiar a trajectory is to that policy, rather than how correct or mature its reasoning is. External rewards provide correctness supervision, while likelihood-based weights adjust trajectories' relative contributions to optimization. Thus, "consolidating reliable reasoning" and "refining emerging capabilities" describe intended learning effects, not capability states measured by confidence. We will clarify this distinction.

To examine this interpretation, we track subsequent correction of initially incorrect responses, separating high-confidence cases by their ProRL advantage sign:

CategorySamplesProRL CorrectedGRPO Corrected
Low confidence45946 (10.02%)34 (7.41%)
High confidence (A>0A>0)33456 (16.77%)21 (6.29%)
High confidence (A≤0A\leq 0)40144 (10.97%)28 (6.98%)

ProRL achieves higher correction rates in all three categories, with the largest improvement over GRPO among high-confidence cases receiving positive advantages (16.77% versus 6.29%). These observations are consistent with our proposed refinement mechanism, but do not establish that positive advantages cause correction or that confidence measures reasoning maturity.

[Q5: How Does ProRL Differ from Existing Confidence- and Entropy-Weighted RL Methods?(w1)]

Several recent works have explored related directions, but they differ from ProRL in how these signals are constructed and incorporated into policy optimization.

Confidence-based RL. EM-RL-Sequence (Agarwal et al., 2025) directly uses sequence log-likelihood as a policy-gradient reward to minimize trajectory-level entropy. ConfClip (Zhang et al., 2025) uses length-normalized sequence probability to construct confidence-weighted correctness rewards and penalties. LENS (Feng et al., 2025) introduces confidence-weighted penalties specifically for incorrect responses, particularly in otherwise uninformative negative groups. ICPO (Wang et al., 2025/2026) constructs pairwise preference advantages from relative generation probabilities within a response group and combines them with external rewards. More recent methods, including ConSteer-RL (Miao et al., 2026) and ACE (Xu et al., 2026), further incorporate token-level or policy-relative confidence into reward shaping and asymmetric error correction.

Entropy-based RL. Related works, including The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models (Cui et al., 2025), Reasoning with Exploration: An Entropy Perspective (Cheng et al., 2025), and GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy (Tan et al., 2025), investigate policy-entropy collapse and introduce entropy-based mechanisms to regulate token-level or sequence-level optimization. Unlike these methods, ProRL uses the average log-likelihood of sampled trajectories as a policy-relative familiarity signal, rather than token-level policy entropy as an uncertainty or exploration bonus.

What distinguishes ProRL. Our contribution is not the general use of confidence or likelihood in RL, but a specific historical-policy-based, group-relative, multiplicative reward redistribution mechanism. Specifically, ProRL:

  1. Computes length-normalized trajectory likelihoods using the historical policy that generated the rollout batch, and converts them into group-normalized weights through a temperature-controlled Softmax.
  2. Multiplicatively applies these weights to external verifier rewards, rather than treating likelihood as an independent reward objective or adding a separate confidence bonus.
  3. Jointly reweights correct and incorrect trajectories within each sampled group, rather than restricting confidence adjustment to incorrect responses or using pairwise preference scores.
  4. Keeps historical-policy weights fixed across multiple updates of the same rollout batch, providing a consistent scoring signal during batch reuse.

We further validate these design choices through likelihood-only, additive-reward, inverse-confidence, subset-reweighting, and historical-versus-current-policy ablations. These results support the importance of the specific reward redistribution and historical-policy scoring designs.

Reviewer_EvHmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to distinguish reasoning trajectories' quality beyond outcome-only rewards in reinforcement learning.

Orgrigal GRPO: outcome-only rewards then boardcast to each token in traj. BUT, can not verify which traj. is good.

propose ProRL, generate a reweighting score w, reweight the R, then continue perform original GRPO.

Results on several benchmark on pass@1&16, outperforms the baseline.

Critical strengths

The method is straightforward, and its motivation is easy to understand.

The improvements are consistent.

The ablation studies provide some support for the method.

Critical weaknesses
  1. assigning positive advantages to high-confidence incorrect trajectories promotes subsequent error correction. why? needs stronger theoretical support to explain

  2. authors said "This design avoids a self-reinforcing feedback loop that could excessively amplify already preferred behaviors and reduce reasoning diversity." why? I dont think stopping gradients can eliminate the influence of weights on policy updates through rewards and relative advantages. High-confidence trajectories may still receive more good updates and become increasingly dominant in sampling.

Questions for authors

see above

2 comments

We thank the reviewer for the constructive feedback and for recognizing the simplicity, consistent empirical improvements, and ablation results of ProRL. We also appreciate the important concerns regarding the correction mechanism and the potential self-reinforcing effects of confidence-based reward reweighting. We are glad to clarify these aspects.

[Q1: Why Do Positive Advantages for High-Confidence Errors Promote Correction?]

Although high-confidence incorrect trajectories might receive positive advantages in a given optimization step, they are not treated as final solutions. Instead, preserving their optimization signals allows the model to further explore similar underlying reasoning patterns and discover correct variants in subsequent generations. Once these corrected trajectories emerge, their reinforcement gradually replaces the erroneous behaviors. Examples in training progress support this hypothesis in appendix J.

Correction evidence. Also, we track initially incorrect response instances according to whether high-confidence errors receive positive (A > 0) or non-positive (A≤0A\leq0) advantages.

CategorySamplesProRL Corrected / RateGRPO Corrected / Rate
Low Confidence45946 / 10.02%34 / 7.41%
High Confidence (A > 0)33456 / 16.77%21 / 6.29%
High Confidence (A ≤ 0)40144 / 10.97%28 / 6.98%
Overall1194146 / 12.23%83 / 6.95%

ProRL's A > 0 correction rate exceeds A≤0A\leq0 by 5.80 percentage points and GRPO's A > 0 rate by 10.48 percentage points. These observational results support our hypothesis but do not establish causality.

Confidence-direction ablation. We also compare ProRL with a inverse-confidence reweighting strategy that continues to prioritize high-confidence trajectories for correct responses, while prioritizing lower-confidence ones for incorrect responses.

MethodPass@1Pass@16
Inverse-confidence20.60%56.21%
ProRL (Ours)27.58%60.66%

ProRL gains 6.98 and 4.45 percentage points in Pass@1 and Pass@16, respectively. This supports the importance of weighting direction, though it does not independently prove the correction mechanism.

[Q2: Does Stop-Gradient Prevent Self-Reinforcement and Preserve Reasoning Diversity?]

We agree with the reviewer. Stop-gradient blocks differentiation through confidence estimation, but confidence weights still affect rewards and advantages and can indirectly reinforce preferred trajectories. We therefore empirically examine whether ProRL exhibits reasoning-diversity collapse during training.

Evaluation protocol. We compare ProRL and GRPO at five training stages on 100 randomly sampled MATH-500 questions, generating 16 responses per question per method. To control for differences in the number of correct responses, we set K=min⁡(nProRL,nGRPO)K=\min(n_{\mathrm{ProRL}},n_{\mathrm{GRPO}}) for each question and stage, where nn counts correct responses with non-empty reasoning. For the first three metrics below, we sample KK correct responses per method without replacement, repeat this procedure 50 times, and average the results. Questions with K<2K<2 are excluded from these metrics.

Metrics.

  • Strategy Count: The number of distinct reasoning strategies among the KK sampled correct responses. Each response receives one primary label from 12 predefined strategy categories using Qwen3-4B-Instruct-2507.

  • Strategy Entropy: H=−∑spsln⁡ps,H=-\sum_s p_s\ln p_s, where psp_s is the fraction of the KK responses assigned to strategy ss. This entropy is not normalized.

  • Correct Distinct-2: Bigram diversity among the KK sampled correct responses: D2=∣⋃iBi∣∑i∣Bi∣,D_2=\frac{\left|\bigcup_i B_i\right|}{\sum_i|B_i|}, where BiB_i is the set of unique adjacent token bigrams within response ii.

  • All Distinct-2: The same bigram-diversity measure using all non-empty reasoning responses, including incorrect ones, without matching KK or resampling.

For both Distinct-2 metrics, tokenization covers words, numbers, LaTeX commands, and punctuation; bigrams never cross response boundaries. All results are averaged across questions, with resampling results first averaged within each question.

Table: Reasoning diversity at different training stages. Both methods share the same initial checkpoint and baseline evaluation.

StageMethodStrategy CountStrategy EntropyCorrect D-2All D-2
InitialShared baseline1.39390.19910.26000.2073
25%ProRL1.38890.22940.44660.5699
25%GRPO1.38440.21270.36720.5599
50%ProRL1.42410.23800.49200.7982
50%GRPO1.36360.22800.38600.7792
75%ProRL1.45220.24600.50100.7784
75%GRPO1.37630.22980.39200.7683
100%ProRL1.43710.24100.50100.6781
100%GRPO1.36540.23100.39200.6601

The data in the table shows, ProRL scores higher on all four metrics at every post-initial stage. These observations provide evidence against reasoning-diversity collapse relative to GRPO.

Reviewer_BkqxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how to use trajectory confidence to refine outcome rewards in RLVR, so that reasoning trajectories with the same outcome can be told apart and so that GRPO still receives a learning signal when all rewards in a group are identical. The proposed method, ProRL, estimates trajectory confidence as the length-normalized log-likelihood under the old policy, applies a within-group softmax, uses the result to multiplicatively reweight the original reward, and then performs a standard GRPO update. The stated goals are to consolidate reliable reasoning, refine abilities that are not yet stable, and correct high-confidence errors while preserving solution diversity. Experiments train Qwen2.5-1.5B-Instruct on MATH-lighteval and DAPO-MATH-17K, and Qwen2.5-7B-Instruct on MATH-lighteval. In all three settings, ProRL reports higher macro-averaged Pass@1 and Pass@16 than the compared methods across four math benchmarks. The paper also includes a CoT quality analysis and ablations on how confidence is computed, the softmax temperature, and which responses are reweighted.

Critical strengths
  1. A simple design that targets the lack of reward resolution. Outcome rewards cannot distinguish the reasoning quality of trajectories that reach the same result, and identical rewards within a group leave GRPO with no task advantage signal. ProRL refines the reward using trajectory likelihood under the old policy. It needs no process reward model or human step-level labels, and it changes the existing GRPO pipeline very little (Eq. 3–7). This makes it practical to adopt.

  2. Consistent average gains across several settings. In the three settings (1.5B/MATH-lighteval, 7B/MATH-lighteval, 1.5B/DAPO-MATH-17K), ProRL improves average Pass@1 over GRPO by 4.22, 2.25, and 1.45 points, and average Pass@16 by 4.55, 1.37, and 2.48 points (Tables 1–3). Results across model sizes and training sets give initial support for the method's empirical effect, and Figure 1 and Tables 4–6 make an effort to analyze its behavior further.

Critical weaknesses
  1. The core explanation, "progressive correction" while preserving diversity, is not well supported. By Eq. 5–6, when all rewards in a group are equal and positive and the confidence weights differ, the advantage depends only on the relative differences between weights. So in groups where every response is wrong but receives the same positive format reward, some high-likelihood wrong responses still get a positive advantage. This restores a non-zero training signal, but the paper does not show that reinforcing wrong trajectories helps the model correct them later. Figure 1d measures how often regenerating from an uncertain token fixes the answer, which is different from a positive update on a complete wrong trajectory during training. The overall CoT quality improvement in Figure 1c also cannot separate this correction story from other possible sources of the gain (§3.3, Figure 1).

    In addition, §3.2 and §4.1 present scoring with the old policy and stopping the gradient as the reason the method avoids self-reinforcing high-likelihood trajectories and keeps exploring. But the stop-gradient only blocks backpropagation through the confidence computation itself. Confidence still enters the policy gradient through the advantage weights, and the old policy is itself updated as training proceeds. The design therefore gives no guarantee against self-reinforcement. Higher Pass@16 shows that more problems are solved within a fixed sampling budget, but on its own it does not show that solution diversity is preserved, and the reward standard deviation in Figure 2 is not a direct measure of exploration. These issues bear on the paper's central explanation of where the performance gains come from. The paper needs evidence drawn from the actual training updates, or it should narrow its claims about correction and diversity accordingly.

  2. Unexplained inconsistencies between the stated evaluation setup and several main-table numbers make it hard to judge how reliable the performance advantage is. §4 describes results as the average of three runs and defines Pass@16 as the fraction of problems with at least one correct answer among 16 samples. With the standard 30-problem AIME24, a three-run average must be an integer multiple of 100/(30×3)100/(30\times3) percentage points. The values 28.33 and 31.67 for Base Model and GRPO in Table 2, and 28.33 for GRPO-VPS in Table 3, cannot be obtained this way. Likewise, if AMC23 has 40 problems, the Pass@1 of 20.67 reported for RL-ZVP in Table 1 cannot come from any correct count under four samples per problem and three runs. These discrepancies may be typos, a different test subset, or a different estimator, but the current description does not allow the reader to tell which.

    The paper also states that all experiments use seed 42, without saying whether the three runs repeat training, evaluation sampling, or both, and it reports no per-run results or uncertainty. Some of the gaps are small; for example, the 7B model's MATH500 Pass@1 is 75.2 versus 77.0. These reporting details therefore directly affect whether the lead can be called consistent. The authors should state the actual number of test problems, what is repeated across runs, and how each metric is computed, check the numbers listed above, and provide the existing per-run results or per-problem outputs for the key comparisons. Until this is clarified, the lead in the averaged numbers cannot be considered reproducible.

Questions for authors
  1. Evaluation setup and raw results (W2). Which sources of randomness in training or evaluation differ across the three runs? Please give the actual number of problems in each test set and how each metric is computed, provide per-run results with the corresponding correct counts, and explain the four values listed in W2. If corrections are needed, do the ranking of the main methods and the reported gaps change? Please also explain how the validation set, the best checkpoint, and the temperature τ were chosen. This information directly affects my assessment of how reliable the performance advantage is.

  2. Evidence for the correction mechanism (W1). Do you have training-time evidence that high-confidence wrong responses receiving a positive advantage lead to correct variants appearing later? For example, does the later accuracy on those problems, or a breakdown of how much each type of group contributes, support an explanation that goes beyond a general preference for high-likelihood outputs? If no such evidence is available, could the paper present Figure 1d as a correlation observed under regeneration, and tone down the argument that stopping the gradient preserves diversity? This affects how I evaluate the paper's mechanistic contribution.

  3. Source of the gains and related work (W1). EM-RL-sequence (Agarwal et al., 2025, §3.2) already uses sequence log-likelihood as a policy-gradient reward. ProRL's length normalization, exponential transform, and multiplication with the external reward make it different, but the current experiments do not show what these choices add beyond a plain likelihood reward. Do you have results under the same training setup for a likelihood-only reward or a simple mixed reward that would separate the two? Clarifying this would help establish the contribution of multiplicative reweighting on its own.

  4. Implementation of key ablations. In Table 4, when is the current-policy likelihood recomputed, and how many updates are made per batch of trajectories? In Table 6, how do the two subset variants keep the total reward unchanged, and how do their exact formulas differ from the full method? These details determine whether the ablations support the "stable scoring" and "joint reweighting" explanations, respectively. If the appendix already covers this, please point to the relevant section.

3 comments

We thank the reviewer for the constructive feedback and for recognizing the simplicity of ProRL and its consistent empirical gains. We also appreciate the insightful questions regarding the correction mechanism, reasoning diversity, evaluation reproducibility, and distinctions from related work. We are pleased to address these questions.

[Q1: Evaluation setup and raw results (W2)]

Each method was trained once using \texttt{seed=42}. We conducted three independent evaluations on the same selected checkpoint, rather than three independent training runs.

For Pass@1, OlympiadBench (674 problems) and MATH500 (500 problems) use greedy decoding, yielding identical predictions across the three evaluations. AMC23 (40 problems) and AIME24 (30 problems) use stochastic sampling, with four responses per problem, resulting in 160 and 120 responses per evaluation, respectively. For Pass@16, all four benchmarks use stochastic sampling with 16 responses per problem. A problem is considered solved if at least one of the 16 responses is correct. We report the mean across the three evaluations, reflecting inference-time sampling variability rather than training variability.

After rechecking and correction, the revised Tables 1, 2, 3 are as follows.

Table 1: Main results of Qwen2.5-1.5B-Instruct fine-tuned on MATH-lighteval (%).

MethodOlympiadBench P@1OlympiadBench P@16MATH500 P@1MATH500 P@16AMC23 P@1AMC23 P@16AIME24 P@1AIME24 P@16Avg. P@1Avg. P@16
Base Model17.5155.1933.0083.2017.5060.003.3310.0017.8452.10
GRPO19.8858.1654.4084.6017.5065.001.6716.6723.3656.11
GRPO-VPS19.1458.3141.4085.6018.1367.503.3316.6720.5057.02
RLCR19.4458.0140.0075.6015.0065.003.3310.0019.4452.15
RL-ZVP19.5859.3554.4084.4020.6370.005.0016.6724.9057.61
ProRL (Ours)20.4760.8358.8086.8024.3875.006.6720.0027.5860.66

Table 2: Main results of Qwen2.5-7B-Instruct fine-tuned on MATH-lighteval (%).

MethodOlympiadBench P@1OlympiadBench P@16MATH500 P@1MATH500 P@16AMC23 P@1AMC23 P@16AIME24 P@1AIME24 P@16Avg. P@1Avg. P@16
Base Model37.9873.8957.4093.0026.2582.508.3326.6732.4969.02
GRPO37.8373.5975.2093.0045.6380.009.1731.1141.9669.43
GRPO-VPS37.6973.2973.8093.6048.7585.008.3323.3342.1468.81
RLCR37.0973.0075.2092.8048.1380.008.3323.3342.1967.28
RL-ZVP38.1373.7476.2093.4047.5080.008.3323.3342.5467.62
ProRL (Ours)40.6574.3377.0093.6050.0082.509.1733.3344.2170.94

Table 3: Main results of Qwen2.5-1.5B-Instruct fine-tuned on DAPO-MATH-17K (%).

MethodOlympiadBench P@1OlympiadBench P@16MATH500 P@1MATH500 P@16AMC23 P@1AMC23 P@16AIME24 P@1AIME24 P@16Avg. P@1Avg. P@16
Base Model17.5155.1933.0083.2017.5060.003.3310.0017.8452.10
GRPO19.7359.7948.8087.0030.0065.003.3326.6725.4759.62
GRPO-VPS20.4760.0949.4085.6030.0070.001.6727.7825.3960.87
RLCR19.2957.8648.0087.0027.5067.502.5023.3324.3258.92
RL-ZVP21.5159.7949.0087.4027.5070.003.3326.6725.3460.97
ProRL (Ours)22.2660.8349.6088.4032.5072.503.3326.6726.9262.10

The detailed results of each evaluation run are provided in the three tables in the Appendix H.

Moreover,refer to the previous work AALC (Li et al., 2025), we use a shared 95%/5% training/validation split while preserving the original problem order. Checkpoints are evaluated every 100 optimization steps and selected based on validation Pass@1.

The temperature parameter τ\tau is selected from the candidate set {0.5,0.7,0.8,1.0}\{0.5, 0.7, 0.8, 1.0\} using validation data only, with 0.7 chosen as the final value. In addition, we also conducted ablation experiments, the corresponding temperature ablation results are reported in Table 5 in our paper. Neither checkpoint selection nor temperature selection uses test-set results.

[Q2: Evidence for the correction mechanism (W1)] {\bf Statistical evidence.} We categorize samples into a positive-advantage group (A > 0) and a non-positive-advantage group (A≤0A\leq0) based on whether the high-confidence incorrect trajectories among the initial wrong responses receive positive advantage, and subsequently track the error correction behavior of ProRL and GRPO during later training.

CategorySamplesProRL Corrected / RateGRPO Corrected / Rate
Low Confidence45946 / 10.02%34 / 7.41%
High Confidence (A>0A>0)33456 / 16.77%21 / 6.29%
High Confidence (A≤0A\leq 0)40144 / 10.97%28 / 6.98%
Overall1194146 / 12.23%83 / 6.95%

In ProRL, the error correction rate is 16.77% for the A > 0 group and 10.97% for the A≤0A\leq0 group, yielding a difference of 5.80 percentage points. Within the A > 0 group, ProRL achieves an error correction rate that is 10.48 percentage points higher than that of GRPO. Overall, the total error correction rates for ProRL and GRPO are 12.23% and 6.95%, respectively.

\noindent{\bf Examples evidence.} Furthermore, we randomly sample 100 prompts for which all rollouts were incorrect before 50% of training progress and evaluate the final checkpoint using Pass@16. We observe that 32 prompts are successfully recovered, corresponding to a recovery rate of 32%. Among these 32 recovered prompts, 25 (78.13%) exhibit the disappearance of previously observed high-confidence incorrect trajectories, followed by the emergence of correct responses.

These observations provide additional evidence of error recovery without establishing a direct causal relationship. The quantitative analysis and representative recovery cases are presented in Appendix I, while Appendix J provides complementary examples of initially incorrect trajectories receiving positive advantages followed by correct reasoning at later training stages.

[Q3: Source of the gains and related work (W1)]

Following EM-RL-Sequence (Agarwal et al., 2025, Sec. 3.2), we compare likelihood-only, negative-likelihood, and additive rewards:

Likelihood-only reward: RiL=∑t=1Tilog⁡πθ(oi,t∣q,oi,<t).R_i^{L}=\sum_{t=1}^{T_i}\log \pi_{\theta}(o_{i,t}\mid q,o_{i,<t}).

Negative likelihood reward:

Ri−L=−∑t=1Tilog⁡πθ(oi,t∣q,oi,<t).R_i^{-L}=-\sum_{t=1}^{T_i}\log \pi_{\theta}(o_{i,t}\mid q,o_{i,<t}).

Additive hybrid reward: ℓiθˉ=1Ti∑t=1Tilog⁡πθ(oi,t∣q,oi,<t)\bar{\ell_i^\theta} = \frac{1}{T_i}\sum_{t=1}^{T_i}\log \pi_{\theta}(o_{i,t}\mid q,o_{i,<t})

Riadd=Riext+λℓiθˉR_i^{\mathrm{add}}=R_i^{\mathrm{ext}}+\lambda\bar{\ell_i^\theta} The results are summarized below:

MethodAvg. P@1Avg. P@16
Additive hybrid23.45%57.34%
Likelihood-only22.83%53.09%
Negative likelihood20.63%52.96%
ProRL (Ours)27.58%60.66%

As the table, ProRL improves over the additive hybrid by 4.13 and 3.32 percentage points, and over likelihood-only by 4.75 and 7.57 percentage points in Pass@1 and Pass@16, respectively.

[Q4: Implementation of key ablations]

About Table 4, each batch undergoes two parameter updates. First, the behavior policy πθold\pi_{\theta_{\mathrm{old}}} samples trajectories and computes their rewards and likelihoods. We then perform the first parameter update, obtaining updated parameters θ1\theta_1. Using πθ1\pi_{\theta_1}, we recompute the likelihoods of the same trajectories sampled by πθold\pi_{\theta_{\mathrm{old}}}, obtaining updated likelihood estimates. We then perform the second update, yielding the final parameters θ2\theta_2.

In Table 4, confidence scores are computed separately using the initial likelihoods (before the first update) and the recomputed likelihoods (after the first update), respectively.

About Table 6, let C\mathcal{C} and I\mathcal{I} denote the correct and incorrect trajectory subsets, respectively, and let RiR_i denote the raw external reward. Full ProRL uses: Rifull=wiRi.R_i^{\mathrm{full}}=w_iR_i.

For the Correct-only variant:

{\sum_{j\in\mathcal{C}}w_jR_j} \sum_{j\in\mathcal{C}}R_j, \quad i\in\mathcal{C},$$ $$R_i^{\mathcal{C}}=R_i, \quad i\in\mathcal{I}.$$ For the Incorrect-only variant: $$R_i^{\mathcal{I}}=\frac{w_iR_i} {\sum_{j\in\mathcal{I}}w_jR_j} \sum_{j\in\mathcal{I}}R_j, \quad i\in\mathcal{I},$$ $$R_i^{\mathcal{I}}=R_i, \quad i\in\mathcal{C}.$$ Each variant therefore preserves the original total reward of the reweighted subset, leaves the other subset unchanged, and preserves the total reward over the full group. Group-relative advantages are then recomputed over all trajectories. In our evaluated batches, incorrect responses could receive positive format rewards despite receiving zero correctness rewards; the Incorrect-only subset consequently had a nonzero normalization denominator. At the same time, we have set that if there is really an Incorrect-only subset with a total reward of 0, we will skip it directly.
Reviewer_BkqxOpenReview ↗

Reply to author's rebuttal

Thank you for the clarification. Your response helped me better understand the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.