- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Progressive Reasoning Reinforcement Learning (ProRL), which reweights trajectory rewards using confidence estimated from the historical policy. The method is motivated by the idea that stable reasoning patterns should be consolidated, while emerging capabilities should be progressively refined. Experiments on mathematical reasoning benchmarks show clear improvements over GRPO.
Critical strengths
-
The motivation is interesting: stable reasoning patterns should be further consolidated, while emerging capabilities should be gradually refined through subsequent optimization.
-
The experimental results show substantial improvements over GRPO.
Critical weaknesses
-
The method is simple and the substantive contribution is limited. Simple yet effective algorithms can be valuable, but the paper does not sufficiently demonstrate why this method is effective. ProRL is essentially confidence-weighted RL, which is highly similar to entropy-aware RL and many existing related methods. The paper does not provide sufficiently deep discussion beyond this. Therefore, although the motivation is appealing, the proposed method does not fully match the claims and appears somewhat overclaimed. In particular, equating stable reasoning patterns and emerging capabilities with confidence seems superficial.
-
The experiments should be broadened. Besides GRPO, only one baseline appears to have been peer-reviewed at a major venue (ICLR 2026), while there are many well-known related methods. The evaluation is also limited to the Qwen2.5 series, which is not a reasoning model. More recent reasoning models such as Qwen3 may exhibit substantially different behavior.
-
Figure 2 does not demonstrate stronger exploration. The higher reward variance of ProRL is directly induced by reward reweighting. At step 0, the exploration behavior of GRPO and ProRL should be the same, while their reward variances are different. Therefore, reward variance is not equivalent to exploration ability.
Questions for authors
-
Are consolidating reliable reasoning patterns and refining emerging capabilities really equivalent to confidence? Why do the authors believe so?
-
There has already been substantial work on confidence-weighted RL and entropy-weighted RL. What is special about this work compared with them?
-
Can the authors provide more empirical evidence to support their claims?
7 comments
[Q3: Reasoning diversity examination(w3)]
Stop-gradient blocks differentiation through confidence estimation, but confidence weights still affect rewards and advantages and can indirectly reinforce preferred trajectories. We therefore empirically examine whether ProRL exhibits reasoning-diversity collapse during training.
Evaluation protocol.
We compare ProRL and GRPO at five training stages on 100 randomly sampled MATH-500 questions, generating 16 responses per question per method. To control for differences in the number of correct responses, we set
for each question and stage, where counts correct responses with non-empty reasoning. For the first three metrics below, we sample correct responses per method without replacement, repeat this procedure 50 times, and average the results. Questions with fewer than two eligible responses are excluded from these metrics.
Metrics.
-
Strategy Count: The number of distinct reasoning strategies among the sampled correct responses. Each response receives one primary label from 12 predefined strategy categories using Qwen3-4B-Instruct-2507.
-
Strategy Entropy:
where is the fraction of the responses assigned to strategy . This entropy is not normalized.
-
Correct Distinct-2: Bigram diversity among the sampled correct responses:
{\sum_i|B_i|},$$ where $B_i$ is the set of unique adjacent token bigrams within response $i$. -
All Distinct-2: The same bigram-diversity measure using all non-empty reasoning responses, including incorrect ones, without matching or resampling.
For both Distinct-2 metrics, tokenization covers words, numbers, LaTeX commands, and punctuation; bigrams never cross response boundaries. All results are averaged across questions, with resampling results first averaged within each question.
Table: Reasoning diversity at different training stages. Both methods share the same initial checkpoint and baseline evaluation.
| Stage | Method | Strategy Count | Strategy Entropy | Correct D-2 | All D-2 |
|---|---|---|---|---|---|
| Initial | Shared baseline | 1.3939 | 0.1991 | 0.2600 | 0.2073 |
| 25% | ProRL | 1.3889 | 0.2294 | 0.4466 | 0.5699 |
| 25% | GRPO | 1.3844 | 0.2127 | 0.3672 | 0.5599 |
| 50% | ProRL | 1.4241 | 0.2380 | 0.4920 | 0.7982 |
| 50% | GRPO | 1.3636 | 0.2280 | 0.3860 | 0.7792 |
| 75% | ProRL | 1.4522 | 0.2460 | 0.5010 | 0.7784 |
| 75% | GRPO | 1.3763 | 0.2298 | 0.3920 | 0.7683 |
| 100% | ProRL | 1.4371 | 0.2410 | 0.5010 | 0.6781 |
| 100% | GRPO | 1.3654 | 0.2310 | 0.3920 | 0.6601 |
ProRL scores higher on all four metrics at every post-initial stage. These observations provide evidence against reasoning-diversity collapse relative to GRPO.
[Q4: Are Reliable Reasoning Patterns and Emerging Capabilities Reflected in Confidence?(w1)]
We agree that confidence is not equivalent to reasoning maturity. ProRL uses length-normalized likelihood under the historical policy as a proxy for how familiar a trajectory is to that policy, rather than how correct or mature its reasoning is. External rewards provide correctness supervision, while likelihood-based weights adjust trajectories' relative contributions to optimization. Thus, "consolidating reliable reasoning" and "refining emerging capabilities" describe intended learning effects, not capability states measured by confidence. We will clarify this distinction.
To examine this interpretation, we track subsequent correction of initially incorrect responses, separating high-confidence cases by their ProRL advantage sign:
| Category | Samples | ProRL Corrected | GRPO Corrected |
|---|---|---|---|
| Low confidence | 459 | 46 (10.02%) | 34 (7.41%) |
| High confidence () | 334 | 56 (16.77%) | 21 (6.29%) |
| High confidence () | 401 | 44 (10.97%) | 28 (6.98%) |
ProRL achieves higher correction rates in all three categories, with the largest improvement over GRPO among high-confidence cases receiving positive advantages (16.77% versus 6.29%). These observations are consistent with our proposed refinement mechanism, but do not establish that positive advantages cause correction or that confidence measures reasoning maturity.
[Q5: How Does ProRL Differ from Existing Confidence- and Entropy-Weighted RL Methods?(w1)]
Several recent works have explored related directions, but they differ from ProRL in how these signals are constructed and incorporated into policy optimization.
Confidence-based RL. EM-RL-Sequence (Agarwal et al., 2025) directly uses sequence log-likelihood as a policy-gradient reward to minimize trajectory-level entropy. ConfClip (Zhang et al., 2025) uses length-normalized sequence probability to construct confidence-weighted correctness rewards and penalties. LENS (Feng et al., 2025) introduces confidence-weighted penalties specifically for incorrect responses, particularly in otherwise uninformative negative groups. ICPO (Wang et al., 2025/2026) constructs pairwise preference advantages from relative generation probabilities within a response group and combines them with external rewards. More recent methods, including ConSteer-RL (Miao et al., 2026) and ACE (Xu et al., 2026), further incorporate token-level or policy-relative confidence into reward shaping and asymmetric error correction.
Entropy-based RL. Related works, including The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models (Cui et al., 2025), Reasoning with Exploration: An Entropy Perspective (Cheng et al., 2025), and GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy (Tan et al., 2025), investigate policy-entropy collapse and introduce entropy-based mechanisms to regulate token-level or sequence-level optimization. Unlike these methods, ProRL uses the average log-likelihood of sampled trajectories as a policy-relative familiarity signal, rather than token-level policy entropy as an uncertainty or exploration bonus.
What distinguishes ProRL. Our contribution is not the general use of confidence or likelihood in RL, but a specific historical-policy-based, group-relative, multiplicative reward redistribution mechanism. Specifically, ProRL:
- Computes length-normalized trajectory likelihoods using the historical policy that generated the rollout batch, and converts them into group-normalized weights through a temperature-controlled Softmax.
- Multiplicatively applies these weights to external verifier rewards, rather than treating likelihood as an independent reward objective or adding a separate confidence bonus.
- Jointly reweights correct and incorrect trajectories within each sampled group, rather than restricting confidence adjustment to incorrect responses or using pairwise preference scores.
- Keeps historical-policy weights fixed across multiple updates of the same rollout batch, providing a consistent scoring signal during batch reuse.
We further validate these design choices through likelihood-only, additive-reward, inverse-confidence, subset-reweighting, and historical-versus-current-policy ablations. These results support the importance of the specific reward redistribution and historical-policy scoring designs.
We thank the reviewer for the constructive feedback and for recognizing ProRL's simplicity, consistent empirical improvements, and ablation results. We appreciate the important concerns regarding the method's novelty, the interpretation of trajectory confidence, its distinction from existing confidence- and entropy-aware RL methods, and the empirical evidence for exploration. We are glad to address these concerns and provide further clarification and experimental evidence.
[Q1: Broader Baselines and Evaluation on Recent Reasoning Models(w2)]
We additionally compare ProRL with GTPO and GRPO-S (Tan et al., 2026) and ConfClip (Zhang et al., 2025) using Qwen2.5-1.5B-Instruct trained on MATH-lighteval. The results are presented below:
Table : Comparison with additional baselines on Qwen2.5-1.5B-Instruct trained on MATH-lighteval (%).
The results show that our method achieves the best overall performance among the compared methods, reaching 27.58% average Pass@1 and 60.66% average Pass@16. Compared with the strongest baseline, GRPO-S for Pass@1 and GTPO for Pass@16, ProRL improves the two metrics by 0.63 and 4.86 percentage points, respectively. Notably, ProRL achieves the highest Pass@16 across all four benchmarks and the highest Pass@1 on MATH500, AMC23, and AIME24. Although its Pass@1 on OlympiadBench is lower than that of some baselines, its consistent gains in the overall averages demonstrate the effectiveness of confidence-aware reward reweighting. These results provide further evidence that ProRL remains competitive against diverse reward-design baselines.
We further compare our method with standard GRPO, GRPO-S, GTPO, ConfClip, and RL-ZVP which was the best-performing baseline (excluding GRPO and our method) in our Qwen2.5-1.5B-Instruct experiments. All methods are trained on Qwen3-1.7B using MATH-lighteval. The results are presented in the following table:
Table: Performance comparison on Qwen3-1.7B trained on MATH-lighteval (%).
Overall, these results demonstrate that ProRL remains effective on Qwen3-1.7B, achieving the best average Pass@1 and Pass@16 among all compared methods.
[Q2: Reward Variance Does Not Establish Stronger Exploration(w3)]
For this question, We agree that reward variance is not equivalent to exploration ability. Figure 2 illustrates that ProRL can provide learning signals in groups where GRPO's advantages vanish. When all trajectories receive the same reward, GRPO produces zero relative advantages. ProRL's confidence-based reweighting can yield nonzero advantages when the shared reward is nonzero and confidence weights differ, including all-incorrect groups receiving a format reward.
These additional learning signals potentially support exploration, but do not establish stronger exploration. At initialization, both methods have identical sampling behavior; their different reward variances arise from reweighting. We will revise the caption accordingly. We additionally examine whether ProRL exhibits reasoning-diversity collapse during training during Rebuttal. The specific answer is in the answer to the following question.