- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper identifies a length-dependent loss of discrimination in reference-answer probability rewards, termed Posterior Concentration Phenomenon (PCP), and proposes RLCPR: uncertainty-aware data sampling combined with concentration-gated length regularization. On Qwen3-4B-Base, RLCPR outperforms same-backbone RLPR on six of seven benchmarks, improving average accuracy while reducing rollout-token consumption.
Critical strengths
- The paper raises a practically relevant question about the reliability of probability-based rewards.
- The method is simple to implement and shows empirical gains across tasks.
- The method reduces rollout-token consumption, with gating and scheduling ablations supporting its design choices.
Critical weaknesses
-
Relevant verifier-free baselines are missing. JEPO[1] and NOVER[2] are discussed in the related work but not evaluated, while DARL[3], an RLPR-based method using the same training corpus and overlapping benchmarks, is also omitted, leaving the comparative evaluation incomplete.
-
Only one RL baseline provides a same-backbone comparison. RLCPR is trained on Qwen3-4B-Base, but Table 1 reports RLVR, General Reasoner, and VeriFree using borrowed Qwen2.5-7B results, leaving only RLPR as a directly comparable RL baseline and preventing a controlled assessment against the other methods.
-
The ablations do not isolate the benefit of uncertainty-based sampling. Table 2 lacks a random-filtering baseline matched to UDS’s retention rate and rollout-token budget, making it difficult to distinguish uncertainty-based selection from generic subsampling effects.
-
The theoretical analysis relies on an unestablished assumption. Proposition 4.1 requires convergence of cross-trajectory residual differences, which is not implied by the cited length-normalized likelihood results; Appendix G acknowledges this gap, limiting the theorem’s explanation of the observed PCP.
References
[1] Beyond Verifiable Rewards: Scaling Reinforcement Learning for Language Models to Unverifiable Data.
[2] NOVER: Incentive Training for Language Models via Verifier-Free Reinforcement Learning.
[3] DARL: Encouraging Diverse Answers for General Reasoning without Verifiers.
Questions for authors
See the Critical Weaknesses above.
9 comments
Individual Response to Reviewer ywQd (Cont'd)
(W4) Residual assumption in Proposition 4.1. We agree that Prop. 4.1 requires the cross-trajectory residual differences to converge, and that this is not implied by the cited length-normalized likelihood results. We state it explicitly as a sufficient condition and discuss it in Appendix G. We would like to clarify that our key theoretical insight is a reduction: posterior-reward differences between long traces reduce to approximation errors alone. Eq. (11) decomposes the log-ratio of two posteriors into exactly two terms:
where the first term is (i) the deterministic, length-induced entropy shift, and the second term$\left(\rho_{n+s}(z^1) - \rho_n(z^2)\right)$ is (ii) the trace-specific approximation errors.
- Eliminating the deterministic term. Term (i) is the only component that depends systematically on reasoning length. Lemma 4.1 proves that it vanishes outside a zero-density set of lengths, under only the branching-factor condition, which is supported empirically by [1]. Hence, differences in length alone cannot systematically separate posterior rewards.
- What remains are approximation errors. Once term (i) is eliminated, any remaining reward difference between two long traces comes only from term (ii): the trace-specific residuals of the log-likelihood concentration approximation (Eq. (6)). In other words, the posterior reward can distinguish two long traces only through how far each trace's log-likelihood deviates from its expected entropy, not through any length-dependent structure.
This reduction is what the theory contributes. It identifies precisely where the discriminative signal of posterior rewards must come from in the long-trace regime, and the residual condition is the single additional assumption needed to conclude Prop. 4.1.
Establishing the convergence of these residuals analytically is difficult. Each residual is itself the approximation error between a trace's realized log-likelihood and its expected entropy, and its distribution depends on the specific model and prompt. Instead, we support this assumption empirically. Fig. 2b and 2c show that both the overall and the within-group variance of posterior rewards contract as reasoning length grows, which is the observable consequence of vanishing residual differences. We will state the scope of this condition more clearly in the revised manuscript.
[1] LLM Probability Concentration: How Alignment Shrinks the Generative Horizon (TMLR 2026)
Thank you for the reply. I still think the experiments are incomplete and rather confusing. The added DARL results on Qwen2.5-7B are better than RLCPR’s, which weakens the claimed advantage of RLCPR. The authors say the two methods are complementary, so I’d like to see DARL + RLCPR. If it improves on DARL, that would also be a useful result. Also, both DARL(https://github.com/huangchongxuan/DARL) and NOVER(https://github.com/thinkwee/NOVER) have public code, so I think the authors should run them on both backbones and report the results. Overall, the paper still needs a lot more experimental work before it can be accepted, so I’ll keep my weak reject.
Individual Response to Reviewer ywQd
We thank Reviewer ywQd for the constructive comments. Following our general response, we address each concern below.
(W1, W2) Baseline coverage and same-backbone comparisons. RLPR is our primary baseline because it isolates the effect of our contribution. RLCPR adopts exactly the same training settings as RLPR (data, optimizer, hyperparameters, and training recipe) and differs only in replacing RLPR's post-generation reward filtering with UDS and CR. This controlled comparison directly measures the benefit of accounting for PCP, and it also shows that RLPR's reward filtering is computationally inefficient. Broader baselines serve a different purpose, namely establishing overall competitiveness, and we have expanded them as follows:
(W3) Isolating the benefit of uncertainty-based sampling. The "+CR" variant in Table 2 already serves as the random-filtering baseline matched in rollout budget. Both RLPR and RLCPR keep the batch size fixed at every optimizer step, so the full-rollout cost is independent of the UDS retention rate. The procedure is as follows:
Randomly filtering questions at any retention rate therefore yields an i.i.d. sample from the same data distribution, with the same per-step batch size and rollout budget. This is statistically equivalent to applying no filtering, i.e., the "+CR" setting. The comparison between RLCPR and "+CR" thus isolates uncertainty-based selection: 82.9 vs. 79.5 on MATH-500, 63.8 vs. 59.7 on Minerva, and 16.4 vs. 15.0 on AIME24.
We would also like to highlight that, because of the buffer strategy, UDS does not reduce computational cost simply by shrinking the training batch. Every optimizer step still trains with a fixed batch size. In our implementation, RLCPR's full-rollout cost is therefore theoretically invariant to the UDS retention rate (Appendix D), and we have updated the implementation paragraph in Section 6.1 to clarify this. RLCPR's savings relative to RLPR come from two other sources. First, filtering happens before generation, so no rollouts are generated only to be discarded. Second, RLCPR produces shorter responses. The table below reports the average response length (tokens) during training: RLCPR's responses grow more slowly than RLPR's, even before CR is activated at step 160.
Table: Average response length (tokens) at a given training step