acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Probability-Based Reinforcement Learning From Posterior Concentration

Abstract

Verifier-free reinforcement learning with probability-based rewards offers a promising way to train LLMs on general reasoning tasks where external verifiers are unavailable. Yet the reliability of these rewards, especially in long-horizon reasoning, remains underexplored. This work identifies a length-dependent failure mode of probability rewards, which we call the **Posterior Concentration Phenomenon (PCP)**. We show that the probability of a reference answer conditioned on a reasoning trace often collapses to a low-variance interval as the trace becomes lengthy. This phenomenon results in nearly indistinguishable rewards, which, under GRPO-based settings, makes probability-based policy optimization unstable and inefficient. Motivated by this, we propose **Reinforcement Learning with Concentration-aware Posterior Rewards (RLCPR)**, a verifier-free RL framework to explicitly account for PCP for better optimization stability and token efficiency. It has two components: uncertainty-aware data sampling, which reduces concentration-prone rollouts before generation, and concentration-aware regularization, which penalizes unnecessarily long traces when posterior rewards collapse. Extensive experiments show that, alongside higher token efficiency, RLCPR outperforms the state-of-the-art verifier-free RL baseline by up to 4.0% on six of seven benchmarks, including general-domain and mathematical reasoning challenges. Source code will be released upon acceptance.

Primary area:
reinforcement learning
Keywords:
Large Language Model, Reinforcement Learning, Verifier-free training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ywQdOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies a length-dependent loss of discrimination in reference-answer probability rewards, termed Posterior Concentration Phenomenon (PCP), and proposes RLCPR: uncertainty-aware data sampling combined with concentration-gated length regularization. On Qwen3-4B-Base, RLCPR outperforms same-backbone RLPR on six of seven benchmarks, improving average accuracy while reducing rollout-token consumption.

Critical strengths
  1. The paper raises a practically relevant question about the reliability of probability-based rewards.
  2. The method is simple to implement and shows empirical gains across tasks.
  3. The method reduces rollout-token consumption, with gating and scheduling ablations supporting its design choices.
Critical weaknesses
  1. Relevant verifier-free baselines are missing. JEPO[1] and NOVER[2] are discussed in the related work but not evaluated, while DARL[3], an RLPR-based method using the same training corpus and overlapping benchmarks, is also omitted, leaving the comparative evaluation incomplete.

  2. Only one RL baseline provides a same-backbone comparison. RLCPR is trained on Qwen3-4B-Base, but Table 1 reports RLVR, General Reasoner, and VeriFree using borrowed Qwen2.5-7B results, leaving only RLPR as a directly comparable RL baseline and preventing a controlled assessment against the other methods.

  3. The ablations do not isolate the benefit of uncertainty-based sampling. Table 2 lacks a random-filtering baseline matched to UDS’s retention rate and rollout-token budget, making it difficult to distinguish uncertainty-based selection from generic subsampling effects.

  4. The theoretical analysis relies on an unestablished assumption. Proposition 4.1 requires convergence of cross-trajectory residual differences, which is not implied by the cited length-normalized likelihood results; Appendix G acknowledges this gap, limiting the theorem’s explanation of the observed PCP.

References

[1] Beyond Verifiable Rewards: Scaling Reinforcement Learning for Language Models to Unverifiable Data.

[2] NOVER: Incentive Training for Language Models via Verifier-Free Reinforcement Learning.

[3] DARL: Encouraging Diverse Answers for General Reasoning without Verifiers.

Questions for authors

See the Critical Weaknesses above.

9 comments
AuthorsOpenReview ↗

Individual Response to Reviewer ywQd

We thank Reviewer ywQd for the constructive comments. Following our general response, we address each concern below.

(W1, W2) Baseline coverage and same-backbone comparisons. RLPR is our primary baseline because it isolates the effect of our contribution. RLCPR adopts exactly the same training settings as RLPR (data, optimizer, hyperparameters, and training recipe) and differs only in replacing RLPR's post-generation reward filtering with UDS and CR. This controlled comparison directly measures the benefit of accounting for PCP, and it also shows that RLPR's reward filtering is computationally inefficient. Broader baselines serve a different purpose, namely establishing overall competitiveness, and we have expanded them as follows:

  • Same-backbone baselines. We added VeriFree on Qwen3-4B-Base, reproduced with its official implementation and the same training data as the other baselines. VeriFree is the most closely related open-source method. It also rewards traces with posterior probabilities. Qwen3-4B-Base now has two reproduced verifier-free baselines, and RLCPR outperforms both on six of seven benchmarks, with an average of 57.9% vs. 56.2% (RLPR) and 53.1% (VeriFree). We also reproduced RLPR on Qwen2.5-7B, so each backbone has a reproduced same-backbone baseline.
  • DARL. We include the DARL results on Qwen2.5-7B reported in its original paper. DARL reports higher scores on several benchmarks, but its WebInstruct result is unavailable, which precludes an average-accuracy comparison. Its diversity-promoting objective is also complementary to concentration-aware training. Because the official DARL implementation has not been released, we could not reproduce it on Qwen3-4B-Base.
  • JEPO. The official implementation of JEPO has not been released, so we could not reproduce it under our training data and backbones.
  • NOVER. NOVER rewards reasoning traces with the perplexity, a different reward formulation from the posterior-probability rewards of VeriFree and RLPR that our analysis and method target. Consistent with closely related work in this line (VeriFree, RLPR, and DARL), we therefore focus our comparison on posterior-probability methods.

(W3) Isolating the benefit of uncertainty-based sampling. The "+CR" variant in Table 2 already serves as the random-filtering baseline matched in rollout budget. Both RLPR and RLCPR keep the batch size fixed at every optimizer step, so the full-rollout cost is independent of the UDS retention rate. The procedure is as follows:

  • The dataloader loads a batch of bb questions.
  • These are filtered either by direct-answer uncertainty before generation (RLCPR) or by EMA-based reward-variance filtering after generation (RLPR).
  • The retained questions are stored in a buffer, and further batches are loaded and filtered until the buffer contains bb questions, i.e., b×Gb \times G rollouts, where GG is the GRPO group size.

Randomly filtering questions at any retention rate therefore yields an i.i.d. sample from the same data distribution, with the same per-step batch size and rollout budget. This is statistically equivalent to applying no filtering, i.e., the "+CR" setting. The comparison between RLCPR and "+CR" thus isolates uncertainty-based selection: 82.9 vs. 79.5 on MATH-500, 63.8 vs. 59.7 on Minerva, and 16.4 vs. 15.0 on AIME24.

We would also like to highlight that, because of the buffer strategy, UDS does not reduce computational cost simply by shrinking the training batch. Every optimizer step still trains with a fixed batch size. In our implementation, RLCPR's full-rollout cost is therefore theoretically invariant to the UDS retention rate (Appendix D), and we have updated the implementation paragraph in Section 6.1 to clarify this. RLCPR's savings relative to RLPR come from two other sources. First, filtering happens before generation, so no rollouts are generated only to be discarded. Second, RLCPR produces shorter responses. The table below reports the average response length (tokens) during training: RLCPR's responses grow more slowly than RLPR's, even before CR is activated at step 160.

Table: Average response length (tokens) at a given training step

Step180160240320
RLPR511.4813.91100.71286.41398.7
RLCPR507.3686.5739.1784.6934.3
AuthorsOpenReview ↗

Individual Response to Reviewer ywQd (Cont'd)

(W4) Residual assumption in Proposition 4.1. We agree that Prop. 4.1 requires the cross-trajectory residual differences to converge, and that this is not implied by the cited length-normalized likelihood results. We state it explicitly as a sufficient condition and discuss it in Appendix G. We would like to clarify that our key theoretical insight is a reduction: posterior-reward differences between long traces reduce to approximation errors alone. Eq. (11) decomposes the log-ratio of two posteriors into exactly two terms:

log⁡P~(y∗∣x,z1)P~(y∗∣x,z2)=(Wn−Wn+s)+(ρn+s(z1)−ρn(z2)),\log \frac{\tilde{P}(y^{\ast} | x, z^1)}{\tilde{P}(y^{\ast} | x, z^2)} = \left(W_n - W_{n+s}\right) + \left(\rho_{n+s}(z^1) - \rho_n(z^2)\right),

where the first term (Wn−Wn+s)\left(W_n - W_{n+s}\right) is (i) the deterministic, length-induced entropy shift, and the second term$\left(\rho_{n+s}(z^1) - \rho_n(z^2)\right)$ is (ii) the trace-specific approximation errors.

  • Eliminating the deterministic term. Term (i) is the only component that depends systematically on reasoning length. Lemma 4.1 proves that it vanishes outside a zero-density set of lengths, under only the branching-factor condition, which is supported empirically by [1]. Hence, differences in length alone cannot systematically separate posterior rewards.
  • What remains are approximation errors. Once term (i) is eliminated, any remaining reward difference between two long traces comes only from term (ii): the trace-specific residuals of the log-likelihood concentration approximation (Eq. (6)). In other words, the posterior reward can distinguish two long traces only through how far each trace's log-likelihood deviates from its expected entropy, not through any length-dependent structure.

This reduction is what the theory contributes. It identifies precisely where the discriminative signal of posterior rewards must come from in the long-trace regime, and the residual condition is the single additional assumption needed to conclude Prop. 4.1.

Establishing the convergence of these residuals analytically is difficult. Each residual is itself the approximation error between a trace's realized log-likelihood and its expected entropy, and its distribution depends on the specific model and prompt. Instead, we support this assumption empirically. Fig. 2b and 2c show that both the overall and the within-group variance of posterior rewards contract as reasoning length grows, which is the observable consequence of vanishing residual differences. We will state the scope of this condition more clearly in the revised manuscript.

[1] LLM Probability Concentration: How Alignment Shrinks the Generative Horizon (TMLR 2026)

Thank you for the reply. I still think the experiments are incomplete and rather confusing. The added DARL results on Qwen2.5-7B are better than RLCPR’s, which weakens the claimed advantage of RLCPR. The authors say the two methods are complementary, so I’d like to see DARL + RLCPR. If it improves on DARL, that would also be a useful result. Also, both DARL(https://github.com/huangchongxuan/DARL) and NOVER(https://github.com/thinkwee/NOVER) have public code, so I think the authors should run them on both backbones and report the results. Overall, the paper still needs a lot more experimental work before it can be accepted, so I’ll keep my weak reject.

Reviewer_PKBsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper improves Verifier-free RL experiments in two ways. First, before generating a complete rollout, it estimates uncertainty from the token entropy when the model directly answers the question. Second, if a group of rollouts are all long and their rewards are almost the same, it adds an extra length penalty. On Qwen3-4B-Base, it shows average improvement on 7 benchmarks compared with RLPR.

Critical strengths

The method tries to avoid spending tokens on less useful groups before generation. The length penalty also applies only under specific conditions. Compared with the main baseline without an external verifier, it improves on six of seven benchmarks. The component experiments also supports the roles of both modules.

Critical weaknesses

(1) The paper links smaller reward gaps to weaker learning signals. But GRPO normalizes rewards within each group. If all rewards only shrink by the same positive factor, the advantages stay the same. Figure 2 does not show that normalized advantages become worse at separating correct and wrong answers. So the main argument still has a missing step. (2) The average over seven benchmarks rises from 56.2 to 58.0 in Table 1. But results from multiple training seeds and confidence intervals for paired gains are not reported. Avg@k evaluation does not measure variation across training runs. I am not clear whether the average gain of 1.8 points is stable.

Questions for authors

(1) How does the link between normalized advantages and answer correctness change with length? If it becomes weaker, is this due to reward ranking, noise, or a numerical stabilizer? Please separate these effects from a smaller raw reward scale. (2) Can the authors report results from independent training seeds and confidence intervals for paired accuracy differences on the same questions? Please separate training variation from evaluation sampling uncertainty. Please also give an interval for the average gain over the seven benchmarks.

4 comments
AuthorsOpenReview ↗

Individual Response to Reviewer PKBs

We thank Reviewer PKBs for the careful and constructive comments. We address them below.

(W1, Q1) GRPO normalization and the link between advantages and correctness. The reviewer is correct that if all rewards in a group shrank by a common positive factor, GRPO normalization would leave the advantages unchanged. PCP, however, is not a uniform rescaling of rewards. It is a loss of ranking between correct and incorrect traces, which normalization cannot repair.

  1. Normalization preserves within-group ranking. GRPO normalization, Ai=(ri−μ)/σA_i = (r_i - \mu)/\sigma, is a monotonically increasing affine map within each group. It rescales the rewards but cannot reorder them. If the posterior reward ranks an incorrect trace above a correct one, that trace still receives the higher advantage after normalization. Our PCP analysis therefore concerns the reward function itself, i.e., whether it still separates correct from incorrect traces as length grows, rather than the advantage scale. Note that Prop. 4.1 is derived without any assumption on trace correctness: it states that the posterior reward depends increasingly little on the specific trace. This leaves less room to separate correct from incorrect traces.
  2. Direct evidence after normalization. The revised Fig. 2d plots the mean GRPO-normalized advantage of correct and incorrect traces against the group's average reasoning length on GPQA-Diamond. The gap between the two narrows as length grows, and traces with incorrect predictions can even receive higher advantages than correct ones in long groups. Since Fig. 2d is computed after normalization, a common scale factor cannot explain this trend.

Similar to [1], we attribute this behavior to a law-of-large-numbers effect. The LLM's token distribution carries a particular preference, so as sequences grow longer, their quality becomes increasingly indistinguishable to the model itself.

(W2, Q2) Multiple training seeds. We trained a second, independent RLCPR run on Qwen3-4B-Base and now report the two-seed average in Table 1. Both runs outperform RLPR on the same six benchmarks, with overall gains of +1.8 and +1.6 points.

Qwen3-4B-BaseMMLU-ProGPQA-DTheoremQAWebInst.MATH-500MinervaAIME24All
RLPR$^\dagger$58.543.457.178.080.761.514.056.2
RLCPR (seed 1)61.340.761.178.982.964.516.358.0
RLCPR (seed 2)61.338.462.580.382.863.116.557.8
RLCPR (mean)61.339.661.879.682.963.816.457.9
Gain, seed 1 / seed 2+2.8 / +2.8−2.7 / −5.0+4.0 / +5.4+0.9 / +2.3+2.2 / +2.1+3.0 / +1.6+2.3 / +2.5+1.8 / +1.6
Gain, mean+2.8−3.8+4.7+1.6+2.2+2.3+2.4+1.7

Because the benchmarks have very different baseline accuracies, we report per-benchmark absolute gains for each seed rather than relying on the average alone. On the six improved benchmarks, the smallest per-seed gain is +0.9 points (WebInstruct), so the direction of improvement is consistent across independent training runs.

[1] Slaves to the Law of Large Numbers: An Asymptotic Equipartition Property for Perplexity in Generative Language Models

Dear Reviewer PKBs,

Thank you again for your careful and constructive review. As the discussion period ends soon, we kindly invite you to look at our general response and our individual response to you.

In brief, we highlighted our central contribution, PCP, in the general response, and addressed both of your concerns with new evidence in the individual response.

We would sincerely appreciate it if you could reconsider your assessment according to these responses. We would also be glad to clarify any remaining questions before the discussion closes.

Best regards, The Authors

Dear Reviewer PKBs,

As the discussion period is about to end, we would like to follow up once more. We believe our responses have fully addressed both of your concerns:

  • GRPO normalization: PCP is a loss of ranking between correct and incorrect traces, not a uniform rescaling, so within-group normalization cannot repair it. The new post-normalization analysis in Fig. 2d confirms this directly.
  • Training seeds: A second independent run reproduces the improvement. Both seeds outperform RLPR on the same six of seven benchmarks (+1.8 and +1.6 points on average).

We have not received further questions, so we hope these responses have resolved your concerns. If so, we would sincerely appreciate it if you could update your assessment accordingly. We remain happy to clarify anything before the discussion closes.

Best regards, The Authors

AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer PKBs,

As the discussion period will end within 5 hours, we would like to kindly remind you of our responses. We would greatly appreciate your feedback, and we are happy to address any remaining questions before the deadline.

Best regards,

Authors

Reviewer_d8acOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates the reliability of posterior probability-based rewards in verifier-free reinforcement learning and identifies the PCP, where rewards become less discriminative as reasoning traces grow longer, weakening policy optimization. To address this issue, the authors propose RLCPR, which combines uncertainty-aware data sampling and concentration-aware length regularization to improve training effectiveness and efficiency. The paper also provides theoretical and empirical analyses of PCP. Experiments on seven reasoning benchmarks demonstrate that RLCPR outperforms the RLPR baseline on six benchmarks, achieving a 1.8 percentage-point improvement in average accuracy while reducing rollout token consumption. Ablation studies further evaluate the contributions of the proposed components.

Critical strengths

The paper identifies an important limitation of probability-based rewards in long reasoning traces and proposes a practical solution. The experimental results demonstrate improvements in both reasoning performance and training efficiency.

Critical weaknesses
  1. The experiments are conducted primarily on a single model, making it difficult to assess the generalizability of the proposed method.
  2. The proposed approach mainly combines uncertainty-based sampling and length regularization, both of which are relatively established techniques. The technical novelty is therefore somewhat limited.
Questions for authors
  1. How well does the proposed method generalize to other model architectures and scales beyond Qwen3-4B?
  2. How can the authors distinguish the benefits of mitigating posterior concentration from those of simply encouraging shorter reasoning traces?
2 comments
AuthorsOpenReview ↗

Individual Response to Reviewer d8ac

We thank Reviewer d8ac for recognizing the importance of PCP in long reasoning traces and the practicality of our solution. We address the concerns below.

(W1, Q1) Generalization beyond Qwen3-4B. We have added experiments on Qwen2.5-7B, which differs from Qwen3-4B-Base in both model generation and scale (Table 1). On Qwen2.5-7B, RLCPR improves over the reproduced RLPR by 1.4 points on average and outperforms it on six of seven benchmarks, with the largest gains on AIME24 (+3.1) and Minerva (+1.7). It also attains the highest average accuracy (52.8%) among all methods with complete results, including the verifier-based RLVR (52.6%) and General Reasoner (52.0%), even though RLCPR requires no external verifier.

Qwen2.5-7BMMLU-ProGPQA-DTheoremQAWebInst.MATH-500MinervaAIME24All
RLPR†53.034.655.574.175.653.313.851.4
RLCPR54.536.255.275.376.555.016.952.8

(W2) Novelty of UDS and CR. We agree that uncertainty-based sampling and length regularization are established techniques. As stated in the general response, our novelty lies primarily in identifying and analyzing posterior concentration as a failure mode of probability-based RL, rather than in introducing these techniques individually. Our analysis also suggests a different way to use them. UDS favors low-uncertainty data, whereas prior works such as UFO-RL [1] and HIVE [2] typically favor data with medium-to-high uncertainty. This difference arises because those methods do not use posterior probabilities as rewards and therefore need not account for a collapse of the reward signal such as PCP. Likewise, CR does not penalize length universally. It is activated only for groups that our analysis identifies as concentrated.

Moreover, UDS and CR are complementary rather than redundant. Uncertainty serves only as a proxy for PCP risk (see Appx. C for a diagnostic experiment on the relation between PCP and uncertainty), so UDS alone cannot prevent PCP. Reasoning lengths may still grow as training proceeds, and under the conditions of Prop. 4.1, posterior rewards are then likely to concentrate again. Consistent with this, Table 2 shows that UDS alone does not improve accuracy and CR alone yields modest gains. Their combination, however, improves over the baseline by 4.0, 6.0, and 2.0 points on MATH-500, Minerva, and AIME24, respectively. Our insight is that UDS and CR need to work together to mitigate PCP effectively.

(Q2) Mitigating PCP vs. simply encouraging shorter reasoning. Table A1 compares our gated regularization with a universal length penalty ("w/o CR gate"). This variant removes the binary PCP indicator in Eq. (17) and penalizes length in every group, and it is worse on all reported benchmarks (−1.6, −4.2, and −0.6 points on MMLU-Pro, TheoremQA, and WebInstruct).

Table A1 (Qwen3-4B-Base)MMLU-ProTheoremQAWebInst.
w/o CR gate (universal length penalty)59.757.679.0
RLCPR (full model)61.361.879.6

Two further ablations point in the same direction. Applying CR from the beginning of training instead of after a cold start reduces accuracy by 4.9–9.3 points (Table 3), and a stronger penalty (λ=1.0\lambda = 1.0) underperforms λ=0.5\lambda = 0.5 on all three benchmarks (Table A3). If the gains came simply from shorter reasoning, penalizing length more broadly, earlier, or more strongly should help; instead, each of these hurts. As discussed in our manuscript, an appropriate reasoning length can still benefit performance, so a length penalty should not be applied universally. Based on our analysis, we regularize only the groups that suffer from PCP and may therefore destabilize training.

[1] UFO-RL: Uncertainty-Focused Optimization for Efficient Reinforcement Learning Data Selection (NeurIPS 2026)

[2] Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Model (NeurIPS 2026)

Thank you for the detailed response and additional experiments.

The new results on Qwen2.5-7B substantially address my concern about generalizability, showing consistent improvements across most benchmarks on a different model generation and scale.

I also appreciate the clarification of the paper's novelty. While the individual components are based on established techniques, the PCP-motivated design and their complementary roles provide a reasonable justification for the proposed approach.

Regarding my second question, the additional ablations offer useful evidence that the improvements cannot be explained simply by applying stronger or more general length penalties. Although these results do not fully establish that mitigating posterior concentration is the direct mechanism behind the gains, they adequately address my main concern.

Overall, I am satisfied with the authors' response and will maintain my positive assessment.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response from authors

We thank all reviewers for their constructive feedback. We appreciate that the reviewers recognize the importance of PCP (d8ac, ywQd), the rollout-token efficiency of RLCPR (d8ac, PKBs, ywQd), and its empirical accuracy gains (d8ac, PKBs, ywQd).

Our central contribution. Our novelty lies primarily in identifying and theoretically analyzing PCP as a failure mode of probability-based RL, rather than in introducing RLCPR as a standalone verifier-free method. This analysis motivates a coordinated intervention: UDS selects examples with lower estimated concentration risk, while CR selectively discourages unnecessarily long reasoning through a PCP-aware gate. We will revise the presentation to make the distinction between the central contribution and its practical implementation clearer. We kindly refer the reviewers to Section 4, which explains how PCP arises, why it matters for group-relative optimization, and how it motivates the design of RLCPR.

Summary of revisions. Following the reviewers' comments, we have revised the manuscript as follows:

  1. Table 1: added reproduced VeriFree results on Qwen3-4B-Base, as well as RLCPR, excerpted DARL, and reproduced RLPR results on Qwen2.5-7B.
  2. Figure 2: added an experiment on GRPO-normalized advantages on GPQA-Diamond (Fig. 2d) and regenerated the MMLU-Pro results, with the corresponding discussion in Section 4.1 and implementation details in Appendix E.1.
  3. Section 6.1: clarified the buffer strategy that keeps the batch size fixed at every optimizer step.
  4. Figure 4 and Section 6.2: corrected a reporting error in the rollout-token count. The original figure multiplied the average response length by the post-filtering batch size without the buffer strategy. The per-step rollout tokens are now computed as 11−rGbℓˉ\frac{1}{1-r}Gb\bar{\ell} for RLPR and Gbℓˉ+kbfGb\bar{\ell} + \frac{kb}{f} for RLCPR, where GG is the GRPO group size, bb is the fixed batch size, ℓˉ\bar{\ell} is the average response length, ff is the update frequency of the uncertainty buffer, and r∈[0,1]r\in[0,1] is RLPR's reward-filtering rate.
  5. Appendix D: updated the complexity analysis under a fixed batch size.
  6. All RLCPR results are now averaged over two training seeds.
  7. Related work now includes discussion with DARL.
  8. Figure 1: updated with the reproduced VeriFree and the seed-average RLCPR.

Reproducibility of RLPR. Additionally, we would like to highlight a reproducibility issue with RLPR (https://github.com/OpenBMB/RLPR/issues/16): several researchers, including us, could not reproduce the results reported in the original paper. Given the computational cost, we reproduced RLPR on Qwen2-7B with two training seeds and report their average in Table 1.

Below, we respond to each reviewer individually.

AuthorsOpenReview ↗

Kind reminder about the author responses and revised manuscript

Dear Reviewers,

We sincerely thank all reviewers for their time and constructive feedback. We are grateful to Reviewer d8ac for the thoughtful follow-up and positive assessment.

As the discussion period is approaching to its end, we would like to kindly invite Reviewers PKBs and ywQd to take a look at our responses and the revised manuscript. To summarize the main updates:

  • Generalization and baselines: new results on Qwen2.5-7B, reproduced VeriFree on Qwen3-4B-Base, reproduced RLPR on Qwen2.5-7B, and DARL, with RLCPR achieving the highest average accuracy on both backbones.
  • Robustness: all RLCPR results are now averaged over two independent training seeds; both runs outperform RLPR on the same six of seven benchmarks.
  • GRPO normalization: a new post-normalization analysis (Fig. 2d) showing that PCP is a loss of ranking between correct and incorrect traces, which within-group normalization cannot repair.
  • Isolating UDS: clarification that, with a fixed batch size, the “+CR” variant serves as the budget-matched random-filtering baseline, which RLCPR outperforms.
  • Theory: clarification that our analysis reduces posterior-reward differences between long traces to approximation errors alone, with the residual condition as the single additional assumption, supported empirically.

We would be happy to clarify any remaining questions before the discussion closes. If our responses have addressed your concerns, we would sincerely appreciate it if you could consider updating your assessment.

Best regards, The Authors

AuthorsOpenReview ↗

Summary of the discussion and revisions

We thank all reviewers and the Area Chair for their time and constructive feedback.

Central contribution. We identify and theoretically analyze the Posterior Concentration Phenomenon (PCP), a length-dependent failure mode in which posterior-probability rewards lose their ability to distinguish reasoning quality. RLCPR (UDS + CR) is the solution motivated by this analysis. We validate it primarily through a controlled comparison with RLPR on Qwen3-4B-Base and Qwen2.5-7B, complemented by comparisons with other relevant baselines, including VeriFree, NOVER, and DARL.

Outcome. Reviewer d8ac maintained a weak accept, and Reviewer ywQd raised their score to weak accept after our additional experiments. We addressed the concerns of Reviewer PKBs with new evidence (see below), although we did not receive further feedback from them during the discussion period. We believe our additional experiments have provided sufficient evidence to address the reviewers' concerns.

Main additions.

  • Qwen2.5-7B results: RLCPR attains the highest average accuracy on both backbones and outperforms all baselines on six of seven benchmarks on Qwen3-4B-Base.
  • Baselines: reproduced VeriFree, RLPR, and NOVER, all of which RLCPR outperforms. A preliminary run shows that adding RLCPR to DARL improves five of seven benchmarks.
  • Two training seeds: both runs outperform RLPR on the same six of seven benchmarks.
  • Post-normalization analysis (Fig. 2d): PCP is a loss of ranking that GRPO normalization cannot repair.
  • Clarifications: the fixed-batch strategy, the residual assumption in Prop. 4.1, and ablations showing that the gains are not explained by shorter traces.

Final version. We will include full-length DARL and DARL + RLCPR results and fair comparisons with all discussed baselines under the same protocol.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.