acceptodds
Under review as a conference paper at ICLR 2027

Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has achieved substantial gains in single-attempt accuracy (Pass@1) on reasoning tasks, yet often suffers from reduced multi-sample coverage (Pass@K), indicating diversity collapse. We identify a structural cause for this degradation: common RLVR objectives, such as GRPO, are indifferent to how probability mass is distributed among correct solutions. Combined with stochastic training dynamics, this indifference induces a self-reinforcing collapse, in which probability mass concentrates on a narrow subset of correct outputs while alternative valid solutions are suppressed. We formalize this collapse mechanism and further characterize the optimal policy structure under two complementary criteria: conditional-entropy optimality and robustness, which identify the Uniform-Correct Policy as uniquely optimal. Motivated by this analysis, we propose Uniform-Correct Policy Optimization (UCPO), a modification to GRPO that adds a conditional uniformity penalty on the policy’s distribution over correct solutions. The penalty redistributes gradient signal toward underrepresented correct responses, encouraging uniform allocation of probability mass within the correct set. Across three models (1.5B–7B parameters) and five mathematical reasoning benchmarks, UCPO achieves a stronger correctness-coverage trade-off: it obtains the highest average Pass@64 across all three model families while preserving Pass@1, avoiding the single-sample accuracy degradation of competing diversity-oriented methods. UCPO achieves gains of up to +10 points on AIME24 at Pass@64 and up to 45% higher equation-level diversity within the correct set.

Primary area:
reinforcement learning
Keywords:
RLVR, Diversity, LLM, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YE4aOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Common RLVR like GRPO collapse probability into a single answer instead of spreading across all correct answers. This paper proposes an algorithm to incentivize a uniform distribution over correct answers. In practice they reweight GRPO’s positive advantages by inverse length-normalized sequence probability.

Critical strengths
  • Simple cheap method with a clear diagnosis. The change is an O(n) reweighting of positive advantages, with no extra rollouts and no reference model.
  • paper has broad comparisons. It compares against six diversity baselines across three models, and UCPO has the best average Pass@64 in all three.
Critical weaknesses
  • The empirical gains are small, noisy, and partly overstated. AIME has 30 problems, so +10 points is only 3 problems.
  • Missing citation for "Vector Policy Optimization: Training for Diversity Improves Test-Time Search" which is a paper with highly related motivation, but I think a different algorithm.
Questions for authors

Are there any situations in which you expect UCPO to underperform GRPO? Adding some conceptual insight around that would be very useful.

1 comment

We thank the reviewer for the thorough assessment and for highlighting the simplicity of the method and the breadth of the comparisons. We address each point raised below.

W1: We agree that differences on AIME may seem coarse: AIME24 has 30 problems, so the Pass@64 gain corresponds to problems which GRPO does not solve in any of 64 attempts. So we evaluate on five benchmarks that differ in size and difficulty, from the 30 problems of AIME24 and AIME25 to the 500 and 675 problems of MATH500 and OlympiadBench, and report averages across them. These benchmarks form the standard evaluation suite for RLVR on mathematical reasoning, used in prior work including the baselines we compare against [1,2,3]. Two results support the consistency of the gains across models and training runs:

Consistency across models. UCPO obtains the largest average Pass@64 gain over GRPO on all three models (+3.46, +3.61, and +4.82 points).

Consistency across runs. On Qwen2.5-7B, UCPO ranks highest in average Pass@64 in both independent training runs (Table 7: 79.89 ± 0.09 vs. 79.44 ± 0.33 for the strongest baseline). Across these same runs, UCPO also provides a better correctness–coverage trade-off (Table 7).

W2: Vector Policy Optimization. Thank you for the pointer; we have added the citation. VPO shares our motivation of preserving useful diversity for test-time search, but its diversity mechanism relies on vector-valued feedback in RL/RLHF. UCPO addresses the more limited feedback setting of math RLVR, where only a single binary correctness reward is available. We use policy scores to reweight accepted responses, allowing responses with the same verifier reward to receive different update weights without introducing additional reward dimensions.

Q1: When could UCPO underperform GRPO? Thank you for this question. We see two cases:

When τ\tau is large. Increasing τ\tau strengthens inverse-score reweighting, making the positive advantages more uneven across accepted responses. Excessively strong reweighting can hurt both accuracy and coverage. We observe degradation at (τ\tau=0.5): average Pass@64 falls from 81.85 under GRPO to 80.85, and average Pass@1 falls from 47.13 to 45.61 across the four benchmarks in Table 8.

When the feedback distinguishes quality among accepted responses. UCPO targets a uniform allocation within the correct set, which is appropriate when the reward is a binary verifier and all accepted responses are equally valid. With graded rewards or preference feedback, accepted responses can differ in quality. GRPO can exploit these reward differences directly, whereas our current redistribution rule is designed for the binary setting. We therefore expect that extending UCPO to richer feedback would require a quality-aware target; applying the current rule without adaptation could discard useful distinctions.

[1] Cui, Ganqu, et al. "The entropy mechanism of reinforcement learning for reasoning language models." 2025

[2] Cheng, Daixuan, et al. "Reasoning with exploration: An entropy perspective." Proceedings of the AAAI Conference on Artificial Intelligence. 2026

[3] Yue, Yang, et al. "Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?." Advances in Neural Information Processing Systems 2025

Reviewer_Xk4aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UCPO addresses the concentration of probability mass among correct responses in RLVR. The paper proposes a uniform distribution over correct solutions as an ideal target and introduces a conditional uniformity penalty. In practice, it gives larger weights to correct responses with lower length-normalized probabilities, while preserving their total advantage mass. Experiments on three models show improved Pass@64 and equation-level diversity, with varying effects on Pass@1.

Critical strengths
  1. The paper makes a useful distinction between overall correctness and diversity within correct responses. This provides a more targeted motivation than increasing entropy across all outputs.

  2. The practical method is lightweight. It reweights existing rollouts without additional models or sampling. The evaluation includes several relevant baselines, diversity measurements, and training-cost comparisons.

Critical weaknesses
  1. The uniformity result relies on inverse sequence-probability weighting. The implementation uses length-normalized probabilities instead. Appendix C.5 acknowledges that this produces a different reweighting target, so the ideal optimality result does not directly justify the implemented update.

  2. The proof of Theorem 3.2 takes the correct-response advantage outside the sampling expectation. However, this advantage depends on the number of correct samples in the group. Its dependence on sample counts needs to be accounted for.

  3. On the 1.5B model, average Pass@1 drops from 41.75 to 39.71. The results support a favorable correctness–coverage trade-off against several baselines, rather than coverage gains without an accuracy cost.

Questions for authors

How should we interpret the objective induced by the practical length-normalized weights? Could you clarify which properties of the ideal uniform-correct objective remain applicable?

Reviewer_mt8BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Uniform-Correct Policy Optimization (UCPO), which reweights verifier-accepted responses to encourage diversity among correct solutions. It motivates the method through an analysis of RLVR collapse and evaluates correctness, Pass@64, and equation-level diversity across three model checkpoints and five mathematical benchmarks.

Critical strengths

Simple and inexpensive intervention. UCPO reuses existing correct rollouts and introduces only scalar advantage reweighting, requiring no additional model calls or sampling.

Consistent improvements over GRPO in average Pass@64. Table 1 reports improvements across all three checkpoints, including an increase from 79.69 to 83.30 on DeepSeek-Distill-Qwen-7B.

Evaluation extends beyond answer accuracy. The paper includes equation-level diversity, controlled finite-policy experiments, coefficient sensitivity, and two independent training runs for Qwen2.5-7B.

Critical weaknesses

The collapse proof mishandles the randomness of group-normalized advantages. Theorem 3.2 takes (A^+) outside the expectation, although it depends on the sampled group and is correlated with response counts. The subsequent gradient-to-logit argument and claimed irreversibility therefore require a more rigorous derivation with explicit assumptions.

The implemented update does not inherit the ideal objective’s unique optimum. Equation (12) multiplies the reweighting by the original positive advantage. When every rollout is correct, all GRPO advantages are zero, so UCPO provides no uniformity correction even for a nonuniform correct-response distribution. Appendix C.5 also acknowledges that length normalization changes the target. The ideal theorem therefore does not establish the practical algorithm’s convergence to uniform correctness.

Pass@K does not directly measure within-correct diversity. For independent samples on a fixed prompt, expected Pass@K is (1-(1-Z_\pi(x))^K), which depends only on total correctness probability. Redistributing probability among correct solutions while holding this probability fixed cannot improve Pass@K. An additional argument is needed to connect the proposed mechanism to changes in correctness probabilities across prompts.

The claim of preserving Pass@1 is too broad. On the 1.5B model, average Pass@1 decreases from 41.75 under GRPO to 39.71 under UCPO. Meanwhile, the average Pass@64 margin over Ent-Adv on DeepSeek-Distill-Qwen-7B is only 0.18 points. These results support a potentially useful trade-off, but not uniformly improved accuracy.

Equation uniqueness is an imperfect measure of reasoning diversity. Appendix H compares extracted formula strings, which can change through notation or formatting without representing different reasoning strategies. The score also depends on the number of correct samples being compared, requiring controls for sample count and response length.

Questions for authors

Can the collapse analysis be rederived while retaining the dependence between group advantages and sample counts? What stationary policies does the implemented UCPO update admit when correctness saturates?

What mechanism connects within-correct reweighting to Pass@K improvements beyond changes in per-prompt correctness probability? Can these effects be separated experimentally?

Do the diversity improvements persist with matched numbers of correct samples and semantic normalization of equations? Can additional training seeds establish the small margins over the strongest baselines?

3 comments

We thank the reviewer for the detailed review. We have addressed each point below.

W1 / Q1: Randomness of group-normalized advantages. Thank you for raising this point. We have updated Theorem 3.2 and its proof (Appendix C.2) to make the dependence of A+A^+ on the sampled group explicit, and stated the assumptions of the analysis (binary rewards, tabular softmax over complete outputs, one on-policy gradient step). Under binary rewards, A+=A+(n)A^+=A^+(n) depends on the group through the number nn of correct rollouts. Since GRPO advantages sum to zero within each group, the logit update of a correct output is Δzy=ηKNyA+(n)\Delta z_y=\frac{\eta}{K}N_yA^+(n). Conditional on nn, the correct rollouts are i.i.d. draws from qθq_\theta, so E[Ny∣n]=n qθ(y∣x)\mathbb E[N_y\mid n]=n\,q_\theta(y\mid x), which gives E[Δzy]=ηK cK,ϵ(Zθ) qθ(y∣x),cK,ϵ(Z)=En∼Bin(K,Z)[nA+(n)],\mathbb E[\Delta z_y]=\frac{\eta}{K}\,c_{K,\epsilon}(Z_\theta)\,q_\theta(y\mid x),\qquad c_{K,\epsilon}(Z)=\mathbb E_{n\sim\mathrm{Bin}(K,Z)}\big[nA^+(n)\big], with cK,ϵ(Z)>0c_{K,\epsilon}(Z)>0 for 0<Z<10<Z<1. The conclusion of Theorem 3.2 holds, now with an explicit constant. Figure 1(b) verifies it quantitatively in the controlled environment: the measured expected logit updates of correct tokens match the prediction across three initialization profiles. The updated analysis yields a strong form of Theorem 3.3: the absolute log-ratio between two correct responses is a submartingale, so in expectation the GRPO update never narrows the gap between them. We have also sharpened Corollary 3.1 to state the mechanism precisely: solutions below the sampling threshold are rarely reinforced, and the drift vanishes as Zθ→1Z_\theta\to1, so the reward provides no signal that restores their mass.

W2 / Q1: From the ideal objective to the implemented update. Thank you for bringing up this point. Theorem 5.1 characterizes the unique optimum of the ideal population objective. The connection between the implemented update and that objective is through its gradient, which we have made explicit in the added Appendix C.8.

By Proposition 5.1, the gradient of the ideal objective weights each correct output by (1−τ)qθ+τu(1-\tau)q_\theta+\tau u: a mixture of the current conditional allocation and the uniform allocation. Eq. (12) retains this mixture construction exactly within each sampled group: wi=1−τn+τvˉiw_i=\frac{1-\tau}{n}+\tau\bar v_i combines count-proportional weights, whose expectation is qθq_\theta, with normalized inverse scores for the redistribution component, preserving the total positive advantage exactly (Proposition 5.2). With unnormalized inverse probabilities, its expected effect is exactly τηKcK,ϵ(Zθ)(u−qθ)\tau\frac{\eta}{K}c_{K,\epsilon}(Z_\theta)(u-q_\theta) (Appendix C.8, (iii)). The redistribution component departs from this in two respects: with length-normalized scores, its population counterpart is the tempered target u~θ∝πθ1−1/T\tilde u_\theta\propto\pi_\theta^{1-1/T} (Appendix C.5), and self-normalization introduces a further finite-sample difference. The redistribution component moves advantage toward correct responses with lower per-token likelihood (Appendix C.8, (ii)). Eq. (12) thus preserves the ideal gradient’s mixture structure, with a length-normalized redistribution component. The unique-optimum result of Theorem 5.1 remains specific to the ideal objective.

Stationary points. When all rollouts in a group are correct, A+=0A^+=0 and both the GRPO and UCPO updates vanish. At Zθ(x)=1Z_\theta(x)=1, every policy supported on correct outputs is stationary for Eq. (12), and UCPO introduces no stationary points beyond those of GRPO (Appendix C.8, (iv); Theorem 3.1). UCPO redistributes positive advantage during the mixed-reward learning regime that generates GRPO’s expected amplification.

W3 / Q2: Pass@K and within-correct diversity. We agree that Pass@K measures coverage rather than within-correct diversity, a distinction made in Section 6.1: since Pass@K=1−(1−Zπ(x))K\mathrm{Pass@}K=1-(1-Z_\pi(x))^K for independent samples on a fixed prompt, redistribution at fixed Zπ(x)Z_\pi(x) cannot improve it. We measure within-correct variation separately through equation-level diversity (Section 6.3).

The connection operates during training: UCPO changes which correct responses are reinforced, and hence the learned per-prompt correctness probabilities. Under GRPO, rarely sampled correct solutions are rarely reinforced (Section 3.2), and RLVR-trained models fall below the base model at large KK [1]. Our hypothesis is that retaining alternative solutions limits this contraction and supports correctness on more prompts. In the τ\tau ablation (Table 8, DeepSeek-R1-Distill-Qwen-1.5B), increasing τ\tau from 00 to 0.20.2 raises average Pass@64 from 81.8581.85 to 84.5084.50. Because reweighting changes both the allocation among correct responses and the correctness probabilities, the ablation cannot fully separate the two effects; the equation-level comparisons (W5) measure the accompanying within-correct variation. We will add this clarification explicitly to Section 6.4.

W4: Pass@1 vs. Pass@K trade-off. We agree that the 1.5B result represents a correctness–coverage trade-off: UCPO improves average Pass@64 by 3.46 points while reducing average Pass@1 by 2.04 points relative to GRPO. Our claim is that UCPO provides a favorable trade-off compared with competing coverage-oriented methods. On DeepSeek-Distill-Qwen-1.5B, every baseline in Table 1 that improves Pass@64 over GRPO also lowers Pass@1. To show this, we compare how much coverage a method gains for the Pass@1 it gives up. We compute the change in average Pass@1 and Pass@64 relative to GRPO for every method in Table R1.

Table R1: Changes in average Pass@1 and Pass@64 relative to GRPO on DeepSeek-Distill-Qwen-1.5B.

MethodΔPass@1ΔPass@64Pass@64 gained per Pass@1 lost
KL-Cov−0.61−0.83no gain
Ent-Reg−0.14+0.161.14
Ent-Adv−1.64+0.730.45
Pass@K Training−6.68+3.050.46
FGRPO−2.67+2.550.96
LAD−5.78+1.960.34
UCPO−2.04+3.461.70

UCPO obtains the largest Pass@64 gain and the highest rate of coverage gained per point of Pass@1. It is higher on both metrics than every baseline with a comparable coverage gain, and the baselines with a smaller Pass@1 cost gain at most 0.73 points of Pass@64.

On both 7B models, UCPO’s Pass@1 is within 0.25 points of GRPO’s (+0.24 and −0.11) while obtaining the largest Pass@64 gain. On DeepSeek-Distill-Qwen-7B, the 0.18-point Pass@64 margin over Ent-Adv comes with a 1.24-point Pass@1 advantage. On Qwen2.5-7B, averaged over two independent runs (Table 7), UCPO improves over Ent-Adv by 1.79 points in Pass@1 (40.78 ± 0.48 vs. 38.99 ± 0.08) and 0.45 points in Pass@64 (79.89 ± 0.09 vs. 79.44 ± 0.33), and ranks highest in average Pass@64 in both runs. The comparison rests on both metrics jointly.

W5/Q3:Equation-level diversity We use the equational diversity measure of DIVER [2], introduced as an intrinsic reward for RLVR and validated there against LLM-based and embedding-based similarity: methods with higher equational diversity exhibit lower semantic similarity. Within the correct set, equation-level variation provides a practical proxy for variation in mathematical derivations. We recompute the metric with both requested controls: (i) for each prompt, m=8 correct responses are subsampled from the K=64 samples, on prompts where every method has at least 8, averaged over 3 draws; (ii) equations are mapped to canonical symbolic form, so equivalent expressions count as identical.

Table R2: Equation-level diversity among correct solutions on DeepSeek-Distill-Qwen-1.5B, with 8 correct responses subsampled per prompt (3 draws) and equations mapped to canonical symbolic form.

BenchmarkGRPOKL-CovEnt-RegEnt-AdvPass@KFGRPOUCPO
AIME0.2860.1640.3090.1590.2420.1640.337
AMC0.3520.3410.2500.2670.2900.2490.335
MATH0.2790.3270.2530.2720.3020.3050.343
Olympiad0.3240.3600.2700.2940.3100.3070.364
Avg.0.3100.2980.2710.2480.2860.2560.345

Under both controls, UCPO retains the highest average diversity (0.345, an 11% improvement over GRPO) and the highest diversity on three of four benchmarks. The ratio normalizes for the number of extracted formulas.

On Qwen2.5-7B, we trained every method twice with independent seeds (Table 7). UCPO achieves the highest average Pass@64 in both runs and has the smallest observed run-to-run standard deviation: 0.09, compared with 0.12–2.57 for the baselines. Across the two runs, its mean Pass@64 exceeds Ent-Adv by 0.45 points and GRPO by 3.76 points. These results support the consistency of UCPO’s coverage advantage across the two independent training runs.

[1] Yue, Yang, et al. "Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?." Advances in Neural Information Processing Systems 2025

[2] Hu, Zican, et al. "Diversity-incentivized exploration for versatile reasoning." International Conference on Learning Representations. 2026.

Dear Reviewer mt8B,

Thank you for your valuable feedback and questions. As the discussion period ends soon, we wanted to check whether our response has addressed your concerns.

We appreciate your time and consideration.

Sincerely,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.