- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Common RLVR like GRPO collapse probability into a single answer instead of spreading across all correct answers. This paper proposes an algorithm to incentivize a uniform distribution over correct answers. In practice they reweight GRPO’s positive advantages by inverse length-normalized sequence probability.
Critical strengths
- Simple cheap method with a clear diagnosis. The change is an O(n) reweighting of positive advantages, with no extra rollouts and no reference model.
- paper has broad comparisons. It compares against six diversity baselines across three models, and UCPO has the best average Pass@64 in all three.
Critical weaknesses
- The empirical gains are small, noisy, and partly overstated. AIME has 30 problems, so +10 points is only 3 problems.
- Missing citation for "Vector Policy Optimization: Training for Diversity Improves Test-Time Search" which is a paper with highly related motivation, but I think a different algorithm.
Questions for authors
Are there any situations in which you expect UCPO to underperform GRPO? Adding some conceptual insight around that would be very useful.
We thank the reviewer for the thorough assessment and for highlighting the simplicity of the method and the breadth of the comparisons. We address each point raised below.
W1: We agree that differences on AIME may seem coarse: AIME24 has 30 problems, so the Pass@64 gain corresponds to problems which GRPO does not solve in any of 64 attempts. So we evaluate on five benchmarks that differ in size and difficulty, from the 30 problems of AIME24 and AIME25 to the 500 and 675 problems of MATH500 and OlympiadBench, and report averages across them. These benchmarks form the standard evaluation suite for RLVR on mathematical reasoning, used in prior work including the baselines we compare against [1,2,3]. Two results support the consistency of the gains across models and training runs:
Consistency across models. UCPO obtains the largest average Pass@64 gain over GRPO on all three models (+3.46, +3.61, and +4.82 points).
Consistency across runs. On Qwen2.5-7B, UCPO ranks highest in average Pass@64 in both independent training runs (Table 7: 79.89 ± 0.09 vs. 79.44 ± 0.33 for the strongest baseline). Across these same runs, UCPO also provides a better correctness–coverage trade-off (Table 7).
W2: Vector Policy Optimization. Thank you for the pointer; we have added the citation. VPO shares our motivation of preserving useful diversity for test-time search, but its diversity mechanism relies on vector-valued feedback in RL/RLHF. UCPO addresses the more limited feedback setting of math RLVR, where only a single binary correctness reward is available. We use policy scores to reweight accepted responses, allowing responses with the same verifier reward to receive different update weights without introducing additional reward dimensions.
Q1: When could UCPO underperform GRPO? Thank you for this question. We see two cases:
When τ is large. Increasing τ strengthens inverse-score reweighting, making the positive advantages more uneven across accepted responses. Excessively strong reweighting can hurt both accuracy and coverage. We observe degradation at (τ=0.5): average Pass@64 falls from 81.85 under GRPO to 80.85, and average Pass@1 falls from 47.13 to 45.61 across the four benchmarks in Table 8.
When the feedback distinguishes quality among accepted responses. UCPO targets a uniform allocation within the correct set, which is appropriate when the reward is a binary verifier and all accepted responses are equally valid. With graded rewards or preference feedback, accepted responses can differ in quality. GRPO can exploit these reward differences directly, whereas our current redistribution rule is designed for the binary setting. We therefore expect that extending UCPO to richer feedback would require a quality-aware target; applying the current rule without adaptation could discard useful distinctions.
[1] Cui, Ganqu, et al. "The entropy mechanism of reinforcement learning for reasoning language models." 2025
[2] Cheng, Daixuan, et al. "Reasoning with exploration: An entropy perspective." Proceedings of the AAAI Conference on Artificial Intelligence. 2026
[3] Yue, Yang, et al. "Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?." Advances in Neural Information Processing Systems 2025