- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces POLY-EPO, a set reinforcement learning approach for balancing exploration and exploitation in language-model reasoning. It derives per-generation marginal advantages from set-level objectives and instantiates this framework with a polychromic objective that jointly rewards task performance and reasoning-strategy diversity. Results on mathematical reasoning benchmarks show promising improvements in pass@ and majority-voting performance.
The paper is clearly written and theoretically well motivated, and I find the marginal set-advantage formulation interesting. However, the empirical evidence for genuinely broader semantic strategy diversity remains limited, and the evaluation leaves several attribution issues unresolved, including different response-length normalization from GRPO, missing comparisons with closely related pass@ and diversity-oriented methods, and unclear effects of the 4096-token response limit. Overall, I find the approach promising, but the current empirical evidence is not yet sufficient to support the main claims cleanly. My overall assessment is weak reject.
Critical strengths
The paper is clearly written, well motivated, and has strong theoretical grounding. The derivation of the marginal set advantage provides a simple general recipe for optimizing set-level objectives with standard policy-gradient-style updates.
Critical weaknesses
1. The empirical evaluation does not cleanly isolate the contribution of POLY-EPO.
POLY-EPO and GRPO use different response-length normalization (Appendix B), introducing a confound beyond the proposed marginal set advantage. In addition, although the paper includes GRPO+DIV as an in-house diversity-bonus baseline and discusses its differences from prior work analytically (Appendices A and D), it lacks empirical comparisons with published methods that directly optimize pass@ or explicitly reward strategy diversity, such as PKPO [1] and Uniqueness-Aware RL [2]. Such comparisons would better establish whether the gains are specific to the polychromic set objective, rather than attributable to multi-sample objectives or cluster-based diversity incentives more generally.
Table 1 further specifies a maximum training response length of 4096 tokens, but the truncation rate during training and corresponding evaluation limit is unclear. This is particularly relevant because the LM judge clusters sampled trajectories, so truncation may affect not only final-answer accuracy but also the strategy assignments that directly determine the set-level learning signal. If the same 4096-token limit is also used at test time, this could further complicate the interpretation of pass@ results, since some otherwise-correct solutions may be truncated before completion. I would therefore like the authors to clarify the evaluation length limit and report response-length distributions and truncation rates during training and evaluation.
2. Evidence of greater semantic strategy diversity on held-out mathematical benchmarks remains limited.
Figure 2 reports diversity during training on POLARIS-53k, using the same Qwen-3-4B-Instruct judge that supplies the training diversity signal. It therefore provides limited independent evidence of broader semantic strategy diversity on held-out problems. The held-out prefix-branching analysis is complementary, but token-level branching may reflect wording differences rather than genuinely distinct solution strategies. Since strategic exploration is a central claim, an independent semantic evaluation on held-out benchmarks, ideally using humans or a separate judge, would substantially strengthen the paper.
References
[1] Walder and Karkhanis, Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems, 2025.
[2] Hu et al., Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs, 2026.
Questions for authors
- How sensitive is POLY-EPO to the choice and capability of the LM judge when the task, policy model, and other training settings are held fixed? Although appendix F uses Gemini-2.0-Flash in synthetic domains, a controlled comparison on the same mathematical reasoning setup would help establish judge sensitivity.
- The paper’s empirical evaluation focuses on single-turn mathematical reasoning. How well does POLY-EPO extend to other domains such as code generation, and what changes to the clustering rubric would be needed?
- The paper argues that the marginal set advantage is generally compatible with standard RL algorithms, but the empirical evaluation uses a single RL backbone. Does the method provide similar benefits with other objectives, such as CISPO or alternative PPO variants?
- Could the authors provide the numerical results underlying the main pass@ and majority-voting plots in tabular form?