acceptodds
Under review as a conference paper at ICLR 2027

Poly-EPO: Training Exploratory Reasoning Models

Abstract

Exploration is a cornerstone of learning from experience: it enables agents to find solutions to complex problems, generalize to novel ones, and scale performance with test-time compute. In this paper, we present a framework for post-training language models (LMs) that explicitly encourages optimistic exploration and promotes a synergy between exploration and exploitation. The central idea is to train the LM to generate sets of responses that are collectively accurate under the reward function and exploratory in their reasoning strategies. We first develop a general recipe for optimizing LMs with set reinforcement learning (set RL) under arbitrary objective functions, showing how standard RL algorithms can be adapted to this setting through a modification to the advantage computation. We then propose Polychromic Exploratory Policy Optimization (Poly-EPO), which instantiates this framework with an objective that explicitly synergizes exploration and exploitation. Across a range of reasoning benchmarks, we show that Poly-EPO improves generalization, as evidenced by higher pass@ coverage, preserves greater diversity in model generations, and effectively scales with test-time compute.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Exploration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_w4cAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces POLY-EPO, a set reinforcement learning approach for balancing exploration and exploitation in language-model reasoning. It derives per-generation marginal advantages from set-level objectives and instantiates this framework with a polychromic objective that jointly rewards task performance and reasoning-strategy diversity. Results on mathematical reasoning benchmarks show promising improvements in pass@kk and majority-voting performance.

The paper is clearly written and theoretically well motivated, and I find the marginal set-advantage formulation interesting. However, the empirical evidence for genuinely broader semantic strategy diversity remains limited, and the evaluation leaves several attribution issues unresolved, including different response-length normalization from GRPO, missing comparisons with closely related pass@kk and diversity-oriented methods, and unclear effects of the 4096-token response limit. Overall, I find the approach promising, but the current empirical evidence is not yet sufficient to support the main claims cleanly. My overall assessment is weak reject.

Critical strengths

The paper is clearly written, well motivated, and has strong theoretical grounding. The derivation of the marginal set advantage provides a simple general recipe for optimizing set-level objectives with standard policy-gradient-style updates.

Critical weaknesses

1. The empirical evaluation does not cleanly isolate the contribution of POLY-EPO.

POLY-EPO and GRPO use different response-length normalization (Appendix B), introducing a confound beyond the proposed marginal set advantage. In addition, although the paper includes GRPO+DIV as an in-house diversity-bonus baseline and discusses its differences from prior work analytically (Appendices A and D), it lacks empirical comparisons with published methods that directly optimize pass@kk or explicitly reward strategy diversity, such as PKPO [1] and Uniqueness-Aware RL [2]. Such comparisons would better establish whether the gains are specific to the polychromic set objective, rather than attributable to multi-sample objectives or cluster-based diversity incentives more generally.

Table 1 further specifies a maximum training response length of 4096 tokens, but the truncation rate during training and corresponding evaluation limit is unclear. This is particularly relevant because the LM judge clusters sampled trajectories, so truncation may affect not only final-answer accuracy but also the strategy assignments that directly determine the set-level learning signal. If the same 4096-token limit is also used at test time, this could further complicate the interpretation of pass@kk results, since some otherwise-correct solutions may be truncated before completion. I would therefore like the authors to clarify the evaluation length limit and report response-length distributions and truncation rates during training and evaluation.

2. Evidence of greater semantic strategy diversity on held-out mathematical benchmarks remains limited.

Figure 2 reports diversity during training on POLARIS-53k, using the same Qwen-3-4B-Instruct judge that supplies the training diversity signal. It therefore provides limited independent evidence of broader semantic strategy diversity on held-out problems. The held-out prefix-branching analysis is complementary, but token-level branching may reflect wording differences rather than genuinely distinct solution strategies. Since strategic exploration is a central claim, an independent semantic evaluation on held-out benchmarks, ideally using humans or a separate judge, would substantially strengthen the paper.

References

[1] Walder and Karkhanis, Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems, 2025.

[2] Hu et al., Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs, 2026.

Questions for authors
  • How sensitive is POLY-EPO to the choice and capability of the LM judge when the task, policy model, and other training settings are held fixed? Although appendix F uses Gemini-2.0-Flash in synthetic domains, a controlled comparison on the same mathematical reasoning setup would help establish judge sensitivity.
  • The paper’s empirical evaluation focuses on single-turn mathematical reasoning. How well does POLY-EPO extend to other domains such as code generation, and what changes to the clustering rubric would be needed?
  • The paper argues that the marginal set advantage is generally compatible with standard RL algorithms, but the empirical evaluation uses a single RL backbone. Does the method provide similar benefits with other objectives, such as CISPO or alternative PPO variants?
  • Could the authors provide the numerical results underlying the main pass@kk and majority-voting plots in tabular form?
Reviewer_L6DnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Poly-EPO, a set-based RL method designed to improve diversity in LLM reasoning. It proposes a marginal advantage estimator and uses an LLM-based judge to group reasoning strategies. The experiments show improvements of pass rate and reasoning diversity.

Critical strengths
  1. The diversity problem, especially in rl, is important.
  2. The paper proposed an novel way to use set rl in llm training.
  3. The experiment results are good. The gains of pass@k is observable.
Critical weaknesses
  1. The baseline lacks fairness. GRPO uses response-length normalization, Poly-EPO uses a fixed maximum length.
  2. The benefit of multiplicative coupling is not fully established. The paper does not have experiments for DIV and EPO under the same rl framework.
  3. About the diversity metric.The more correct answers a model gets, the more likely it is to show different categories. The authors should also report strategy diversity when controlling for the number of correct answers.
Questions for authors
  1. Can the authors provide baselines with identical length normalization?

  2. How does Poly-EPO compare with an additive set-level objective?

  3. About robustness, can authors provided results under different judges and random seeds?

Reviewer_Uqx4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper adapts set reinforcement learning to language-model post-training by constructing subsets from a shared pool of sampled responses, evaluating each subset, and assigning each response a marginal set advantage. Poly-EPO instantiates this recipe with a polychromic objective: the product of average task reward and reasoning-strategy diversity, with diversity estimated through LM-judge clustering. This can provide positive learning signals to unsuccessful responses that contribute complementary strategies.

Critical strengths
  1. A plausibly practical recipe for set-based objectives: Reusing a pool of responses to construct subsets and then computing marginal advantages makes set-level credit assignment compatible with standard policy-optimization pipelines. In the main configuration, the 70 subsets reuse eight sampled responses rather than requiring 70 separate rollout groups.
  2. The pass@k improvements are accompanied by majority-vote evaluations, which provide stronger evidence of practical utility than coverage alone. GRPO+DIV also uses the same clustering judge, making it a relevant control for whether the particular use of diversity information matters. This shows that the resulting diversity can be useful.
Critical weaknesses
  1. The incremental benefit over established multi-sample optimization is not demonstrated. The central motivation—improving exploration and the collective utility of multiple responses rather than concentrating on individual-response reward—is already directly addressed by pass@k-oriented training. In particular, Walder and Karkhanis, Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems (NeurIPS 2025), optimize this collective utility through jointly transformed rewards. The paper acknowledges this line of work in Appendix A, but the main experiments compare only with GRPO and the authors' additive GRPO+DIV baseline; the synthetic experiments do not close this gap either.
  2. The current comparisons do not cleanly attribute the gains to the proposed objective. Appendix B specifies per-response length normalization for GRPO, but fixed maximum-length normalization for Poly-EPO. This changes the relative weighting of long and short trajectories in addition to changing the advantage estimator. Consequently, the GRPO comparison cannot isolate the contribution of marginal set advantages and the polychromic objective.
Questions for authors
  1. Do you have a controlled comparison with PKPO under the same model, data, rollout budget, and optimization settings? A representative subset of the existing benchmarks, reporting pass@1, pass@k, and majority-vote accuracy, would be sufficient to materially inform my assessment.
  2. What length normalization is used for GRPO+DIV? Why do GRPO and Poly-EPO use different normalizations, and do the principal gains persist when they use the same normalization with comparable tuning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.