acceptodds
Under review as a conference paper at ICLR 2027

Policy Split: Incentivizing Dual-Mode Exploration in LLM Reinforcement with Dual-Mode Entropy Regularization

Abstract

To encourage diverse exploration in reinforcement learning (RL) for large language models (LLMs) without compromising accuracy, we propose Policy Split, a paradigm that bifurcates the policy into normal and high-entropy modes with a high-entropy prompt. While sharing model parameters, the two modes undergo collaborative dual-mode entropy regularization tailored to distinct objectives. Specifically, the normal mode optimizes for task correctness, while the high-entropy mode incorporates a preference for exploration, and the two modes learn collaboratively. Experiments across model sizes show consistent improvements in average reasoning accuracy together with entropy recovery. On creative-writing tasks, the trained high-entropy mode also produces more diverse outputs and improves several judged quality dimensions. Further analysis shows that the two modes develop distinct behavioral patterns and provide complementary learning signals. Code will be released.

Primary area:
generative models
Keywords:
large reasoning model, reinforcement learning, entropy regularization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qNjzOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Policy Split uses prompts to give one model a normal mode and a high-entropy mode. The normal mode is trained mainly for correctness, while the high-entropy mode also uses entropy and between-mode KL signals. The two modes learn from shared rollouts. The authors evaluate reasoning on Qwen3 1.7B, 4B, and 8B, and assess diversity and quality on creative writing. The results show higher average reasoning accuracy in the normal mode and more diverse outputs in the high-entropy mode.

Critical strengths
  1. Using different training objectives for reliable answers and exploration is intuitive. Sharing rollouts gives the normal mode a way to learn from correct answers discovered by the exploration mode.

  2. The experiments include several representative entropy-based baselines and ablations of the mode coefficients, between-mode signals, and rollout sharing. Creative writing and prompt-variant experiments provide additional evidence about the behavior of the two modes.

Critical weaknesses
  1. The paper includes some entropy-coefficient tuning and rollout-sharing controls, but the need for two modes remains unclear. A single-mode method with more extensive entropy tuning or an entropy schedule might achieve similar results.

  2. The average accuracy gains over the strongest entropy-based baseline are modest, and the difference at 1.7B is not significant. Meanwhile, the reported end-to-end training time is about 1.4 times that of GRPO. Although the experiments cover three sizes and creative writing, they use only Qwen3 models. It remains unclear whether the gains justify the extra cost and transfer to other model families.

Questions for authors
  1. With the same total training cost and a similar tuning budget, can a single-mode entropy-advantage method achieve comparable pass@1 and pass@k by tuning the entropy coefficient or its schedule? What evidence shows that separating the modes is necessary?

  2. Does Policy Split still outperform a strong single-mode baseline when both receive the same training-time budget? This matters given the reported 1.4× training cost.

Reviewer_mBg2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Policy Split, which uses system prompts to create normal and high-entropy modes within one LLM and trains them with mode-specific advantages: correctness-only for normal mode and correctness plus mode-conditioned entropy/KL regularization for high-entropy mode, with rollout sharing. It validates on Qwen3-1.7B/4B/8B across reasoning benchmarks, pass@k, mode KL, ablations, prompt robustness, and creative writing.

Critical strengths
  1. The dual-mode formulation cleanly decouples accuracy preservation from entropy-driven exploration, which is a sensible and practically relevant direction.
  2. The mode-KL and pass@k analyses provide useful evidence that the high-entropy mode explores differently and can improve large-k coverage.
Critical weaknesses
  1. The core claim of a controllable high-entropy mode is not sufficiently established: prompt-only activation is limited, HE-mode accuracy drops substantially, and deployment-time mode selection is not addressed.
  2. Gains over strong entropy baselines are small, and attribution to the mode-conditioned KL term versus rollout sharing and extra computation is unclear.
Questions for authors
  1. How does Policy Split compare with Share-GRPO + Entropy Advantage under matched compute, data, and rollout budgets?
  2. What is the oracle-reranking accuracy gain from HE-mode samples, and does the pass@k benefit hold at 1.7B/4B?
Reviewer_URYQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Policy Split, an RL post-training method that splits one LLM into two prompt-conditioned modes. The normal mode has no system prompt; the high-entropy (HE) mode is activated by a fixed natural-language system prompt. The two modes share parameters. The normal mode is trained with standard GRPO correctness advantages. The HE mode adds a clamped, detached bonus to its advantage, built from the token log-probabilities of both modes (bt=−αlog⁡πθHE−(1−α)log⁡πθb_t=-\alpha\log\pi^{HE}_\theta-(1-\alpha)\log\pi_\theta). This bonus is meant to raise HE-mode entropy and push it away from the normal mode. Rollouts are shared between the modes, following Share-GRPO. The authors train Qwen3-1.7B/4B/8B for one epoch on 20K DeepMath problems. The normal mode improves average accuracy over MATH-500, AIME-24/25, GPQA-Diamond, and MMLU-Pro by 0.7–1.75 points over the original models, and by 0.07–0.73 points over the best entropy baseline. Its entropy also rises. The HE mode diverges from the normal mode (KL up to 1.09), reaches the highest pass@8 (86.53 vs. 84.91 on 8B), and scores higher on LLM-judged creative-writing quality and diversity.

Critical strengths
  1. Placing the accuracy objective and the exploration objective in two prompt-conditioned modes of one model is an intuitive way to work around the accuracy–entropy trade-off. It needs no extra parameters, and the exploration mode can be switched on at inference.

  2. Controlled baseline comparison. Five entropy-related baselines are reimplemented under identical training settings across three model sizes. Creative writing is scored by three different LLM judges.

Critical weaknesses
  1. The accuracy gains are small, and the significance claims ignore training variance. Over the strongest baseline, the average improvement is +0.07 (1.7B, not significant), +0.73 (4B), and +0.65 (8B). The reported p-values and standard errors come from 8 evaluation samples of a single training run per method. They therefore capture decoding noise only, not run-to-run training variance. Plain GRPO barely changes these already-RL-trained models (−0.14, 0.00, +0.33), which suggests that differences under one point may lie within seed-to-seed variation. The largest per-benchmark gains are on AIME, which has only 60 problems.

  2. Ablations are run on the model where the main effect is not significant. All ablations (Tables 3–4) use Qwen3-1.7B, where Policy Split does not significantly beat Entropy Advantage. Most differences are 0.1–0.8 points from single runs. Without sharing, Policy Split (56.61) is below Entropy Advantage (56.71). The claim that the two modes "provide complementary learning signals" rests on the +0.17 gain from sharing.

  3. Novelty is incremental. The clamped detached entropy advantage (κ=2) comes from Cheng et al. (2025), the log-probability entropy estimate from Yao et al. (2025b), and rollout sharing from Share-GRPO. The new element is the prompt-conditioned mode with the inter-mode KL term. Early in training, however, the two modes are nearly identical (KL ≈ 0.01), so bt≈−log⁡πθHEb_t\approx-\log\pi^{HE}_\theta: essentially the same surprisal signal as an entropy advantage. The isolated effect of the KL term is shown only by the α ablation on 1.7B (55.98 vs. 56.78, single run).

  4. The objective is under-specified and partly unjustified. Entropy and KL are estimated from a single sampled token's log-probability, which is high-variance and can be negative for KL. Shared rollouts are scored under the other mode's ratio, e.g. πθ(oHE∣q)/πθold(oHE∣q)\pi_\theta(o^{HE}\mid q)/\pi_{\theta_{old}}(o^{HE}\mid q), even though they were sampled from πθold(⋅∣s,q)\pi_{\theta_{old}}(\cdot\mid s,q). This is not a valid importance ratio, and the resulting bias is not discussed. With sharing, it is unclear which rollouts form each advantage group. Eq. 8 is malformed (the summation and the factor GG are garbled), so the exact algorithm cannot be reconstructed from the paper.

  5. Training compute is not matched to the baselines. Policy Split uses 2× the modeling compute and about 1.4× the wall-clock time of GRPO. No baseline receives a comparable budget, for example two gradient passes per batch or 16 rollouts per prompt. Training is also short: one epoch of 20K problems at batch size 256, about 78 steps. Robustness over longer training, where entropy-bonus methods often become unstable, is untested.

  6. The creative-writing evidence is weak and confounded.

    • Only 13 prompts are used, and no confidence intervals are reported.
    • The HE prompt explicitly asks for diverse thinking, yet Original + HE prompt and GRPO + HE prompt are not evaluated. The prompt's effect therefore cannot be separated from the training's effect.
    • HE-mode responses are much longer (ΔLen ≈ 1.5k tokens on 8B), and LLM judges are known to favor length, but length is not controlled.
    • The normal mode of Policy Split does not beat the original model under the GPT-5 judge.
    • An anomalous DeepSeek-V3 imagery score for the normal mode (75.88 vs. ~84 for all others) is not explained.
  7. The pass@k analysis lacks the obvious control. The HE mode's higher pass@8 is compared only with models sampled at T=0.6. Without the original, GRPO, or Entropy Advantage models sampled at higher temperature (e.g., T=1.0), and without a 4 normal + 4 HE sample mix, the claim that the HE mode "broadens exploration" beyond what temperature provides is not established. This claim is made explicitly for creative writing but never tested for reasoning.

  8. Entropy recovery is treated as a goal without justification. The paper presents higher normal-mode entropy as a contribution, but does not show that it causes better accuracy or exploration. Entropy Advantage reaches comparable or higher entropy (0.84 vs. 0.82 on 1.7B) with similar accuracy.

Questions for authors
  1. Can you report at least 3 training seeds per method (at least Policy Split, GRPO, and Entropy Advantage on 4B and 8B) with mean ± std? Please also specify the statistical test behind the p-values and its unit of analysis. If the gains over Entropy Advantage are within seed variance, the main claim would need to be reframed.

  2. Does the improvement persist against compute-matched baselines, i.e., GRPO and Entropy Advantage given the same 2× modeling budget, and with longer training (several epochs)?

  3. For creative writing, can you add Original + HE prompt and GRPO + HE prompt, use length-controlled or length-normalized judging, enlarge the prompt set with confidence intervals, and explain the DeepSeek-V3 imagery anomaly?

  4. How does the HE mode's pass@k compare to the original, GRPO, and Entropy Advantage models sampled at T=1.0, and to a 4 normal + 4 HE sample mixture?

  5. Can you precisely specify the sharing implementation: which rollouts form each advantage group, which importance ratio is used for rollouts sampled from the other mode, and how this bias is handled? What is the effect of the inter-mode KL term (α=0.5 vs. α=1.0) on 4B and 8B?

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your time and thoughtful reviews. Please read the authors’ response and share your follow-up comments promptly, indicating whether your concerns have been addressed. Please raise any additional questions early enough to give the authors sufficient time to respond.

Thank you.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.