- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Policy Split uses prompts to give one model a normal mode and a high-entropy mode. The normal mode is trained mainly for correctness, while the high-entropy mode also uses entropy and between-mode KL signals. The two modes learn from shared rollouts. The authors evaluate reasoning on Qwen3 1.7B, 4B, and 8B, and assess diversity and quality on creative writing. The results show higher average reasoning accuracy in the normal mode and more diverse outputs in the high-entropy mode.
Critical strengths
-
Using different training objectives for reliable answers and exploration is intuitive. Sharing rollouts gives the normal mode a way to learn from correct answers discovered by the exploration mode.
-
The experiments include several representative entropy-based baselines and ablations of the mode coefficients, between-mode signals, and rollout sharing. Creative writing and prompt-variant experiments provide additional evidence about the behavior of the two modes.
Critical weaknesses
-
The paper includes some entropy-coefficient tuning and rollout-sharing controls, but the need for two modes remains unclear. A single-mode method with more extensive entropy tuning or an entropy schedule might achieve similar results.
-
The average accuracy gains over the strongest entropy-based baseline are modest, and the difference at 1.7B is not significant. Meanwhile, the reported end-to-end training time is about 1.4 times that of GRPO. Although the experiments cover three sizes and creative writing, they use only Qwen3 models. It remains unclear whether the gains justify the extra cost and transfer to other model families.
Questions for authors
-
With the same total training cost and a similar tuning budget, can a single-mode entropy-advantage method achieve comparable pass@1 and pass@k by tuning the entropy coefficient or its schedule? What evidence shows that separating the modes is necessary?
-
Does Policy Split still outperform a strong single-mode baseline when both receive the same training-time budget? This matters given the reported 1.4× training cost.
Dear Reviewers,
Thank you for your time and thoughtful reviews. Please read the authors’ response and share your follow-up comments promptly, indicating whether your concerns have been addressed. Please raise any additional questions early enough to give the authors sufficient time to respond.
Thank you.
Best,
AC