- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MOER, a mix-policy training method for LLM reasoning. It tries to balance the learning of expert's trajectory and the policy model's own exploration with three methods:
- Gradually shorten the expert's prefix with cosine annealing schedule. 2: roll back to the full expert trace when a prefix-conditioned completion is incorrect. 3. uses GRPO for on-policy tokens and a trust-region SFT loss for off-policy tokens. The paper tests MOER on Qwen2.5-Math and Qwen2.5-Instruct across math, general reasoning, and coding tasks, with stability, ablations, and hyperparameter studies.
Critical strengths
-
The storyline is coherent: starting with the issues of reduced exploration and bad completions that arise from expert guidance, the paper introduces prefix decay, rollback, and a harmonized objective. The mapping between the problems, the proposed methods, and the experimental validation is clear
-
The experiments are comprehensive, covering many model scales and various tasks. And the paper also provides clear ablation studies.
-
The discussion of the completion distribution is interesting, and the ablation shows clear instability when completions are treated as on-policy tokens.
Critical weaknesses
-
The novelty feels partly like a combination of existing mix-policy, prefix, and trust-region ideas.
-
Table 3 gives some qualitative explanations for why PG, SR, and COff may interact, but these are mainly qualitative, post-hoc mechanistic explanations. Since the results appear to rely on a single experimental run, it is difficult to rule out randomness in the reported outcomes. Does the full combination provide a stable improvement? Do the differences attributed to the individual components exceed training noise?
Questions for authors
-
What are the completion failure rate and rollback rate during the training?
-
See Weakness-2
2 comments
Thank you for the clarification. The additional explanation addresses my main concern, and I have decided to raise my score.
Thank you for clarifying your remaining concern. We agree that small differences between adjacent rows in Table 3 cannot establish each component's contribution in isolation. The ablations should therefore be read together with the targeted controls and training dynamics, since each component addresses a distinct failure mode.
Together, HO stabilizes optimization, PG balances guidance and exploration, SR guarantees correctness, and COff aligns completion updates with their distribution. We will revise Section 5.4 and Table 3's discussion to state these component-specific conclusions explicitly and cite the corresponding curves and controls alongside the average-score differences.