acceptodds
Under review as a conference paper at ICLR 2027

MOER: Mix-Policy Optimization under Expert Prefix Guidance with Selective Rollback

Abstract

Mix-policy optimization incorporates off-policy expert data into the reinforcement learning (RL) loop to transcend the capability boundary of on-policy RL. However, existing methods either provide static full-trace guidance that suppresses self-exploration, or adopt expert prefix guidance without correctness guarantees, leading to suboptimal performance or even training collapse. In this paper, we propose MOER, a mix-policy optimization framework that harmonizes expert guidance with self-exploration. MOER injects a decaying expert chain-of-thought prefix into the prompt and lets the model complete the remaining reasoning, with the prefix length progressively annealed to encourage self-exploration. A selective rollback mechanism replaces failed completions with the original full expert trace, guaranteeing the correctness of every off-policy trajectory. Furthermore, a harmonized mix-policy objective optimizes on-policy tokens with a GRPO loss while supervising both expert prefix and completion tokens with a trust-region SFT loss, substantially stabilizing training. Extensive experiments show that MOER achieves the highest average scores in mathematical and general domain reasoning tasks, outperforming all basic and mix-policy baselines, and generalizes consistently across diverse tasks, model scales, and base model categories.

Primary area:
optimization
Keywords:
optimization, reinforcement learning, supervised fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_3ZoAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MOER, a mix-policy training method for LLM reasoning. It tries to balance the learning of expert's trajectory and the policy model's own exploration with three methods:

  1. Gradually shorten the expert's prefix with cosine annealing schedule. 2: roll back to the full expert trace when a prefix-conditioned completion is incorrect. 3. uses GRPO for on-policy tokens and a trust-region SFT loss for off-policy tokens. The paper tests MOER on Qwen2.5-Math and Qwen2.5-Instruct across math, general reasoning, and coding tasks, with stability, ablations, and hyperparameter studies.
Critical strengths
  1. The storyline is coherent: starting with the issues of reduced exploration and bad completions that arise from expert guidance, the paper introduces prefix decay, rollback, and a harmonized objective. The mapping between the problems, the proposed methods, and the experimental validation is clear

  2. The experiments are comprehensive, covering many model scales and various tasks. And the paper also provides clear ablation studies.

  3. The discussion of the completion distribution is interesting, and the ablation shows clear instability when completions are treated as on-policy tokens.

Critical weaknesses
  1. The novelty feels partly like a combination of existing mix-policy, prefix, and trust-region ideas.

  2. Table 3 gives some qualitative explanations for why PG, SR, and COff may interact, but these are mainly qualitative, post-hoc mechanistic explanations. Since the results appear to rely on a single experimental run, it is difficult to rule out randomness in the reported outcomes. Does the full combination provide a stable improvement? Do the differences attributed to the individual components exceed training noise?

Questions for authors
  1. What are the completion failure rate and rollback rate during the training?

  2. See Weakness-2

2 comments

Thank you for clarifying your remaining concern. We agree that small differences between adjacent rows in Table 3 cannot establish each component's contribution in isolation. The ablations should therefore be read together with the targeted controls and training dynamics, since each component addresses a distinct failure mode.

  1. Harmonized objective (HO): stabilizing mixed-policy optimization. HO bounds and scales off-policy update weights while retaining clipped GRPO for on-policy tokens. It changes LUFFY from 48.4/57.3 to 48.6/58.3 (Math/General). Although the Math gain is small, Figure 4 shows reduced entropy and response-length fluctuations and less late-stage instability, supporting HO's primary role as a stabilization mechanism.
  2. Decaying prefix guidance (PG): preserving exploration as guidance is withdrawn. PG supplies useful reasoning contexts early and gradually shifts learning toward self-generated reasoning. Compared with full expert guidance (+HO-FG+SR; 47.1/57.7), full MOER achieves 49.1/59.0 and improves all nine benchmarks. Figure 4(b) further shows higher late-stage entropy, while Figure 9 shows a narrowing accuracy gap between on-policy rollouts and prefix-conditioned completions. These results support PG's role in maintaining exploration and internalizing expert-supported reasoning.
  3. Selective rollback (SR): preserving a correct learning signal. Our additional discard-failed-completion experiment directly tests whether SR is merely a filtering mechanism. This variant removes failed prefix-conditioned completions, while groups with correct completions remain unchanged. As shown in the third table in the general response, it exhibits late-stage entropy collapse and underperforms rollback, achieving 48.7/58.4 versus 49.1/59.0. Discarding a failure leaves the update dependent on the remaining on-policy rollouts. Rollback instead replaces the failure with a verified expert trace, preserving both a correct trajectory and a useful relative-advantage signal. Consistent with this mechanism, removing SR entirely (+HO+PG+COff) causes training collapse and reduces Math to 46.2 (Figure 5). SR thus supplies a correctness guarantee that filtering alone cannot provide.
  4. Completion as off-policy tokens (COff): matching the objective to the data distribution. Expert prefixes change the sampling context, and SR introduces success-conditioned selection. Figure 3 accordingly shows lower probabilities for completion tokens than for purely on-policy tokens. Treating completions as on-policy (+HO+PG+SR) yields 48.9/58.7 versus 49.1/59.0 for MOER, and Figure 4(c) shows late-stage instability and excessive response-length growth. The small score increment alone does not capture COff's role in handling this distribution mismatch.

Together, HO stabilizes optimization, PG balances guidance and exploration, SR guarantees correctness, and COff aligns completion updates with their distribution. We will revise Section 5.4 and Table 3's discussion to state these component-specific conclusions explicitly and cite the corresponding curves and controls alongside the average-score differences.

Thank you for the clarification. The additional explanation addresses my main concern, and I have decided to raise my score.

Reviewer_enQxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MOER (Mix-policy Optimization under Expert prefix guidance with selective Rollback). MOER is a mixed-policy training framework combining (1) decaying expert prefixes, (2) a selective rollback mechanism that replaces failed completions with full expert traces, (3) separate GRPO and trust-region SFT learning objectives. This method aims to establish a better balance between autonomous self-exploration and expert guidance. Experiments across mathematical, general, and coding tasks report improved performance and training stability over existing baselines.

Critical strengths
  • This paper focuses on a practical challenge: integrating expert supervision into RL while preserving exploration and training stability. It is a meaningful direction to improve the effectiveness of RL algorithms.
  • The framework designed by this paper is simple and general (especially the selective rollback mechanism), which can be readily integrated into existing mixed-policy training pipelines.
  • Experiments involve multiple benchmarks and OOD evaluation.
Critical weaknesses
  • Limited novelty and contribution of expert prefix decay: This paper claims that decaying expert prefix guidance is one of the three key designs of MOER. However, Prefix-RFT[1] already reduces the expected expert prefix length through cosine scheduling of the sampling interval. Although MOER directly anneals the prefix ratio, the conceptual distinction and empirical benefit of this scheduling variant remain insufficiently established neither theoretically nor experimentally.
  • Incomplete analysis of rollback mechanism: I think the proposed selective rollback mechanism needs more detailed analysis, as there are other simple approaches such as discarding failed samples directly. Besides, rollback mechanism may also change the advantage of other rollouts in the same prompt group. For example, suppose the original reward distribution for a prompt group was [0,0,0,0][0,0,0,0]; if a rollback occurs and replaces one of these with an expert trajectory that has a reward of 1, the reward becomes [0,0,0,1][0,0,0,1], and the corresponding advantage also changes. Therefore, the rationale behind the performance gains may not stem entirely from ensuring the correctness of off-policy samples.

Typos:

  • p.9, L465: furhter supports
  • p.4, L187: CosAneal

[1] Huang, Z., Cheng, T., Qiu, Z., Wang, Z., Xu, Y., Ponti, E., & Titov, I. (2025). Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling. ArXiv, abs/2507.01679.

Questions for authors
  • Could the authors further elaborate on the innovation boundaries of MOER, either through theoretical analysis or supplementary experiments?
  • The baseline performance for TRAPO fails completely. And Section 5.1 states that the best-performing checkpoint is selected for evaluation. However, the original TRAPO paper reports competitive results using the same Qwen2.5-Math-7B model. Could the authors provide further clarification regarding this discrepancy?
3 comments

Comment 2 [cont.]

Our additional experiment isolates the failed-completion case. The variant discards failed completions, so those groups are updated from on-policy rollouts alone, while groups with a correct completion remain identical to MOER. The first row below is the no-rollback ablation in Table 3. We find that discarding failed completion leads to entropy collapse in the later stage of training, thereby underperforming selective rollback.

Failed completion handlingMathGeneralEntropy
Keep failed completion (no rollback)46.258.30.17
Discard failed completion48.758.40.04
Rollback (MOER)49.159.00.25

Dear reviewer,

We hope our rebuttal addresses your concerns and would greatly appreciate it if you could review our responses. If any questions remain, we would be happy to provide further clarification. In the following, we would like to further clarify some of your concerns.


Comparison with Prefix-RFT and methodological contribution

As further discussed in our response to Reviewer iFYE, MOER's contribution lies in jointly addressing incorrect prefix-conditioned completions, distribution shift, and the transition from expert imitation to self-exploration. Selective rollback guarantees a correct off-policy trajectory. Treating completion tokens as off-policy accounts for the distribution shift, while decaying prefix guidance progressively reduces expert reliance. Our analysis in Appendix A also provides an insight for off-policy objective design: effective gradient weights should be bounded, properly scaled, and decrease with token probability.

Under our common reproduction protocol, the controlled ablations further distinguish these contributions. MOER without selective rollback achieves 46.2/58.3 on Math/General (Table 3), while MOER with the Prefix-RFT objective achieves 46.9/57.1 (Table 5). Both outperform our Prefix-RFT reproduction (44.3/55.9), and full MOER performs best at 49.1/59.0. We also evaluate CHORD, a more recent baseline, which achieves 45.2/57.4: it outperforms our Prefix-RFT reproduction but remains below MOER. Please see our response to Reviewer iFYE for the detailed comparison.


TRAPO reproduction and objective controls

Since the original TRAPO checkpoint has not been released, we reproduced it under the same settings as the other methods. Under this shared setup, our TRAPO reproduction collapsed during training, with rapid response-length growth in the early stage.

We additionally tested two cross-over controls: MOER with TRAPO's objective and TRAPO with MOER's objective. Both variants collapsed under the same setting. These results indicate that stable training depends on the joint design of the objective and the prefix-guided rollout mechanism, rather than on either choice alone.

Thank you again for your time and constructive feedback. If the additional results and explanations address your concerns, we would appreciate your reconsideration of the score.

Best regards,
The authors

Dear Authors,

Thank you for your detailed responses and additional experiments. I appreciate your efforts to clarify the methodological contributions and provide further evidence for selective rollback.

While the rebuttal addresses some of my concerns, I remain unconvinced that the novelty relative to existing methods, particularly Prefix-RFT, has been sufficiently established. In addition, although the new ablations provide useful evidence, the explanation for TRAPO's training collapse and the resulting performance discrepancy remains insufficiently conclusive.

Therefore, I will maintain my original score.

Best regards, Reviewer

Reviewer_EWejOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+2)
Read review
Paper summary

This paper proposes MOER, which uses a decaying expert chain-of-thought prefix to balance model exploration against expert guidance. A selective rollback mechanism keeps the expert guidance correct, and the GRPO loss is further modified to ensure stability. The method outperforms the baselines on both math and general-domain benchmarks.

Critical strengths

The method is natural, the structure is clear, and the paper is easy to read.

Critical weaknesses

My main concern is about the core contribution:

  1. Mixing in an expert prefix to guide GRPO is almost a common trick. The authors need to think more deeply about what problem the paper actually solves.
  2. What distinguishes the cosine schedule from a fixed or fully random prefix length, and why was cosine chosen over other decay schedules?
  3. Does the quality of the prefix, i.e., the expert data itself, affect the guidance? For example, what happens when the expert data comes from models with different response styles?
  4. The paper lacks discussion of recent hybrid-policy work such as RL-PLUS[1], CalibRL[2], ReMix[3].

[1] Dong, Yihong, et al. "Rl-plus: Countering capability boundary collapse of llms in reinforcement learning with hybrid-policy optimization." 2026. [2] Huang, Zhuoxu, et al. "Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning." 2026. [3] Liang, Jing, et al. "Squeeze the soaked sponge: Efficient off-policy reinforcement finetuning for large language model." 2025.

Questions for authors

See weaknesses above.

Reviewer_iFYEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MOER combines reinforcement learning with expert demonstrations for language-model reasoning. It gives the model an expert prefix that becomes shorter during training, replaces failed prefix-conditioned completions with full expert solutions, and uses GRPO for ordinary rollouts and weighted trust-region supervision for expert-guided trajectories. Experiments cover several Qwen2.5 sizes, mathematical and general-reasoning benchmarks, and a separate coding task. On Qwen2.5-Math-7B, the method reports averages of 49.1 for mathematics and 59.0 for general reasoning, the highest among the reproduced baselines. The paper also examines training stability and the contribution of individual components.

Critical strengths
  • The experiments cover mathematical and general reasoning across several Qwen2.5 variants, with additional evaluations at 14B and on code generation.
  • The component ablations and training curves provide useful evidence about changes in accuracy, entropy, and response length during training.
Critical weaknesses
  • Prefix-RFT scores 44.3 here versus 51.8 in its original paper under the same stated model, benchmarks, and metrics, so differences in training and evaluation settings need to be reconciled before drawing conclusions about its limitations.
Questions for authors
  • What explains the gap between your Prefix-RFT result and the originally reported 51.8, and how does the original checkpoint perform under your evaluation pipeline?
  • Can you report results across independent training runs for MOER and ReLIFT to establish whether the 49.1 versus 48.6 mathematical performance difference is reliable?
3 comments

Dear reviewer,

We hope our rebuttal addresses your concerns and would greatly appreciate it if you could review our responses. If any questions remain, we would be happy to provide further clarification.

Thanks for the clarification, and I’ll keep my original score since my main concerns are still open.

Thank you for your continued engagement. We further clarify the comparison with Prefix-RFT and supplement our response to Comment 3 below.

1. Prefix-RFT reproduction, methodological differences, and controlled comparisons

The 44.3 score is from our own reproduction, while 51.8 is the original Prefix-RFT paper's reported result. The original checkpoint has not been released, and the public code does not include a complete training recipe. We therefore adopted the reported hyperparameters and aligned the remaining training and evaluation settings with LUFFY under our common protocol. Although the stated base model, benchmarks, and metrics are the same, the two absolute scores should not be interpreted as a strictly controlled comparison. Our claims below are accordingly scoped to the methods reproduced under our protocol.

A correctness check can be added to Prefix-RFT, but the resulting completions still differ from ordinary on-policy samples: the expert prefix changes the sampling context, while correctness filtering introduces success-conditioned selection. MOER addresses these coupled issues through the joint design of decaying prefix guidance, selective rollback, and a harmonized mix-policy objective. Selective rollback guarantees a correct off-policy trajectory. Treating completion tokens as off-policy accounts for the distribution shift, while decaying guidance gradually shifts learning from expert imitation toward self-exploration.

An insight for off-policy objective design. Our analysis in Appendix A compares different objectives through their gradient-weight curves rather than their surface loss forms. Scaled objectives nearly overlap in the high-token-probability regime and mainly differ on low-probability tokens. Effective stabilization requires gradient weights that are bounded, properly scaled, and decreasing with token probability. This principle motivates our scaled trust-region objective and provides practical guidance for designing objectives that combine on-policy and off-policy tokens.

Within this common protocol, the controlled comparisons below isolate the contributions of rollback and the objective. Both MOER variants outperform our reproduced Prefix-RFT on both average metrics, while full MOER achieves the best results:

Method / variantMathGeneral
Prefix-RFT (our reproduction)44.355.9
MOER without selective rollback (Table 3)46.258.3
MOER with the Prefix-RFT off-policy objective (Table 5)46.957.1
MOER49.159.0

The no-rollback variant shows that the other MOER designs improve over our reproduced Prefix-RFT under this protocol, although training becomes unstable. Replacing only our off-policy objective with that of Prefix-RFT also preserves some gains from rollback and prefix guidance, but remains below full MOER. Rollback and the harmonized objective therefore play complementary roles rather than duplicating the same effect.

We additionally evaluate CHORD, a more recent mix-policy baseline, under the same protocol. CHORD achieves 45.2/57.4 on Math/General, improving over our Prefix-RFT reproduction (44.3/55.9), while MOER remains higher at 49.1/59.0.

2. Additional evidence for Comment 3: MOER versus ReLIFT across training runs

The most direct evidence would be repeated same-model training with different seeds. As each run takes three days, this was not feasible within the rebuttal period. We instead examine independently trained runs on three base models, which test cross-setting replication rather than providing a seed-level significance estimate.

For the main Qwen2.5-Math-7B setting, the 49.1 versus 48.6 Math scores are already averages over six benchmarks with 1,590 test problems; AIME 2024, AIME 2025, and AMC are further averaged over 32 samples. This aggregation reduces evaluation noise, while the independent training runs below examine whether the advantage over ReLIFT recurs across settings.

Base modelMOER MathReLIFT MathMOER GeneralReLIFT General
Qwen2.5-Math-7B49.148.659.057.3
Qwen2.5-Math-1.5B38.134.140.637.1
Qwen2.5-Instruct-7B42.341.463.859.6

MOER outperforms ReLIFT in all six domain-average comparisons, with Math gains of +0.5, +4.0, and +0.9 on the three models, and wins 22 of 27 per-benchmark comparisons. These runs do not estimate same-model seed variance, but the consistent advantages show that MOER's improvement over ReLIFT is not isolated to the single 49.1 versus 48.6 comparison.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General responses

We thank the reviewers for their constructive feedback. In the following, we highlight our responses to the reviewers' comments:


Methodological contribution

1. An insight for off-policy objective design. As discussed in Appendix A, the off-policy losses used by different methods take many different forms. However, once we plot their gradient weight curves (Figure 6), they are essentially the same. Our experiments further show that, regardless of the form of the loss, training works as long as the gradient weight is bounded, properly scaled (generally, overall smaller than the gradient weight of SFT), and decreasing in the token probability, and the differences among such objectives are small. Based on this finding, we identified through experiments the objective that suits our method (the scaled trust-region objective, Table 5). We not only propose a loss objective that performs best empirically with the other designed MOER components, but also highlight a useful insight for objective design when both on-policy and off-policy tokens are considered, which we believe is a meaningful contribution.

2. Selective rollback. Expert prefix guidance is unstable without a correctness guarantee. Removing selective rollback leads to training collapse, with the off-policy score dropping to zero and entropy exploding (Figure 5), and lowers the Math average from 49.1 to 46.2 (Table 3). Rollback restores a correct off-policy trajectory at no additional computational cost.

3. Treating completion tokens as off-policy. The expert prefix steers the model into contexts that the current policy rarely reaches, and correctness filtering further introduces a selection bias. Consistent with this, completion tokens have substantially lower probabilities than pure on-policy tokens (Figure 3). Treating them as on-policy causes late-stage instability and excessively long responses (Figure 4(c)), whereas treating them as off-policy yields the best overall result (49.1/59.0 vs. 48.9/58.7).

Together, these designs account for the stable gains of MOER.


Additional baselines

MOER outperforms both added baselines under the same protocol.

MethodMathGeneralEntropyResponse length
MOER49.159.10.25~2500
RL-PLUS47.057.70.10~3200
CalibRL45.356.70.16~1400

Decay schedules

The three schedules produce similar results, so the cosine schedule is a simple default rather than the source of MOER's gains.

ScheduleMathGeneral
Cosine49.159.0
Linear49.059.2
Exponential49.158.0

Discarding failed completion

Discarding failed completion is identical to our method when the completion is correct, while degenerating to on-policy RL when it is wrong. The experiment results indicate that discarding failed completion underperforms rollback.

Failed completion handlingMathGeneralEntropy
Keep failed completion (no rollback)46.258.30.17
Discard failed completion48.758.40.04
Rollback (MOER)49.159.00.25

Paper revisions (highlight in blue)

  1. Add the RL-PLUS and CalibRL comparisons in Table 1 and update the related work correspondingly.

  2. Add the decay-schedule ablation in Appendix F.

  3. Add the result of discarding failed completion in Table 3.

  4. Revision to the contribution statement

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.