acceptodds
Under review as a conference paper at ICLR 2027

MInTRL: Off-policy Intervention can boost On-policy RL

Abstract

Reinforcement learning with verifiable rewards is typically performed on-policy, keeping training data close to the current policy but limiting learning to trajectories that the policy can discover itself. Off-policy methods such as supervised fine-tuning, on the other hand, can leverage external knowledge beyond the base model's capabilities, but may suffer from large distribution shift. The key challenge is thus to expand exploration without sacrificing learnability. In this work, we introduce Minimal Intervention Reinforcement Learning (MInTRL), which expands the exploration frontier through sparse, local interventions in otherwise on-policy rollouts. During generation, a judge-intervention policy periodically reviews the current policy's output, replaces erroneous suffixes with short corrections, and immediately returns control to the policy. During training, MInTRL adopts a sequence-level advantage-regression objective that eliminates the need for importance sampling. We show that sparse, local interventions can substantially improve coverage beyond finite-budget on-policy sampling while preserving the overall on-policy nature of the resulting trajectories. Across math and code benchmarks, MInTRL consistently outperforms standard on-policy and off-policy baselines. Ablations show that MInTRL remains effective with self-intervention and across different judge policies, while performance peaks at moderate intervention intensity, highlighting the importance of intervening minimally. These results establish minimal intervention as an effective paradigm for enhancing on-policy RL.

Primary area:
reinforcement learning
Keywords:
RL, LLM reasoning, distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2Re5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MInTRL, a semi-on-policy RLVR framework that adds sparse, local corrections to the policy’s own rollouts. During generation, a judge–intervention model periodically reviews the policy’s output. When it finds an error, it truncates the trajectory at the first erroneous step, writes a short correction of at most 64 tokens, and returns control to the policy. To learn from these mixed-provenance trajectories without importance sampling, the method uses a sequence-level advantage-regression objective derived from KL-regularized RL, with the baseline estimated only from pure on-policy control rollouts. Intervention tokens are anchored either to the current policy’s log-probabilities (MInTRL-Proxy) or to a constant (MInTRL-Const). A stylized support analysis motivates a coverage–learnability trade-off in the number of corrections. Experiments use Qwen3-1.7B and Qwen3-4B in non-thinking mode, compared against GRPO, SFT+GRPO, on-policy distillation and MENTOR. MInTRL-Const achieves the best average on math and code in all four settings. Ablations show that performance peaks when about 2–4% of tokens come from the intervention model.

Critical strengths
  1. Well-motivated question. The paper addresses a real tension in RLVR: on-policy training is stable but limited to what the policy itself samples, while off-policy data adds knowledge but shifts the distribution.
  2. Simple, practical method. Corrections are local and control returns to the policy immediately, so the method is easy to add to existing RLVR pipelines. During the intervention phase it runs 2.0–2.76× faster than MENTOR.
  3. Large gains in the tested setting. On Qwen3-1.7B in non-thinking mode, MInTRL-Const beats the strongest baseline on math by 9.44 pp.
  4. Useful empirical observation. Fig. 5 suggests that moderate intervention helps and excessive intervention hurts.
Critical weaknesses
  1. The sources of the gains are not sufficiently isolated. Compared with GRPO, Table 1 changes both the objective and rollout construction. The no-intervention regression baseline appears only as the relative zero of Fig. 5, without absolute scores across the main settings. The experiments do not separate the contribution of teacher-authored corrections from judge-guided truncation, or establish whether online local intervention offers benefits over a simpler assisted warm start.
  2. The theory provides limited explanation of the empirical results. The learnability threshold ϵtrain\epsilon_{train} is assumed rather than derived from the training objective. Theorem 3.5 bounds the visible support under sparse optimal corrections, but does not guarantee additional successful coverage or characterize how intervention affects subsequent policy updates. In particular, it does not explain why performance falls roughly 5 pp below the no-intervention baseline at around 40% intervention in Fig. 5. The observed trend is consistent with the proposed trade-off, but does not establish that learnability limits cause the decline.
  3. The comparisons with prior work and the evaluation scope are limited. MENTOR is the only intervention-based baseline evaluated directly, leaving the relative benefits over other approaches to learning from expert-guided or corrected trajectories unclear. Since both guided rollouts and advantage-regression objectives build on prior work, the distinct contribution of their combination needs clearer positioning and targeted comparisons. Evaluation is also restricted to Qwen3 students with at most 4B parameters, leaving generalization across model families and scales untested.
Questions for authors
  1. How does the assumed learnability threshold ϵtrain\epsilon_{train} relate to the optimization dynamics of the regression objective? What evidence links the performance decline under heavy intervention in Fig. 5 to learnability limits?
  2. To isolate the source of the gains, please report: (a) the absolute score of the no-intervention regression baseline in each Table 1 setting; (b) a judge-only variant, where the policy resamples from the truncated prefix and no teacher tokens are added; (c) a variant without the judge, where teacher tokens are inserted at random step boundaries at a matched fraction; and (d) GRPO trained on the same semi-on-policy rollouts, with ratios computed against π_t.
  3. How large is the gap to GRPO at step 200, compared with step 500? Would rejection-sampling fine-tuning on the successful semi-on-policy rollouts, followed by the same GRPO phase, match MInTRL?
  4. Does the four-review budget concentrate intervention within the first roughly 2k student-generated tokens for math and 512 for code? How do truncation and corrective continuations affect where interventions occur in the completed responses?
3 comments

We thank the reviewer for the careful reading and the detailed, constructive feedback.

W1 / Q2. Isolating the sources of the gains.

We thank the reviewer for these concrete suggestions. Since MInTRL differs from GRPO in both the rollouts and the objective, we change one factor at a time:

  • (a) keeps the regression objective but removes all interventions;
  • (b) keeps the judge's truncation but removes the teacher tokens;
  • (c) keeps the teacher tokens but replaces the judge's placement with random step boundaries at a matched intervention rate;
  • (d) keeps the intervened rollouts but replaces regression with the GRPO objective.

We ran these on Qwen3-1.7B math. Due to compute limits we could not run every setting of Table 1.

(a) and (d): trained variants, Pass@1 (mean@32, AIME25/AIME26/HMMT25 average) at step 200, the end of the intervention phase.

VariantRolloutsObjectivePass@1
GRPOon-policyGRPO11.91
(a) Regression, no interventionon-policyregression18.75
(d) GRPO on semi-on-policy rolloutssemi-on-policyGRPO, ratios against π_t22.05
MInTRL-Constsemi-on-policyregression28.92

Both components contribute: interventions help under either objective (11.91 → 22.05, 18.75 → 28.92), regression helps on either type of rollouts (11.91 → 18.75, 22.05 → 28.92), and only the combination reaches 28.92.

(b) and (c): rollout variants, rollouts of the base policy on 1,024 training prompts (8 per prompt).

RolloutsReward
On-policy0.188
(b) Judge-only: truncate at the flagged step, policy resamples0.204
(c) Random step + teacher tokens, matched intervention rate0.241
MInTRL: judge + teacher tokens0.262

Judge-only yields only a small gain, so most of the improvement comes from the teacher-authored corrections rather than from truncation. Random placement at the same rate recovers part of the gain, but judge placement is significantly better (paired difference +0.021).

W2 / Q1. Theory and the decline under heavy intervention.

We thank the reviewer for the comments and agree that the learnability threshold is assumed rather than derived. We want to emphasize that learnability is not precisely defined by current theory; the threshold formalizes the empirical observation that policy optimization degrades as the training data become more off-policy.

Relation to the objective. Heavy intervention hurts the regression update in two ways, both driven by how improbable the corrected tokens are under πt\pi_t, which is the same quantity (νN\nu^N) that the learnability condition of Theorem 3.5 bounds:

  1. Anchor shift. for MINTRL-Const, each intervention token shifts the regressed log-ratio by β2(log⁡πt(ys)−κ)<0\beta_2(\log\pi_t(y_s)-\kappa)<0. With sparse corrections the shift is small, and the update follows the reward. Under heavy intervention it can exceed the advantage, so heavily corrected rollouts are reinforced whether or not they succeed.
  2. Squeeze effect. When a heavily corrected rollout receives a negative update (e.g., in MInTRL-Proxy, whose anchor has no shift), the negative gradient falls on many tokens that already have very low probability under πt\pi_t. Such updates squeeze probability mass toward the policy's most likely tokens, leading to training collapsing [1].

Beyond Fig. 5, three observations point the same way:

  1. Prior work: off-policy RL with stale data first improves and then collapses as the data drift away from the policy [2], and learning from expert traces requires explicit policy shaping [3].
  2. Our fully off-policy baseline, SFT+GRPO, trains on complete teacher solutions and thus exposes the most correct trajectories, yet it is below MInTRL-Const as shown in Table 1.
  3. A stronger judge–intervention model from a different family (DeepSeek-V4-Flash, Appendix G) does not consistently help, consistent with its corrections being less likely under the Qwen policy.

W3. Comparisons with other intervention baselines

Due to compute constraints, we are unable to re-run all baselines. Instead, we compare with the reported results of concurrent works that use the same student–teacher pair (Qwen3-1.7B student, Qwen3-4B-Instruct-2507 teacher) to improve on on-policy distillation. Their reported AIME results compare as follows:

Method (reported numbers)AIME25AIME26
Relay-OPD [4]32.8130.52
MAESTRO [5]33.1330.10
SCOUT [6]28.80—
MInTRL-Const40.7341.56

With the same student–teacher pair, MInTRL outperforms all concurrent methods.

Q3. Comparisons at step 200 and RFT.

We thank the reviewer for the question. The table below reports Pass@1 at step 200 on Qwen3-1.7B math:

MethodPass@1
Base model8.02
RFT (SFT on successful semi-on-policy rollouts)9.86
GRPO11.91
MInTRL-Const28.92

At step 200, MInTRL-Const outperforms GRPO by 17.0 pp, whereas RFT on the successful semi-on-policy rollouts improves over the base model by only 1.8 pp. Due to compute constraints, we could not continue the RFT run to step 500 with the subsequent GRPO phase. However, since RFT barely improves over the base model, the subsequent GRPO phase would start from nearly the same policy as GRPO itself, and we therefore do not expect RFT followed by GRPO to match our algorithm performance.

Q4. Where interventions occur.

Yes the four reviews cover roughly the first 2k tokens for math and 512 for code. This design is chosen for efficiency: (1) early interventions are the most informative, since an early correction steers the direction of the entire rollout; (2) each review requires the teacher to process the full prefix, so restricting reviews to the early part of the response keeps the teacher's KV cache and prefill cost small.
In our experiments, since truncated suffixes are discarded and each correction adds at most 64 tokens, interventions mainly stay within roughly the first 2k tokens of the completed response.

[1] Yi Ren and Danica J. Sutherland. Learning Dynamics of LLM Finetuning. ICLR 2025.

[2] Haizhong Zheng, Jiawei Zhao, Beidi Chen. Prosperity before collapse: How far can off-policy RL reach with stale data on LLMs? ICLR 2026.

[3] Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, Yue Zhang. Learning to reason under off-policy guidance. NeurIPS 2025.

[4] Haolei Xu et al. Pass the Baton: Trajectory-Relayed On-Policy Distillation. arXiv:2607.26057, 2026.

[5] Yuhao Wang et al. From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillation. arXiv:2609.37510, 2026.

[6] Langlin Huang et al. On the Off-Policy Teacher in On-Policy Distillation. arXiv:2609.38360, 2026.


We would be happy to clarify any remaining concerns during the discussion period. We would greatly appreciate it if you could consider increasing the score once all concerns have been resolved.

Dear Reviewer 2Re5,

Since the Author/Reviewer discussion period concludes on Nov. 18, we would be happy to address any remaining concerns or questions.

Thanks!

Reviewer_sJZMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to incorporate off-policy guidance into on-policy RL while minimizing distribution shift. It proposes MInTRL, which uses a judge model to locally correct errors in student-generated trajectories and optimizes them through advantage regression.

Critical strengths
  1. The paper studies an important problem in RL training and the proposed method is intuitive and practical. Sparse local corrections allow the student to retain control over most of the generation process.

  2. The experiments show consistent improvements across models and benchmarks. The analysis of intervention intensity is interesting, showing that moderate intervention works better than excessive intervention.

Critical weaknesses
  1. Prior work such as MENTOR has already explored selective expert guidance in RL rollouts, while other methods have studied local error correction and advantage regression. The main contribution seems to be the combination of these existing techniques.

  2. The theoretical analysis assumes that interventions help explore trajectories that are difficult to sample but still learnable. However, the experiments mainly show improvements in downstream performance without directly verifying this mechanism.

  3. The paper does not isolate the contribution of advantage regression by comparing different optimization objectives on the same intervention-generated trajectories.

Despite these concerns, I find the problem important and the proposed method practical, with promising experimental results. Hence, my final rating is Weak Accept.

Questions for authors
  1. In Figure 1, the advantage regression formula seems to omit the coefficient β\beta compared with Equation 6.

  2. In Algorithm 1, the judge returns KEEP or REVISE, but the error position ee is used later without being explicitly returned.

  3. Appendix I shows that intervention is used for the first 200 training steps, followed by standard GRPO for the remaining 300 steps. How much does each phase contribute to the final improvement?

2 comments

We thank the reviewer for the evaluation and the thoughtful, constructive feedback.

W1. Novelty compared to existing works

We thank the reviewer for this comment. We would like to clarify that the two components of MInTRL are not combined arbitrarily, but are designed for each other. First, our sparse online intervention produces semi-on-policy rollouts: mostly tokens on policy, plus a few off policy tokens. Standard policy-gradient objectives handle such trajectories poorly, because the teacher tokens require behavior-policy importance weights, which are either unavailable for a black-box teacher or extremely small under the current policy. Our regression objective estimates its baseline only from on-policy control rollouts and uses a relaxed anchor for intervention tokens, so it learns from these trajectories without any importance weighting. This pairing, rather than either component alone, is what lets MInTRL turn sparse external corrections into a stable on-policy learning signal.

W2. Direct evidence for the mechanism.

We thank the reviewer for this suggestion. The mechanism has two parts: interventions expose correct trajectories that the policy rarely samples, and these trajectories remain learnable. We added direct tests of both parts (Qwen3-1.7B, math, 1,024 training problems per probe):

  1. Interventions expose correct trajectories that the policy rarely samples. With the base policy, intervened rollouts are correct more often than on-policy rollouts on the same problems (reward 0.262 vs. 0.188, gain +0.075 [0.062, 0.087]). Of the 450 problems that the base policy fails in all 8 on-policy samples, 193 (43%) are solved at least once with intervention.
  2. These trajectories are learnable. We take the tasks that the base policy fails in pass@8 but solves at least once with intervention (121 problems in the training pool). After the 200-step intervention phase of MInTRL, the policy's own pass rate on them (fresh samples, no intervention) rises from 8.3% to 42.0%, against 16.4% for GRPO. Problems that were solvable only with intervention become solvable without it.
  3. The training dynamics are consistent with this. Over the intervention phase, the forward KL on intervention tokens decreases and the reward gap between intervened and on-policy rollouts narrows toward zero (Fig. 4b). The policy increasingly produces on its own what the interventions supplied.

W3. Isolating advantage regression.

We thank the reviewer for this suggestion and ran exactly this comparison. We use the same semi-on-policy rollout construction as MInTRL, but optimizes the GRPO objective instead of advantage regression. All other settings are identical, so only the objective differs. Results at the end of the intervention phase (step 200, Pass@1, mean@32):

AIME25AIME26HMMT25Avg.
GRPO objective27.7124.2714.1722.05
Regression objective32.7133.3320.7328.92

It suffices to see that advantage regression outperforms GRPO objective on every benchmark.

Q1. We thank the reviewer for the comment. Yes the coefficient β\beta is missing and we will fix it in the revised version.

Q2. We thank the reviewer for catching this. We will update line 4 of Algorithm 1 to "π_JI reviews c and returns (a, e), a ∈ {KEEP, REVISE}".

Q3. The intervention phase accounts for most of the improvement. For example, on 1.7B math, MInTRL-Const achieves 28.92 by the end of Phase 1 and 35.45 by the end of Phase 2, while the base model scores 7.40. The training dynamics show the same pattern: on math, the training reward of MInTRL-Const rises steeply during the intervention phase and stays roughly flat during the subsequent GRPO phase, as shown in Fig. 3.


We would be happy to clarify any remaining concerns during the discussion period.

I appreciate the authors' efforts in addressing my concerns and am happy to maintain my original rating.

Reviewer_2isBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MInTRL introduces short teacher corrections into otherwise student-generated rollouts. It trains on these mixed trajectories using an advantage-regression objective, then switches to standard on-policy RL. Experiments on math and code compare different intervention strengths, teacher models, and reference anchors.

Critical strengths
  1. Sparse local correction is a sensible way to introduce external guidance while letting the student generate most of the trajectory.

  2. The results are promising, especially for the smaller student.

Critical weaknesses
  1. Checkpoints are selected on the reported evaluation benchmarks, introducing selection bias. Appendix E partly addresses this with final-checkpoint results, but the small gains over MENTOR also need training-seed uncertainty.

  2. The theory assumes optimal corrections and does not directly justify the practical mean-reward baseline or constant token anchor. Its connection to the implemented algorithm remains unclear.

Questions for authors

Please refer to the weakness part.

2 comments

We thank the reviewer for the careful reading and the insightful comments.

W1: Checkpoint selection and small gain over MENTOR

We thank the reviewer for the comment but respectfully disagree. In Appendix E, we report the final-checkpoint performance of every method, with no checkpoint selection. Specifically, on Qwen3-1.7B, MInTRL-Const improves over MENTOR at the final checkpoint by +11.60 pp on math (35.45 vs. 23.85, a 49% relative gain) and +11.21 pp on code (61.95 vs. 50.74). This is a large gain, not a small one.

Due to time and GPU constraints, we cannot repeat all experiments with multiple seeds. To gauge run-to-run variation, we reran MInTRL-Const twice on Qwen3-1.7B math for 200 steps, which is the end of the intervention phase (Pass@1, mean@32). Here is the results.

MInTRL-ConstAIME25AIME26HMMT25Avg.
Run 132.7133.3320.7328.92
Run 229.2729.4819.5826.11

The two runs differ by 2.8 pp on average, far less than the gains of MInTRL over MENTOR.

W2. Theory vs. practice.

We thank the reviewer and agree that the scope of the theory should be stated more clearly. The theory mainly answers one question: under a finite sampling budget, which correct solutions are visible to the current policy πt\pi_t, and how much can sparse corrections enlarge this set (i.e., Theorem 3.5). The mean-reward baseline and the constant anchor answer a different question: how the policy learns from these rollouts once they are collected. They are not derived from the theory. They are practical design choices that we justify empirically, and we will say so explicitly.

The theory also does not need the corrections to be optimal. The proof uses only one property: every corrected token ysy_s satisfies πt(ys∣x,y<s)≥ν\pi_t(y_s \mid x, y_{<s}) \ge \nu. The corrected rollout process μN\mu_N agrees with πt\pi_t everywhere except at no more than NN corrected positions, so every rollout yy it produces satisfies πt(y∣x)≥νNμN(y∣x).\pi_t(y \mid x) \ge \nu^{N} \mu_N(y \mid x). Any correct rollout that the corrections make visible is therefore at most a factor ν−N\nu^{-N} less likely under πt\pi_t. This holds whether or not the corrections are optimal, and also when the corrected tokens are sampled from πJI\pi_{JI} rather than fixed. If a correction is poor, the resulting rollout is simply incorrect and receives zero reward, and the support envelope still holds. We will therefore restate Assumption 3.4 in a weaker form.


We would be happy to clarify any remaining concerns during the discussion period. We would greatly appreciate it if you could consider increasing the score once all concerns have been resolved.

Dear Reviewer 2isB,

Since the Author/Reviewer discussion period concludes on Nov. 18, we would be happy to address any remaining concerns or questions.

Thanks!

Comments on the paper

1 comment

Dear reviewers,

The authors have now submitted their responses. Please review them and share your thoughts on whether your concerns have been addressed. I encourage you to engage with one another’s assessments, discuss any remaining disagreements, and update your reviews and scores as appropriate.

Thank you.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.