acceptodds
Under review as a conference paper at ICLR 2027

SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation

11 views

Abstract

On-policy distillation (OPD) improves student models by training them on trajectories induced by their own policy, making it a promising approach for mitigating exposure bias in agent training. However, most OPD studies focus on single-turn settings, while realistic LLM agents interact with environments over multiple turns. In this regime, early errors can alter future observations and compound across the trajectory, and standard dense token-level OPD becomes brittle, as it may over-penalize semantically valid alternatives, reinforce local degeneracies such as repeated actions, and propagate unreliable teacher supervision on off-distribution histories. We propose SAGE-OPD, a verifier-free selective intervention framework specifically designed for multi-turn OPD. Instead of applying teacher supervision uniformly, SAGE-OPD uses pre-execution validity checks and teacher judgment to decide whether each student response requires supervision. SAGE-OPD further weights the selected turns by teacher confidence, assigning greater relative weight to turns with more decisive teacher predictions. Finally, normalizing weights within each batch preserves the relative allocation of supervision while maintaining unit average effective token weight. With a Qwen3-1.7B student and Qwen3-8B teacher, SAGE-OPD raises ALFWorld unseen success from 61.94% to 70.15% and ScienceWorld success from 4.70% to 6.71% over standard OPD, while remaining competitive on SearchQA. Component ablations isolate intervention and confidence weighting; three-seed replications, expert-action comparisons, and shared-state continuation evaluations provide complementary evidence of robustness and agent behavior. These findings support feedback-conditioned turn-level supervision allocation for multi-turn OPD.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, LLM agents, multi-turn fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KTiQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SAGE-OPD, a selective intervention framework for multi-turn on-policy distillation. The main idea is that applying teacher supervision uniformly across all turns may not be optimal, especially when student actions are already reasonable or when the teacher's supervision may be unreliable.

The method uses deterministic validity checks and teacher judgments to categorize student responses into SKIP, WEAK, and STRONG intervention levels. These intervention weights are combined with teacher confidence, measured using average top-1 token probability, and normalized to determine the contribution of each turn to the distillation loss.

The authors evaluate SAGE-OPD on ALFWorld, ScienceWorld, and SearchQA using different student and teacher model sizes. The method shows improvements over standard OPD, particularly on ALFWorld and ScienceWorld. The paper also includes comparisons against other distillation methods, component ablations, shuffled-label experiments, statistical analysis, expert-action comparisons, and shared-state continuation evaluations to support the proposed approach.

Critical strengths

1. Well-motivated problem and a useful approach to multi-turn distillation.

The paper addresses an important limitation of standard OPD for multi-turn agents. Applying teacher supervision at every token may unnecessarily penalize valid alternative actions, and mistakes early in a trajectory can affect subsequent interactions. The idea of deciding when the student actually needs supervision, rather than treating all turns equally, makes sense. I also like that the approach preserves student-generated trajectories instead of replacing student actions with teacher actions.

2. Good experimental coverage and meaningful ablations.

The paper has a fairly comprehensive experimental evaluation. The authors compare against several OPD methods across multiple benchmarks and model configurations. The improvements on ALFWorld are meaningful, particularly the increase in unseen success from 61.94% to 70.15% with the 1.7B student and 8B teacher.

I also appreciate the effort to validate individual components. The intervention-only, confidence-only, NoConf, and shuffled-label ablations provide useful evidence that the proposed weighting scheme contributes to performance. The additional expert-action comparisons, statistical tests, and three-seed experiments with the 32B teacher make the evaluation stronger than a straightforward comparison of final success rates.

Overall, the experiments suggest that selectively allocating supervision is a promising direction for improving multi-turn agent training.

Critical weaknesses

1. It is still unclear whether the teacher is identifying the right turns for intervention.

This is my main concern with the paper. The authors show that SKIP labels are more common in successful trajectories and have higher agreement with expert actions. However, these results establish a correlation between intervention labels and action quality rather than directly demonstrating that the selected turns are the ones where teacher supervision improves learning.

The expert-action agreement results also raise some questions. SKIP has higher agreement than STRONG, which is encouraging, but the absolute agreement rates are low, and WEAK has even lower agreement than STRONG (1.14% versus 4.26%). I understand that multiple actions can be valid and exact agreement with one expert action is not necessarily the right metric, but it is still unclear how accurately the teacher distinguishes between actions needing no correction, partial correction, and strong correction.

Another concern is that the same teacher is responsible for both deciding whether intervention is needed and providing the distillation targets. If the teacher misjudges an action, the resulting supervision may reinforce that mistake. The paper acknowledges this possibility but does not directly evaluate it.

2. The confidence-weighting mechanism and its connection to the claimed benefits need stronger justification.

I am not fully convinced that average top-1 token probability is the right measure of confidence for deciding how much supervision to apply. This metric averages probabilities across the entire response, including reasoning and formatting tokens. It is unclear whether it measures confidence in the actual action or simply how predictable the response tokens are. A teacher could have high confidence in formatting or common reasoning patterns while still being wrong about the action.

The authors provide confidence-related ablations and show that the metric is associated with task completion, which is useful. However, this does not establish that it reflects teacher correctness or that it is better than action-specific confidence or other uncertainty measures.

More broadly, the paper motivates SAGE-OPD through problems such as over-correction of valid actions, repeated actions, and unreliable supervision after earlier mistakes. While the overall benchmark improvements are promising, the experiments do not consistently demonstrate that these specific problems are reduced. For example, in the injected-error continuation experiment, invalid actions decrease but consecutive repetitions increase.

Finally, SAGE-OPD requires additional teacher calls, increasing training cost. The paper compares against an OPD two-rollout baseline, but it is not an exact compute-matched comparison. It remains unclear how much improvement comes from better supervision allocation versus the additional computation used for intervention judgments.

Questions for authors
  1. How do we know that the teacher is selecting the right turns for intervention? The correlation with successful trajectories and expert actions is useful, but can we more directly measure whether applying supervision at a particular turn actually improves learning? For example, could you compare the downstream benefit of supervising turns labeled SKIP, WEAK, and STRONG under a controlled training setup?

  2. How reliable are the intervention labels, especially WEAK? In the expert-action analysis, WEAK has only 1.14% agreement compared with 4.26% for STRONG. Why does WEAK have lower agreement than STRONG? I understand that exact expert agreement is limited when multiple actions are valid, but is there another way to validate whether these categories represent different levels of intervention need?

  3. Could using the same teacher for intervention judgments and distillation targets introduce correlated errors? If the teacher incorrectly decides an action needs correction, the supervision it provides could also be incorrect. Have you tried using a separate model as the intervention judge, or compared intervention labels from independent teachers?

  4. Why use average top-1 token probability as teacher confidence? Since the score includes reasoning, formatting, and action tokens, how do we know it reflects confidence in the actual action rather than token predictability? Would action-specific confidence, entropy, or another uncertainty metric be more appropriate? A comparison against these alternatives would help justify this design choice.

  5. Can you directly quantify the failure modes that motivate the paper? The introduction discusses over-correcting valid actions, repeated actions, and unreliable teacher supervision after mistakes. How much does SAGE-OPD reduce each of these compared with standard OPD? In particular, the continuation experiment shows fewer invalid actions but does not consistently show fewer repetitions. It would help to understand which of the proposed failure modes are actually responsible for the observed improvements.

  6. How much of the benefit comes from selecting the right turns rather than simply supervising fewer tokens? The shuffled-label and confidence-only ablations are helpful. Would random turn selection or simple heuristics achieve similar results when matched for the amount of supervision? This could help isolate the value of the teacher's intervention judgments.

  7. How does SAGE-OPD compare with standard OPD under the same total compute budget? The additional teacher calls increase GPU-hours by approximately 15–22% in the reported configurations. The two-rollout OPD baseline is useful, but does not appear to be exactly compute-matched. Could you compare the methods under equal GPU-hour budgets to better establish the practical benefit of selective intervention?

Reviewer_8SHGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SAGE-OPD allocates teacher supervision selectively during multi-turn on-policy distillation. A deterministic validity check or a teacher judge assigns each student response SKIP, WEAK, or STRONG; the resulting weight is multiplied by the teacher's mean top-1 token probability and normalized across the optimizer batch. The student still executes its own response: intervention changes loss weights, not the acting policy. The method requires neither task-success labels nor a separately trained reward model, but does use environment/parser checks and teacher judgments.

The evaluation uses Qwen3 students of 0.6B and 1.7B with 8B and 32B teachers on ALFWorld and ScienceWorld, plus 8B-teacher SearchQA experiments. The headline 1.7B/8B setting improves ALFWorld unseen success from 61.94% to 70.15% and ScienceWorld success from 4.70% to 6.71%. Evidence includes component and shuffled-label controls, paired task-level tests, three-seed OPD comparisons with the 32B teacher, expert-action agreement, shared-state continuation, and runtime measurements. The contribution is an empirical method for allocating supervision; the theoretical appendix characterizes reweighting and oracle approximation rather than guaranteeing improved task success.

Critical strengths
  1. The method addresses a meaningful distinction between an acceptable student action and an action that matches the teacher's preferred tokens. Response-conditioned gating retains student-generated interaction and reuses the teacher scoring pass for confidence. Table 2 (p. 8) provides unusually useful controls: full SAGE-OPD exceeds shuffled-label supervision on both ALFWorld splits in all four configurations, and exceeds normalized NoConf on both splits in all four. This supports a contribution from the allocation of supervision, beyond simply weakening the entire distillation loss.

  2. The empirical support extends beyond a favorable headline run. For 1.7B/8B, Appendix D.1 reports 16 task-level wins versus 5 losses against OPD on ALFWorld unseen, with an exact McNemar p-value of 0.027. For 0.6B/8B, ScienceWorld mean score improves by 16.77 points, with paired t-test and Wilcoxon p-values of 0.001. The three-seed 32B-teacher results improve mean unseen success for both students (68.66% to 71.89%, and 73.88% to 76.86%; Table 11, p. 26). The paper also reports the 15.5%-21.5% GPU-hour overhead and explicitly acknowledges negative or inconclusive results. This combination makes the narrower empirical claim credible.

Critical weaknesses
  1. The incremental value of the additional teacher judge is not yet established robustly relative to cheaper weighting. In the headline 1.7B/8B setting, normalized confidence-only already reaches 68.66% unseen success versus 70.15% for the full method, a net difference of only two of 134 tasks. For 1.7B/32B, full SAGE-OPD trails both confidence-only and standard OPD in the original run (73.88% versus 74.63%; Table 2). The three-seed study compares only OPD and SAGE-OPD with the 32B teacher; it does not replicate the headline 8B-teacher ablations. This matters because the judge is the main added runtime cost. The gate diagnostics are informative proxies, not a direct estimate of when teacher targets are useful: expert agreement is low and non-monotonic across WEAK/STRONG, and Appendix D.4 predicts student task completion rather than teacher-target correctness. The authors acknowledge these limitations; they nevertheless limit the strength of the mechanistic and cost-benefit conclusions.

  2. Some comparative and reproducibility claims need tighter specification. Appendix C (p. 16) describes original runs with a 4,096-token per-turn limit, but specifies 512-token limits for the added Guided-OPD runs without a complete method-by-experiment configuration map. Its Entropy-Aware OPD implementation reduces to reweighting a forward-KL base loss, so the comparison should be identified as an adaptation and its fidelity justified. SOD also changes rollout count and adds environment rewards. These disclosed differences do not invalidate the matched OPD comparison, but weaken claims about the relative standing of the broader baseline set. The omitted one-shot demonstrations, unspecified top-K value used in cached distributions, and incomplete description of parser fallback behavior further impede exact reproduction (Appendices C and E).

Questions for authors
  1. Clarification and decisive evidence: For the 1.7B/8B configuration, can you provide paired task outcomes and additional training seeds for full SAGE-OPD versus normalized confidence-only? Does the judge's incremental benefit persist when compared at equal GPU-hours, and what is the corresponding confidence-only runtime? This would directly affect whether the added judge is justified.
  2. Clarification: Please map every main-table and ablation method to its generation limit, training budget, checkpoint rule, KL direction, top-K value, and reward use. Were 512 versus 4,096 tokens actually used in the reported comparisons? Please also supply the omitted demonstrations and explain what action is executed when parsing fails. The question is whether the reported comparisons share the intended controls, not whether every baseline must have an identical objective.
  3. Clarification and mechanism check: The judge sees compact state context, at most 20 admissible actions, and a truncated response, whereas token targets use the full history (Appendix E, p. 27). How often is the candidate action absent from the displayed action list or truncated, and are failure labels associated with these cases? An independent audit of a modest sample of SKIP/WEAK/STRONG decisions, allowing multiple valid actions, would help distinguish useful intervention from judge-context artifacts.
  4. Clarification: Appendix D.7 already acknowledges that the shared-state continuation results do not establish reduced sensitivity to injected errors. My remaining question concerns uncertainty: can you provide confidence intervals for the between-method success differences under the clean and error conditions, accounting for the four rollouts nested within each of the 60 instances? This would help assess how precisely Table 9 supports the reported continuation advantage.
1 comment

Thank you for the author's reply. I will keep my original score.

Reviewer_VV6iOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SAGE-OPD selectively applies token-level on-policy distillation in multi-turn agents. Deterministic validity checks and a teacher’s judgment assign each student response a skip, weak, or strong intervention label. Selected turns are further weighted by the teacher’s average top-token probability, and batch normalization restores unit average token weight. The paper evaluates Qwen3 students and teachers on ALFWorld, ScienceWorld, and SearchQA. It reports ablations, post-hoc comparisons with ALFWorld expert actions, shared-state continuations, training cost, and three-seed replications for the 32B-teacher setting.

Critical strengths
  1. A simple, implementable intervention rule. The method combines response assessment with existing teacher distributions and adds no inference-time machinery. The shuffled-label, confidence-only, and normalization ablations indicate that the allocation of supervision matters.

  2. A concrete gain in one important configuration. With a 1.7B student and 8B teacher, ALFWorld unseen success rises from 61.94% for standard OPD to 70.15%. The paired task comparison reports 16 wins and 5 losses for SAGE-OPD, which is more persuasive than the aggregate rate alone.

Critical weaknesses
  1. Evidence for a consistent advantage is limited. The prominent 8B-teacher results and component ablations are single training runs; the three-seed replications cover the 32B teacher instead. In the main 1.7B/32B run, SAGE-OPD trails standard OPD and several specialized baselines on ALFWorld unseen success, although its three-seed mean exceeds OPD. ScienceWorld success remains low in absolute terms, and SearchQA gains are small. The reported paired tests are uncorrected for multiple comparisons. These qualifications make the overall advantage less secure than the strongest result suggests.

  2. The evidence does not establish that the gate reliably identifies harmful actions. The gate and token targets come from the same teacher. In the independent ALFWorld expert comparison, exact action agreement is 12.7% for SKIP, 4.3% for STRONG, and 1.1% for WEAK; the categories do not form a monotonic ordering. Exact agreement is an imperfect measure, but the current analysis provides limited direct validation of intervention correctness. The shared-state “error” test injects an invalid action that leaves the environment unchanged, so it does not test recovery from the state-changing errors used to motivate the method.

Questions for authors
  1. Do the 1.7B-student/8B-teacher ALFWorld gain and the principal ablation differences persist across independent training seeds? A comparison with the strongest baselines at matched total GPU-hours, including intervention queries, would materially affect my assessment.

  2. Can the intervention labels be evaluated against action-level progress or counterfactual outcomes, including errors that change the environment state? This would clarify whether the gate is identifying harmful choices rather than mainly teacher preferences or formatting differences.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Follow-up Reply to Reviewer KTiQ

We sincerely thank the reviewer for the positive reassessment and thoughtful follow-up questions. We appreciate the recognition of our additional experimental evidence and are pleased to clarify the remaining concerns.

Q1. Incremental benefit of the intervention judge

As detailed in our General Response, across three independent training seeds, SAGE-OPD achieves 69.65% versus 68.91% unseen success for normalized confidence-only, with per-seed differences of +1.49%, −2.98%, and +3.73%, respectively. Thus, the full method improves performance in two of three seeds, with a positive average gain of +0.75%. Importantly, their measured training costs are nearly identical: 33.6 versus 33.2 GPU-hours (+1.2%).

To isolate intervention independently of confidence weighting, Intervention-only improves over vanilla OPD from 62.19% to 65.67% (+3.48%), consistently across all three seeds. Furthermore, full SAGE-OPD outperforms Shuffled by +5.97% across all three seeds, demonstrating the benefit of response-specific intervention assignments.

These complementary results support the independent utility of intervention selection. The modest incremental improvement over confidence-only is consistent with partially overlapping functions: intervention judgments identify responses requiring correction, while teacher confidence modulates supervision according to distributional certainty. Because both adapt supervision to the student response, their benefits need not be fully additive.

Q2. Reliability of intervention categories

Our independent PDDL-based validation evaluates 2,378 ALFWorld turns, allowing multiple acceptable actions rather than requiring exact agreement with a single expert recommendation. Environmentally harmful actions receive intervention 19.8% more frequently than acceptable actions (95% CI [4.3%, 32.8%]). Among 692 independently acceptable actions, SAGE-OPD suppresses supervision on 73.5%, providing direct evidence of meaningful supervision allocation.

We further examined the distinction between WEAK and STRONG. WEAK occurs significantly earlier in episodes (normalized position 0.578 vs. 0.707) and is more frequently associated with exploratory actions (14.5% vs. 2.6%). These differences suggest that the two categories capture different decision contexts rather than simply different degrees of action incorrectness. This also helps explain why WEAK-only supervision can achieve higher success than STRONG-only (65.42% vs. 57.96%): earlier, exploratory turns may provide more broadly useful learning signals, whereas STRONG often targets later, more severe deviations. Importantly, the intervention weights specify supervision intensity within the combined objective, not a ranking of standalone training effectiveness.

Q3. Cross-judge consistency.

Qwen3-8B and Qwen3-32B differ in model capacity and intervention thresholds, so identical label distributions are not necessarily expected. Qwen3-32B assigns intervention more frequently than Qwen3-8B (49.9% vs. 22.9%), contributing to the difference in overall agreement (64.9%). The confusion matrix over 2,279 turns is:

Qwen3-8B \ Qwen3-32BSKIPWEAKSTRONG
SKIP1,128162468
WEAK916154
STRONG51336

Importantly, disagreement is predominantly one-directional: Qwen3-32B more frequently assigns stronger intervention to turns labeled SKIP or WEAK by Qwen3-8B, while 336 of 342 (98.2%) STRONG decisions from Qwen3-8B are independently classified as STRONG by Qwen3-32B. This indicates that the judges primarily differ in their propensity to intervene, rather than frequently reversing the original judge's strong-intervention decisions, providing complementary cross-model support for the intervention mechanism.

Overall, our results demonstrate that intervention selection provides meaningful benefits beyond vanilla OPD, while confidence weighting captures partially overlapping improvements. The full method achieves the strongest average performance with only 1.2% additional training cost over confidence-only, supporting the value of combining both components for effective multi-turn distillation.

We sincerely appreciate the reviewer's constructive feedback and would be grateful if these clarifications could motivate reconsideration of the overall rating.

AuthorsOpenReview ↗

Updated Manuscript and Summary of Revisions

We thank all reviewers for their time and constructive feedback. We have updated the PDF manuscript, with revisions highlighted in blue, to add multi-seed robustness results and component ablations, planner-based and counterfactual validation of intervention decisions, and analyses of judge consistency and alternative confidence measures. We have also added comparisons at comparable training costs and clarified implementation details and statistical reporting.

With only one day remaining in the discussion period, we would appreciate any further feedback. We hope these revisions address your concerns and would be grateful if you would reconsider your assessment in light of the additional evidence and clarifications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.