- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SAGE-OPD, a selective intervention framework for multi-turn on-policy distillation. The main idea is that applying teacher supervision uniformly across all turns may not be optimal, especially when student actions are already reasonable or when the teacher's supervision may be unreliable.
The method uses deterministic validity checks and teacher judgments to categorize student responses into SKIP, WEAK, and STRONG intervention levels. These intervention weights are combined with teacher confidence, measured using average top-1 token probability, and normalized to determine the contribution of each turn to the distillation loss.
The authors evaluate SAGE-OPD on ALFWorld, ScienceWorld, and SearchQA using different student and teacher model sizes. The method shows improvements over standard OPD, particularly on ALFWorld and ScienceWorld. The paper also includes comparisons against other distillation methods, component ablations, shuffled-label experiments, statistical analysis, expert-action comparisons, and shared-state continuation evaluations to support the proposed approach.
Critical strengths
1. Well-motivated problem and a useful approach to multi-turn distillation.
The paper addresses an important limitation of standard OPD for multi-turn agents. Applying teacher supervision at every token may unnecessarily penalize valid alternative actions, and mistakes early in a trajectory can affect subsequent interactions. The idea of deciding when the student actually needs supervision, rather than treating all turns equally, makes sense. I also like that the approach preserves student-generated trajectories instead of replacing student actions with teacher actions.
2. Good experimental coverage and meaningful ablations.
The paper has a fairly comprehensive experimental evaluation. The authors compare against several OPD methods across multiple benchmarks and model configurations. The improvements on ALFWorld are meaningful, particularly the increase in unseen success from 61.94% to 70.15% with the 1.7B student and 8B teacher.
I also appreciate the effort to validate individual components. The intervention-only, confidence-only, NoConf, and shuffled-label ablations provide useful evidence that the proposed weighting scheme contributes to performance. The additional expert-action comparisons, statistical tests, and three-seed experiments with the 32B teacher make the evaluation stronger than a straightforward comparison of final success rates.
Overall, the experiments suggest that selectively allocating supervision is a promising direction for improving multi-turn agent training.
Critical weaknesses
1. It is still unclear whether the teacher is identifying the right turns for intervention.
This is my main concern with the paper. The authors show that SKIP labels are more common in successful trajectories and have higher agreement with expert actions. However, these results establish a correlation between intervention labels and action quality rather than directly demonstrating that the selected turns are the ones where teacher supervision improves learning.
The expert-action agreement results also raise some questions. SKIP has higher agreement than STRONG, which is encouraging, but the absolute agreement rates are low, and WEAK has even lower agreement than STRONG (1.14% versus 4.26%). I understand that multiple actions can be valid and exact agreement with one expert action is not necessarily the right metric, but it is still unclear how accurately the teacher distinguishes between actions needing no correction, partial correction, and strong correction.
Another concern is that the same teacher is responsible for both deciding whether intervention is needed and providing the distillation targets. If the teacher misjudges an action, the resulting supervision may reinforce that mistake. The paper acknowledges this possibility but does not directly evaluate it.
2. The confidence-weighting mechanism and its connection to the claimed benefits need stronger justification.
I am not fully convinced that average top-1 token probability is the right measure of confidence for deciding how much supervision to apply. This metric averages probabilities across the entire response, including reasoning and formatting tokens. It is unclear whether it measures confidence in the actual action or simply how predictable the response tokens are. A teacher could have high confidence in formatting or common reasoning patterns while still being wrong about the action.
The authors provide confidence-related ablations and show that the metric is associated with task completion, which is useful. However, this does not establish that it reflects teacher correctness or that it is better than action-specific confidence or other uncertainty measures.
More broadly, the paper motivates SAGE-OPD through problems such as over-correction of valid actions, repeated actions, and unreliable supervision after earlier mistakes. While the overall benchmark improvements are promising, the experiments do not consistently demonstrate that these specific problems are reduced. For example, in the injected-error continuation experiment, invalid actions decrease but consecutive repetitions increase.
Finally, SAGE-OPD requires additional teacher calls, increasing training cost. The paper compares against an OPD two-rollout baseline, but it is not an exact compute-matched comparison. It remains unclear how much improvement comes from better supervision allocation versus the additional computation used for intervention judgments.
Questions for authors
-
How do we know that the teacher is selecting the right turns for intervention? The correlation with successful trajectories and expert actions is useful, but can we more directly measure whether applying supervision at a particular turn actually improves learning? For example, could you compare the downstream benefit of supervising turns labeled SKIP, WEAK, and STRONG under a controlled training setup?
-
How reliable are the intervention labels, especially WEAK? In the expert-action analysis, WEAK has only 1.14% agreement compared with 4.26% for STRONG. Why does WEAK have lower agreement than STRONG? I understand that exact expert agreement is limited when multiple actions are valid, but is there another way to validate whether these categories represent different levels of intervention need?
-
Could using the same teacher for intervention judgments and distillation targets introduce correlated errors? If the teacher incorrectly decides an action needs correction, the supervision it provides could also be incorrect. Have you tried using a separate model as the intervention judge, or compared intervention labels from independent teachers?
-
Why use average top-1 token probability as teacher confidence? Since the score includes reasoning, formatting, and action tokens, how do we know it reflects confidence in the actual action rather than token predictability? Would action-specific confidence, entropy, or another uncertainty metric be more appropriate? A comparison against these alternatives would help justify this design choice.
-
Can you directly quantify the failure modes that motivate the paper? The introduction discusses over-correcting valid actions, repeated actions, and unreliable teacher supervision after mistakes. How much does SAGE-OPD reduce each of these compared with standard OPD? In particular, the continuation experiment shows fewer invalid actions but does not consistently show fewer repetitions. It would help to understand which of the proposed failure modes are actually responsible for the observed improvements.
-
How much of the benefit comes from selecting the right turns rather than simply supervising fewer tokens? The shuffled-label and confidence-only ablations are helpful. Would random turn selection or simple heuristics achieve similar results when matched for the amount of supervision? This could help isolate the value of the teacher's intervention judgments.
-
How does SAGE-OPD compare with standard OPD under the same total compute budget? The additional teacher calls increase GPU-hours by approximately 15–22% in the reported configurations. The two-rollout OPD baseline is useful, but does not appear to be exactly compute-matched. Could you compare the methods under equal GPU-hour budgets to better establish the practical benefit of selective intervention?
Thank you for the author's reply. I will keep my original score.