- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies turn-level supervision allocation in multi-turn agentic on-policy distillation. It derives a turn-level divergence-drift analysis and uses the resulting relative-divergence principle to assign larger distillation weights to more divergent turns. Since the exact theoretical quantity is intractable, DT-OPD uses a group-relative proxy based on token-level teacher-student KL. Experiments on ALFWorld, Search-QA, and WebShop show consistent improvements over OPD and other turn-aware baselines without extra rollouts or trainable modules.
Critical strengths
- Meaningful problem and practical method. The paper addresses a relevant limitation of uniform turn-level supervision in multi-turn OPD. The proposed method is simple, easy to integrate, and requires no additional rollouts, teacher evaluations, or trainable models.
- Clear mathematical motivation. Instead of introducing heuristic weights directly, the paper derives a turn-level teacher-student divergence drift and uses it to motivate relative-divergence-based supervision allocation. This gives the method a clearer theoretical basis than purely heuristic adaptive weighting.
- Useful controlled ablations. The shuffle, reverse-weighting, and trajectory-normalization ablations are informative. In particular, the degradation caused by shuffled and reversed weights suggests that the gains come from the alignment between divergence and turn weights rather than from non-uniform weighting alone.
Critical weaknesses
- Gap between the theoretical quantity and the practical proxy. The theory motivates a state-specific relative divergence, while the implemented method uses a group-relative normalized score across turns that may correspond to different states. The paper explicitly acknowledges that the practical score is not an estimator of the theoretical quantity, but does not empirically validate how well the proxy preserves the intended ordering.
- Limited novelty and unclear necessity of explicit divergence reweighting. The general principle of assigning more optimization effort to poorly matched or difficult examples is intuitive and related to existing hard-example and difficulty-aware distillation methods. Moreover, vanilla KL distillation already produces small gradients on well-aligned examples. The paper could better clarify what explicit turn reweighting contributes beyond this implicit behavior.
- High divergence does not necessarily imply more reliable teacher supervision. In agentic settings, large teacher-student divergence may arise because the student has entered an off-support state where teacher guidance is less reliable. DT-OPD always assigns larger relative weights to high-divergence turns and does not distinguish useful disagreement from potentially unreliable disagreement.
- Limited mechanism analysis and visualization. The paper is text-heavy and lacks sufficient figures to explain or validate the proposed mechanism. There is no method overview, no visualization of which turns receive high weights, and no direct analysis of the relationship among divergence scores, turn weights, and actual training behavior. The current Figure 1 mainly serves as a sanity check and provides limited insight into why DT-OPD works.
Questions for authors
Q1. Can the authors empirically validate the relationship between the theoretical relative divergence and the practical group-relative score? For example, on a subset of states, could multiple responses be sampled from the same state to estimate the theoretical quantity and report its rank correlation with the practical proxy?
Q2. Vanilla OPD already gives small gradients to well-aligned turns and stronger corrective gradients to poorly aligned turns. What additional behavior does explicit DT-OPD reweighting provide beyond this implicit adaptation? A comparison among turn-level KL, vanilla gradient contribution, and DT-OPD weight would be helpful.
Q3. What kinds of turns actually receive large DT-OPD weights? Could the authors provide analysis by turn position, trajectory stage, or representative examples, and discuss how the method behaves when high divergence is caused by off-support states rather than student underfitting?
6 comments
W2:Novelty and necessity of explicit divergence reweighting
W2-1:Difference between DT-OPD and hard-example / difficulty-aware distillation methods
Call back: How does DT-OPD differ from existing hard-example and difficulty-aware distillation methods?
Conclusion: DT-OPD differs from hard-example methods in two essential respects. First, the weighting criterion differs: hard-example methods weight by student-side learning difficulty, whereas DT-OPD weights by the teacher–student divergence. Second, the weighting granularity differs: hard-example methods operate at the sample or token level, whereas DT-OPD weights at the turn level, the decision unit of agentic tasks.
Table W2-1
| Dimension | Hard-example / difficulty-aware weighting | DT-OPD |
|---|---|---|
| Weighting granularity | Sample or token | Turn, the decision unit of agentic tasks |
| Weighting criterion | Student-side learning difficulty | Teacher–student divergence |
| Task applicability | General | Only for multi-turn interactive tasks such as agent interactions |
| Extra overhead | Requires additional forward passes or evaluation | Zero extra overhead; reuses the existing OPD distribution |
Explanation: First, the weighting granularity differs. The motivation of our method stems from the inherent granularity mismatch of agentic tasks, rather than the intuitive assumption that difficult examples should be learned more.
- Sample-level supervision is too coarse. A trajectory contains T decisions, and a trajectory-level weight cannot locate the specific decision step that has not been mastered; in multi-turn tasks, early errors continue to be amplified in later rounds.
- Token-level supervision is too fine, which is the approach adopted by OPD. OPD assigns a coefficient to each token individually and sets no weight at the turn level, splitting the response as a complete action and flattening turn-level decision information.
- The turn is the basic decision unit. A single turn corresponds to one action and serves as the unit on which the environment feedback and subsequent decisions depend.
Second, the weighting criterion differs. Difficulty and teacher–student divergence are different measures, and hard-example weighting cannot align with the teacher signal. The distillation budget should be allocated to where the teacher knows but the student has not yet mastered—i.e., the teacher–student divergence drift characterized by Theorem 4.1.2—rather than to positions the student merely judges as difficult; the two are often inconsistent in agentic tasks.
Summary: The essential difference between DT-OPD and hard-example methods is that DT-OPD builds weighting on the turn, the decision unit of agentic tasks, and allocates the same distillation budget according to teacher–student divergence rather than student-side difficulty.
W2-2:Necessity of explicit divergence reweighting
Call back: Vanilla OPD already exhibits implicit weighting behavior; is explicit turn-level reweighting in DT-OPD still necessary?
Conclusion: It is necessary. Implicit weighting is only per-token "scaling," deciding how much each token updates individually; explicit turn-level reweighting adds the "allocation" that implicit weighting lacks, deciding how the same distillation budget should be divided across turns—i.e., which decisions should be learned more and which less.
Explanation: First, implicit weighting cannot achieve cross-turn allocation; this is a structural limitation rather than insufficient strength.
- Implicit weighting acts only at the token level and is a pointwise judgment. The vanilla KL gradient scales with the teacher–student difference of a single token: better-aligned tokens receive smaller updates, while worse-aligned ones receive larger updates. However, each token's update is determined solely by its own difference and cannot perceive information about other turns in the same batch; it therefore cannot determine whether this turn should receive more or less supervision than other turns.
- Agentic tasks require turn-level allocation. A single turn corresponds to one action, and the environment gives feedback only after the complete response; decisions are made at the turn level, so supervision allocation should match the decision unit.
Second, DT-OPD's explicit turn-level reweighting supplies allocation capability at this level, rather than merely amplifying the scaling magnitude. It explicitly allocates a fixed distillation budget across turns: the number of weights is on the same order as the number of decisions, not the number of tokens, aligning the supervision unit, the decision unit, and the environment-response unit.
Summary: In summary, explicit turn-level reweighting in DT-OPD is necessary: implicit weighting is only per-token "scaling," deciding how much each token updates on its own; explicit turn-level reweighting adds the cross-turn "allocation".
W3:High divergence does not necessarily imply more reliable teacher supervision
W3:High divergence does not necessarily imply more reliable teacher supervision
Call back: High divergence does not necessarily imply more reliable teacher supervision.
Conclusion: We thank you for raising this valuable question. Through analysis, we find that this is a problem shared across the entire OPD family of methods, rather than a new problem introduced by DT-OPD; DT-OPD itself neither resolves nor aggravates this issue—this paper addresses how the distillation budget is allocated, not the reliability judgment of the supervision signal. We will treat reliability judgment as an independent future research direction.
Explanation:
First, the supervision unreliability induced by off-support is a shared premise limitation of the OPD family, independent of the weighting mechanism. Both vanilla OPD and DT-OPD use the teacher distribution at states visited by the student as the learning signal. When the student leaves the support set, the teacher supervision obtained by both methods is equally unreliable. This limitation is an inherent boundary of this family of methods and is not introduced by reweighting.
Second, this paper focuses on the allocation strategy and does not address signal reliability judgment; the two are independent of each other.
-
What DT-OPD addresses is: under the premise that teacher supervision is reliable, to which turns the limited distillation budget should be allocated. Theorem 4.1.2 provides the solution based on within-group relative divergence.
-
The reviewer raises an orthogonal question: on which states is the teacher signal trustworthy. This is an independent reliability-judgment direction suited to separate study; incorporating it into the weighting mechanism would conflate the two objectives.
Summary: Off-support reliability is a shared premise issue of the OPD family; DT-OPD inherits rather than introduces it, and does not aggravate it structurally. This paper focuses on "how to allocate"; reliability judgment is a future direction of independent value, and inspired by your comment we decide to pursue it as a research direction.
Weakness 1:Gap between the theoretical quantity and the practical proxy. Thank you for your review. Below is our response. Please feel free to let us know if anything remains unclear.
We thank you for this insightful question.
W1-1:Does cross-state within-group normalization break state specificity?
Call back: The theory relies on the state-specific relative teacher–student divergence ρ(a|s); does the cross-state within-group normalization used in implementation break state specificity?
Conclusion: It does not break it. Theorem 4.1.2 answers which response should be reinforced for a single state, whereas the group normalization of DT-OPD addresses the next question that naturally extends this theorem: how to allocate the same distillation budget across the turns of a prompt. The latter is an allocation problem, which requires only the relative order among turns rather than the absolute values of individual states; the weights in Eq. (16) happen to depend solely on the within-group order, and Lemma B.1.2 guarantees that they are invariant to a common within-group shift and order-preserving. Cross-state normalization is therefore a viable proxy for the theoretical quantity at the criterion level.
W1-2:Empirical validation that the proxy score preserves the intended ranking
Call back: An empirical validation is needed for the extent to which the practical proxy preserves the theoretical ranking.
Conclusion: It preserves it, with strong agreement. We ran experiments on Search-QA to examine how well the practical proxy retains the ranking of the theoretical values; the experiments cover 64 prompts, 379 states, and 1,388 turns. We quantify the agreement between the proxy ranking and the theoretical ranking with the Spearman correlation coefficient rs, and measure the proportion of supervision in the top 25% by supervision weight, Φ_{25\\%}: among the top 25% turns deemed important by theory, how much supervision signal they receive under the practical proxy; the results are reported in Table W1-2.
Table W1-2 shows that the Spearman correlation between the theoretical ranking and the ranking produced by the practical proxy is rs=+0.693, which falls in the 0.6–0.8 range of strong correlation. This indicates that the proxy and theoretical rankings are strongly correlated: the turns that theory prioritizes are also ranked high by the proxy, and the supervision budget is likewise assigned to the turns designated by theory.
Summary: The proxy score is strongly positively correlated with the theoretical relative teacher–student divergence; the ranking property is preserved, and the supervision budget can be directed to the turns designated by theory, so that there is no disconnect between theory and implementation.
Table W1-2