acceptodds
Under review as a conference paper at ICLR 2027

Divergence-Calibrated Turn Importance for Multi-Turn Agentic On-Policy Distillation

Abstract

On-policy distillation (OPD) has become a prevailing recipe for transferring multi-turn agentic capabilities from large teachers to small deployable students. However, effective multi-turn distillation remains challenging: OPD supervises at the token level, whereas an agentic decision is made by a complete turn, so token-level supervision can flatten turn-level decisions. Existing methods have recognized the multi-turn agentic property, but they either introduce environment rewards that pure OPD lacks or aim to skip unnecessary supervision, rebalance the loss budget, and keep supervision reliable. This leaves an open question: how should supervision be allocated across turns by their importance when no environment reward is available? We propose Divergence-Calibrated Turn Importance for Multi-Turn Agentic On-Policy Distillation (DT-OPD) to answer this question. We demonstrate that, under natural-gradient updates, the teacher-student divergence drift induced by a sampled response is determined by the response's stop-gradient negated divergence score and by its relative divergence. Our analysis lifts the divergence from the token level to the turn level, aligning it with the unit at which the agent acts. DT-OPD therefore turns the drift into a practical turn-level teacher-student divergence proxy and redistributes supervision across turns accordingly. Extensive experiments on ALFWorld, Search-QA, and WebShop, distilling Qwen3 and Qwen2.5 students from 0.6B to 4B using GRPO teachers up to Qwen3-30B-A3B, show that DT-OPD attains the best average in each benchmark at every student scale, and ablation studies and training dynamics further confirm that both of our core contributions are essential to this performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi‑turn LLM Agents, On‑policy Distillation, Turn‑level Importance, Divergence Calibration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Lxk8OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies turn-level supervision allocation in multi-turn agentic on-policy distillation. It derives a turn-level divergence-drift analysis and uses the resulting relative-divergence principle to assign larger distillation weights to more divergent turns. Since the exact theoretical quantity is intractable, DT-OPD uses a group-relative proxy based on token-level teacher-student KL. Experiments on ALFWorld, Search-QA, and WebShop show consistent improvements over OPD and other turn-aware baselines without extra rollouts or trainable modules.

Critical strengths
  1. Meaningful problem and practical method. The paper addresses a relevant limitation of uniform turn-level supervision in multi-turn OPD. The proposed method is simple, easy to integrate, and requires no additional rollouts, teacher evaluations, or trainable models.
  2. Clear mathematical motivation. Instead of introducing heuristic weights directly, the paper derives a turn-level teacher-student divergence drift and uses it to motivate relative-divergence-based supervision allocation. This gives the method a clearer theoretical basis than purely heuristic adaptive weighting.
  3. Useful controlled ablations. The shuffle, reverse-weighting, and trajectory-normalization ablations are informative. In particular, the degradation caused by shuffled and reversed weights suggests that the gains come from the alignment between divergence and turn weights rather than from non-uniform weighting alone.
Critical weaknesses
  1. Gap between the theoretical quantity and the practical proxy. The theory motivates a state-specific relative divergence, while the implemented method uses a group-relative normalized score across turns that may correspond to different states. The paper explicitly acknowledges that the practical score is not an estimator of the theoretical quantity, but does not empirically validate how well the proxy preserves the intended ordering.
  2. Limited novelty and unclear necessity of explicit divergence reweighting. The general principle of assigning more optimization effort to poorly matched or difficult examples is intuitive and related to existing hard-example and difficulty-aware distillation methods. Moreover, vanilla KL distillation already produces small gradients on well-aligned examples. The paper could better clarify what explicit turn reweighting contributes beyond this implicit behavior.
  3. High divergence does not necessarily imply more reliable teacher supervision. In agentic settings, large teacher-student divergence may arise because the student has entered an off-support state where teacher guidance is less reliable. DT-OPD always assigns larger relative weights to high-divergence turns and does not distinguish useful disagreement from potentially unreliable disagreement.
  4. Limited mechanism analysis and visualization. The paper is text-heavy and lacks sufficient figures to explain or validate the proposed mechanism. There is no method overview, no visualization of which turns receive high weights, and no direct analysis of the relationship among divergence scores, turn weights, and actual training behavior. The current Figure 1 mainly serves as a sanity check and provides limited insight into why DT-OPD works.
Questions for authors

Q1. Can the authors empirically validate the relationship between the theoretical relative divergence and the practical group-relative score? For example, on a subset of states, could multiple responses be sampled from the same state to estimate the theoretical quantity and report its rank correlation with the practical proxy?

Q2. Vanilla OPD already gives small gradients to well-aligned turns and stronger corrective gradients to poorly aligned turns. What additional behavior does explicit DT-OPD reweighting provide beyond this implicit adaptation? A comparison among turn-level KL, vanilla gradient contribution, and DT-OPD weight would be helpful.

Q3. What kinds of turns actually receive large DT-OPD weights? Could the authors provide analysis by turn position, trajectory stage, or representative examples, and discuss how the method behaves when high divergence is caused by off-support states rather than student underfitting?

6 comments
AuthorsOpenReview ↗

Weakness 1:Gap between the theoretical quantity and the practical proxy. Thank you for your review. Below is our response. Please feel free to let us know if anything remains unclear.

We thank you for this insightful question.

W1-1:Does cross-state within-group normalization break state specificity?

Call back: The theory relies on the state-specific relative teacher–student divergence ρ(a|s); does the cross-state within-group normalization used in implementation break state specificity?

Conclusion: It does not break it. Theorem 4.1.2 answers which response should be reinforced for a single state, whereas the group normalization of DT-OPD addresses the next question that naturally extends this theorem: how to allocate the same distillation budget across the turns of a prompt. The latter is an allocation problem, which requires only the relative order among turns rather than the absolute values of individual states; the weights in Eq. (16) happen to depend solely on the within-group order, and Lemma B.1.2 guarantees that they are invariant to a common within-group shift and order-preserving. Cross-state normalization is therefore a viable proxy for the theoretical quantity at the criterion level.


W1-2:Empirical validation that the proxy score preserves the intended ranking

Call back: An empirical validation is needed for the extent to which the practical proxy preserves the theoretical ranking.

Conclusion: It preserves it, with strong agreement. We ran experiments on Search-QA to examine how well the practical proxy retains the ranking of the theoretical values; the experiments cover 64 prompts, 379 states, and 1,388 turns. We quantify the agreement between the proxy ranking and the theoretical ranking with the Spearman correlation coefficient rsr_s, and measure the proportion of supervision in the top 25% by supervision weight, Φ_{25\\%}: among the top 25% turns deemed important by theory, how much supervision signal they receive under the practical proxy; the results are reported in Table W1-2.

Table W1-2 shows that the Spearman correlation between the theoretical ranking and the ranking produced by the practical proxy is rs=+0.693r_s = +0.693, which falls in the 0.6–0.8 range of strong correlation. This indicates that the proxy and theoretical rankings are strongly correlated: the turns that theory prioritizes are also ranked high by the proxy, and the supervision budget is likewise assigned to the turns designated by theory.

Summary: The proxy score is strongly positively correlated with the theoretical relative teacher–student divergence; the ranking property is preserved, and the supervision budget can be directed to the turns designated by theory, so that there is no disconnect between theory and implementation.

Table W1-2

MetricValueRandom baseline
Spearman rsr_s+0.6930
Supervision Φ_{25\\%}0.4300.250
AuthorsOpenReview ↗

W2:Novelty and necessity of explicit divergence reweighting

W2-1:Difference between DT-OPD and hard-example / difficulty-aware distillation methods

Call back: How does DT-OPD differ from existing hard-example and difficulty-aware distillation methods?

Conclusion: DT-OPD differs from hard-example methods in two essential respects. First, the weighting criterion differs: hard-example methods weight by student-side learning difficulty, whereas DT-OPD weights by the teacher–student divergence. Second, the weighting granularity differs: hard-example methods operate at the sample or token level, whereas DT-OPD weights at the turn level, the decision unit of agentic tasks.

Table W2-1

DimensionHard-example / difficulty-aware weightingDT-OPD
Weighting granularitySample or tokenTurn, the decision unit of agentic tasks
Weighting criterionStudent-side learning difficultyTeacher–student divergence
Task applicabilityGeneralOnly for multi-turn interactive tasks such as agent interactions
Extra overheadRequires additional forward passes or evaluationZero extra overhead; reuses the existing OPD distribution

Explanation: First, the weighting granularity differs. The motivation of our method stems from the inherent granularity mismatch of agentic tasks, rather than the intuitive assumption that difficult examples should be learned more.

  1. Sample-level supervision is too coarse. A trajectory contains T decisions, and a trajectory-level weight cannot locate the specific decision step that has not been mastered; in multi-turn tasks, early errors continue to be amplified in later rounds.
  2. Token-level supervision is too fine, which is the approach adopted by OPD. OPD assigns a coefficient to each token individually and sets no weight at the turn level, splitting the response as a complete action and flattening turn-level decision information.
  3. The turn is the basic decision unit. A single turn corresponds to one action and serves as the unit on which the environment feedback and subsequent decisions depend.

Second, the weighting criterion differs. Difficulty and teacher–student divergence are different measures, and hard-example weighting cannot align with the teacher signal. The distillation budget should be allocated to where the teacher knows but the student has not yet mastered—i.e., the teacher–student divergence drift characterized by Theorem 4.1.2—rather than to positions the student merely judges as difficult; the two are often inconsistent in agentic tasks.

Summary: The essential difference between DT-OPD and hard-example methods is that DT-OPD builds weighting on the turn, the decision unit of agentic tasks, and allocates the same distillation budget according to teacher–student divergence rather than student-side difficulty.


W2-2:Necessity of explicit divergence reweighting

Call back: Vanilla OPD already exhibits implicit weighting behavior; is explicit turn-level reweighting in DT-OPD still necessary?

Conclusion: It is necessary. Implicit weighting is only per-token "scaling," deciding how much each token updates individually; explicit turn-level reweighting adds the "allocation" that implicit weighting lacks, deciding how the same distillation budget should be divided across turns—i.e., which decisions should be learned more and which less.

Explanation: First, implicit weighting cannot achieve cross-turn allocation; this is a structural limitation rather than insufficient strength.

  1. Implicit weighting acts only at the token level and is a pointwise judgment. The vanilla KL gradient scales with the teacher–student difference of a single token: better-aligned tokens receive smaller updates, while worse-aligned ones receive larger updates. However, each token's update is determined solely by its own difference and cannot perceive information about other turns in the same batch; it therefore cannot determine whether this turn should receive more or less supervision than other turns.
  2. Agentic tasks require turn-level allocation. A single turn corresponds to one action, and the environment gives feedback only after the complete response; decisions are made at the turn level, so supervision allocation should match the decision unit.

Second, DT-OPD's explicit turn-level reweighting supplies allocation capability at this level, rather than merely amplifying the scaling magnitude. It explicitly allocates a fixed distillation budget across turns: the number of weights is on the same order as the number of decisions, not the number of tokens, aligning the supervision unit, the decision unit, and the environment-response unit.

Summary: In summary, explicit turn-level reweighting in DT-OPD is necessary: implicit weighting is only per-token "scaling," deciding how much each token updates on its own; explicit turn-level reweighting adds the cross-turn "allocation".

AuthorsOpenReview ↗

W3:High divergence does not necessarily imply more reliable teacher supervision

W3:High divergence does not necessarily imply more reliable teacher supervision

Call back: High divergence does not necessarily imply more reliable teacher supervision.

Conclusion: We thank you for raising this valuable question. Through analysis, we find that this is a problem shared across the entire OPD family of methods, rather than a new problem introduced by DT-OPD; DT-OPD itself neither resolves nor aggravates this issue—this paper addresses how the distillation budget is allocated, not the reliability judgment of the supervision signal. We will treat reliability judgment as an independent future research direction.

Explanation:

First, the supervision unreliability induced by off-support is a shared premise limitation of the OPD family, independent of the weighting mechanism. Both vanilla OPD and DT-OPD use the teacher distribution at states visited by the student as the learning signal. When the student leaves the support set, the teacher supervision obtained by both methods is equally unreliable. This limitation is an inherent boundary of this family of methods and is not introduced by reweighting.

Second, this paper focuses on the allocation strategy and does not address signal reliability judgment; the two are independent of each other.

  1. What DT-OPD addresses is: under the premise that teacher supervision is reliable, to which turns the limited distillation budget should be allocated. Theorem 4.1.2 provides the solution based on within-group relative divergence.

  2. The reviewer raises an orthogonal question: on which states is the teacher signal trustworthy. This is an independent reliability-judgment direction suited to separate study; incorporating it into the weighting mechanism would conflate the two objectives.

Summary: Off-support reliability is a shared premise issue of the OPD family; DT-OPD inherits rather than introduces it, and does not aggravate it structurally. This paper focuses on "how to allocate"; reliability judgment is a future direction of independent value, and inspired by your comment we decide to pursue it as a research direction.

Reviewer_GJDBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to allocate supervision across turns in multi-turn agentic on-policy distillation without using environment rewards during student distillation. It derives a local teacher–student reverse-KL drift identity under Fisher–Rao natural-gradient updates, with a frozen-occupancy assumption for the policy-level result. Motivated by this analysis, DT-OPD constructs length-normalized turn scores from token-level KL estimates, clips and normalizes them within groups of responses generated for the same prompt, and converts them into positive, stop-gradient weights with approximately unit mean across turns. Each weight is broadcast to the corresponding turn's tokens. The practical group-relative score is a heuristic surrogate, not an exact estimate of the state-specific relative divergence, as acknowledged in Section 4.2 and Appendix B.1. The evaluation covers ALFWorld, Search-QA, and WebShop using several Qwen3 and Qwen2.5 student–teacher configurations, with weighting ablations, training diagnostics, and timing comparisons.

Critical strengths
  1. A coherent and potentially practical turn-weighting mechanism with low reported overhead. Sections 4.2–4.3 and Eqs. (14)–(17) describe a clear sequence: aggregate token-level KL estimates into length-normalized turn scores, normalize them within prompt groups, and apply positive, stop-gradient weights to complete turns. The method reuses distributions evaluated by OPD without requiring additional rollouts or teacher evaluations. In the main-text cost analysis, Table 4 reports 0.54 s versus 0.45 s for weight construction plus advantage computation, consistent with small incremental overhead. Improving supervision allocation without adding an inference stage is a meaningful practical contribution.

  2. Useful initial evidence across distinct tasks and model configurations. Tables 1–2 cover embodied interaction, retrieval-assisted question answering, and shopping, using Qwen3 and Qwen2.5 configurations. Some gains are substantial enough to warrant attention: Qwen2.5-3B WebShop accuracy improves from the strongest baseline’s 71.4 to 75.8, while Qwen3-0.6B ALFWorld average success improves from 75.8 to 78.9. Table 3 also includes relevant shuffled-weight, reversed-weight, and trajectory-normalization controls: the full method scores 78.1 versus 75.0, 75.0, and 70.3, respectively. This breadth and the targeted controls provide useful initial support beyond a single-task comparison with vanilla OPD, subject to the reliability concerns below.

Critical weaknesses
  1. The reliability of the smaller gains and the evaluation protocol remain insufficiently established. Table 1 reports Search-QA improvements of 0.8, 0.6, and 0.6 percentage points over the strongest same-configuration baselines. For Qwen3-0.6B WebShop, Table 2 shows a 0.4-point Score gain over SOD and an accuracy tie with OPD at 82.0. Tables 1–3 give point estimates without across-training-run uncertainty; Figure 1 shows training dynamics rather than establishing whether these differences are reproducible. Looking to the appendix for further details does not resolve this concern: it states that evaluation protocols and checkpoint-selection rules are shared, but does not specify independent-training-seed counts, the concrete selection rule, evaluation-set sizes and sampling procedure, or whether final evaluation is separate from checkpoint selection. These missing details and uncertainty estimates limit confidence in the claimed consistent improvement; they do not establish that the evaluation was unfair.

  2. The benefit of supervision reallocation is not fully isolated, and the weighting procedure is underspecified for reproduction. Section 4.3 explicitly conditions preservation of the token-level budget on weights being uncorrelated with response length. Approximately unit mean across turns does not guarantee that ∑(i,t)∈Gni,twi,tD=∑(i,t)∈Gni,t\sum_{(i,t)\in G} n_{i,t}w^D_{i,t}=\sum_{(i,t)\in G}n_{i,t}, where ni,tn_{i,t} counts loss-bearing tokens. The controls in Table 3 are useful, but shuffling preserves the distribution of turn weights, not necessarily their length-weighted sum. No weight–length correlation or exact token-mass-matched control is reported, including in the supplementary ablation discussion. The results therefore do not fully exclude a change in aggregate coefficient scale as a contributor to the gains. Sections 4.2–4.3 also introduce λD\lambda_D, qloq_{\mathrm{lo}}, qhiq_{\mathrm{hi}}, ε\varepsilon, and a truncated-support KL estimate without specifying the experimental parameter values or the complete estimator. Checking the algorithm and configuration tables in the appendix still leaves the parameter-selection procedure, support-selection rule, probability renormalization, and treatment of omitted mass unspecified. These details directly determine the proposed weights and should be supplied. Together with the missing control, this prevents a convincing, reproducible attribution of the improvement to the allocation rule itself.

Questions for authors

The questions below correspond directly to Weaknesses 1–2, and their answers could materially affect my assessment. Clarification requests concern the reported experiments or implementation; supporting-evidence requests address the existing claims, not extensions of the work. Existing results or implementation details may resolve these concerns without new experiments. No suggestions for future work are included.

1、Weakness1:Clarification request. For the comparisons in Tables 1–3, what are the evaluation splits, their sizes, and the episode-sampling procedure? Are ALFWorld task instances and per-category episode counts fixed across methods, and how are the reported averages computed? Please specify the checkpoint-selection metric and data, and whether final evaluation uses a held-out test set distinct from the validation curves. Please also clarify why the evaluation uses Qwen3/Qwen2.5 rather than newer open-weight Qwen3.5 models at comparable scales (0.8B, 2B, and 4B), including any availability, compatibility, or compute-budget constraints relevant to interpreting the results.

2、Weakness1:Supporting-evidence request. How many independent training runs underlie the main comparisons and Table 3’s ablation? Can you report seed counts and mean ± standard deviation, together with uncertainty in the performance differences on common evaluation instances? If further runs are needed, a focused comparison of DT-OPD, OPD, and the relevant strongest baseline over at least three independent training seeds would help establish robustness. Please distinguish independent training runs from repeated decoding of one checkpoint.

3、Weakness2:Clarification request. What values of λD\lambda_D, qloq_{\mathrm{lo}}, qhiq_{\mathrm{hi}}, and ε\varepsilon were used in Sections 4.2–4.3, how were they selected, and were they shared across settings? Please provide the exact truncated-support KL estimator, including support selection, probability renormalization, and treatment of omitted mass. Also specify the implemented loss reduction and any additional weight normalization, since these determine whether the token-mass concern applies to the actual implementation.

4、Weakness2:Supporting-evidence request. What is the observed relationship between turn weights and response lengths? Does the improvement persist when weights are renormalized so that ∑(i,t)∈Gni,twi,t=∑(i,t)∈Gni,t\sum_{(i,t)\in G}n_{i,t}w_{i,t}=\sum_{(i,t)\in G}n_{i,t}, matching the token-weight mass of unit OPD weights on the same sampled tokens? This control should use the actual loss-bearing-token mask and the appropriate loss-reduction scope. It fixes aggregate token coefficients, not the realized loss value or gradient norm. If the implementation already performs this normalization, please document it instead.

7 comments
AuthorsOpenReview ↗

Weakness1:The reliability of the gains and the evaluation protocol. Thank you for your valuable comments and careful review. Below is our response. Please feel free to let us know if anything remains unclear.

Thank you for your comments and review.

W1-1:The reliability of the gains

Call back: Are the gains on Search-QA and WebShop reliable and reproducible?

Conclusion: The gains of our DT-OPD are reliable and reproducible. We conducted DT-OPD training with three independent random seeds on Search-QA and in the WebShop setting with Qwen3-0.6B as the student model. The corresponding experimental results are detailed in Tables W1-1-T1 and W1-1-T2. The high consistency of performance across multiple independent training runs fully validates the reproducibility and reliability of our method, confirming that the observed performance gains are not an artifact of a single run.

Table W1-1-T1 Search-QA

Student modelMethodssuccess rate (%)
Qwen3-0.6BVanilla14.1
GRPO38.8
OPD40.0
SOD38.7
TCOD37.9
DT-OPD40.68 ± 0.22
Qwen3-1.7BVanilla30.2
GRPO40.8
OPD43.4
SOD43.0
TCOD42.2
DT-OPD44.09 ± 0.12
Qwen3-4BVanilla31.7
GRPO47.0
OPD47.5
SOD45.9
TCOD47.0
DT-OPD48.05 ± 0.20

Table W1-1-T1 shows that DT-OPD outperforms the best baseline under matched configurations across all three student model sizes; the standard deviation over three repeated training runs is 0.12–0.22, which is substantially smaller than the performance margin of 0.55–0.69. The coefficients of variation for the three student models are 0.54%, 0.27%, and 0.41%, indicating high stability of training results across different random seeds.** This demonstrates that the reported 0.6–0.8 score improvement in the original manuscript exceeds random training fluctuations, constituting a real, independently reproducible gain rather than an incidental result from a single experiment.

Table W1-1-T2 WebShop

Student modelMethodsScore (%)Accuracy (%)
Qwen3-0.6BVanilla33.12.3
GRPO44.916.4
SOD90.678.8
TCOD8779.2
OPD88.682
DT-OPD90.73 ± 0.2383.07 ± 1.22

Table W1-1-T2 shows that for DT-OPD trained with 3 distinct random seeds on the WebShop benchmark, the mean accuracy and standard deviation are 83.07 ± 1.22. It outperforms baselines including GRPO, SOD and TCOD, and is slightly higher than the strongest baseline OPD (82.0), with an improvement of 1.07. The mean Score and standard deviation are 90.73 ± 0.23, which is substantially better than Vanilla, GRPO, TCOD and OPD, and roughly on par with the top baseline SOD (90.6). This further demonstrates that the gains of DT-OPD on WebShop are reliable and reproducible.

Summary: The additional random seed experiments demonstrate that the performance gains of our proposed DT-OPD on the reviewer-concerned Search-QA and WebShop benchmarks are reliable and reproducible.


W1-2:Evaluation Protocol

Call back: You are concerned about the details of the evaluation protocol.

Conclusion: As requested, we summarize the details of the evaluation protocol in Tables W1-2-T1 and W1-2-T2, including the independent-training-seed counts, concrete selection rule, evaluation-set sizes, and sampling procedure raised in your comment.

Table W1-2-T1 Evaluation Protocol Settings

Protocol ItemRule
Checkpoint selectionFixed training endpoint: Step 150
Validation metricTask success rate
Final evaluationPerformed only once, after training
Evaluation set sizeALFWorld 128 / Search-QA 51,713 / WebShop 128
Sampling procedureALFWorld and WebShop: 128 instances randomly sampled before training; Search-QA: all instances evaluated
Metric aggregationSuccess rate or accuracy computed separately for the full evaluation set and for each category or subset
Independent training seeds3
Applicable methodsDT-OPD, GRPO, OPD, SOD, and TCOD (all)

Table W1-2-T2 Benchmark Details

BenchmarkFull Sample PoolFinal Evaluation SizeSample Usage
ALFWorld274 instances128 instances128 instances randomly sampled
Search-QA51,713 questions51,713 questionsAll 51,713 questions evaluated
WebShop500 instances128 instances128 instances randomly sampled

Summary: The full details of the evaluation protocol are provided. All tables in the original manuscript use the same protocol, and the evaluation pipeline can be reproduced step by step.

AuthorsOpenReview ↗

Weakness 2:The isolation of the benefit of supervision reallocation and the reproducibility of the weighting procedure

W2-1:Correlation between weights and response length

Call back: Are the weights correlated with response length?

Conclusion: The weights show only a weak or very weak correlation with response length. We computed the correlation between the weights and response length on the Search-QA benchmark, which contains 51,713 samples. Under the three settings, the Pearson correlation coefficients ρ(n,w)\rho(n,w) between the weights and response length are +0.382 (weak), +0.328 (weak), and +0.118 (very weak), respectively. These results indicate that the weights assigned by our method have only a weak or very weak correlation with response length.

Table W2-1 Pearson correlation coefficients between weights and response length

Base ModelbenchmarkR = Σn·w/Σnρ(n,w)Matching constant c
Qwen3-0.6BSearch-QA1.170+0.3820.892
Qwen3-1.7BSearch-QA1.197+0.3280.926
Qwen3-4BSearch-QA1.019+0.1180.989

Table W2-1 reports the Pearson correlation coefficients ρ between the weights and response length under the three settings, namely +0.382, +0.328, and +0.118; the total token supervision R and the matching constant c are also listed in the same table. According to the Pearson correlation interpretation scale, 0.8–1.0 denotes very strong correlation, 0.6–0.8 strong, 0.4–0.6 moderate, 0.2–0.4 weak, and 0.0–0.2 very weak or no correlation. Among the three ρ values, +0.382 and +0.328 fall into the weak correlation range, while +0.118 falls into the very weak correlation range. Thus, the weights and response length have only a weak or very weak correlation.


W2-2: Does the gain persist after matching the total token-level supervision?

Call back: Can DT-OPD still obtain a performance gain when its total token-level supervision is strictly matched to that of OPD?

Conclusion: When the total token-level supervision of DT-OPD and OPD is strictly matched, the performance gain of DT-OPD persists. We ran three experiments on the Search-QA benchmark, which contains 51,713 samples. In each group, all weights were globally rescaled by a single constant c so that the group strictly satisfies Σ n·w = Σ n. The three Search-QA settings were retrained for 150 steps and re-evaluated; the results are reported in Table W2-2.

Table W2-2 Performance under matched total token-level supervision

Base ModelbenchmarkOPDDT-OPD (mean ± SD)DT-OPD-matchedΔ
Qwen3-0.6BSearch-QA40.040.7 ± 0.2340.6−0.1
Qwen3-1.7BSearch-QA43.444.1 ± 0.1744.2+0.1
Qwen3-4BSearch-QA47.548.0 ± 0.2148.00.0

Table W2-2 reports the success rates (%) of the three student sizes on Search-QA: the OPD column lists the values reported in Table 1 of the original manuscript, the DT-OPD column lists the mean ± standard deviation over three seeds, the DT-OPD-matched column lists the results of retraining for 150 steps after matching the total token-level supervision, and Δ is the difference between DT-OPD-matched and the DT-OPD mean. In all three groups, Δ falls within the standard deviation over the three random seeds of the corresponding DT-OPD run. This means that after aligning DT-OPD's total supervision with OPD, DT-OPD retains essentially the same gain as before alignment, indicating that the gain comes from DT-OPD's own reweighting rather than the overall scale of total supervision.

Summary: After matching the total token-level supervision, the performance advantage of DT-OPD over OPD is retained. If the gain originated from the aggregate coefficient scale, flattening this scale should produce an observable performance drop; no such drop appears in the results. This indicates that this factor is not the source of the gain, and the improvement should be attributed to DT-OPD's own weight allocation rule.

AuthorsOpenReview ↗

Weakness 2:The isolation of the benefit of supervision reallocation and the reproducibility of the weighting procedure

W2-3: Hyperparameters, truncated-support KL estimator, and implementation details

W2-3.1 Hyperparameter values, selection procedure, and sharing

Answer: the values are listed in Table W2-3. λDλ_D and εε are fixed across all settings, and only qloq_lo / qhiq_hi is adjusted according to the number of turns within each benchmark group. See Table W2-3.

Table W2-3 Hyperparameters

ParameterValueRole
λDλ_D2.0Temperature in Eq. (16)
qloq_lo / qhiq_hi0.005 / 0.995 (0.02 / 0.98 for ALFWorld)Within-group percentile truncation in Eq. (15)
ε1ε_1 (min–max denominator)1e-8Prevents division by zero
ε2ε_2 (self-calibration denominator)1e-8Prevents division by zero
kk64Top-k support

Selection rationale:

  1. λ_D = 2.0 is determined by the contrast budget. D̃ lies in [0,1], so the exponent lies in [0, 2] and the exponential value falls between 1 and e², which avoids overflow and keeps the weight scale comparable across settings.
  2. q is determined by the order of magnitude of the number of spans within a group. An ALFWorld prompt group contains about 270 spans; q=0.02 removes about 5 points on each side, which exactly removes the degenerate span with response_length=1. This span has extremely large variance and, if not truncated, would become the min–max anchor of the whole group and compress the discriminability of the remaining spans. A Search-QA group contains several times as many spans as ALFWorld, so q=0.005 is used.
  3. k is determined by the concentration of the student distribution. The measured per-token student entropy ranges from 0.2 to 0.5 nats, and the mass is highly concentrated; top-64 already covers almost all probability mass. k is therefore a cost knob rather than a behavioral knob.

Summary: Within the same benchmark and student size, DT-OPD and all baselines share the same set of values; across benchmarks, only the rule above distinguishes them: 0.005 for Search-QA and 0.02 for the remaining benchmarks.


W2-3.2 Exact form of the truncated-support KL estimator

Conclusion: the estimator is defined on the student's top-k support; student probabilities are renormalized within the support, while teacher probabilities are taken after full-vocabulary normalization over the same set of tokens.

Specifically, at position ℓ:

  1. Support selection. Take the top-k token set from the student's next-token distribution (k=64), not the teacher's top-k, because the weights should be determined by what the student sees.
  2. Student-side weights. Take the softmax of the student's log probabilities within the support, i.e., divide by the sum of probability mass inside the support, so that the weights sum to 1 over the support.
  3. Teacher-side values. The teacher's log probabilities retain their original values after full-vocabulary normalization and are not divided again by their mass on the support.
  4. Combination. Multiply the weights from Step 2 by the difference between the student log probability and the teacher log probability, and sum over all tokens in the support; this gives the truncated divergence estimate at that position.
  5. Omitted mass. The student-side renormalization absorbs the omitted mass, so the target quantity of this estimator is the KL on the support, rather than an approximation to the full-vocabulary KL.

W2-3.3 Reduction in implementation and additional weight normalization

Answer: the reduction is exactly the same as in standard OPD; DT-OPD only multiplies the per-token term of OPD by an additional turn-level weight wtw_t, with no additional weight normalization. After Eq. (16) yields wtw_t for each turn, it is broadcast directly to every token under that turn, with no intermediate scaling, truncation, or renormalization.

Reviewer_HoF9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors consider methods for weighting turns in multi-turn OPD based on their "importance." The top-level approach is to first decompose the problem into turn-level divergences. Then the authors use information geometry to analyze to contribution of each turn to the process. The authors first decompose the change the turn divergence along a tangent vector -- the natural gradient of the OPD loss function. This gives them a first order estimate of the change in turn divergence. They then decompose this into a predictable part using a decomposition. This is normalized in the group and then across turns which creates a turn-level weight for DT-OPD. The authors then perform experiments for both ALF-World and SearchQA. Results show improvements in some cases.

Critical strengths

This paper has the following strengths:

  • The authors begin with intuition that "all turns are not equal." They provide a concrete basis for their approach based on divergence.
  • The initial ideas in Section 4 are interesting and compelling. The information geometry approach is intuitive and the authors do a reasonable job of transitioning from the initial turn-level divergence formulation to a concrete weighting strategy.
  • The paper also impressively transitions from a complex derivation to a practical algorithm and experiments across multiple different benchmarks.
  • The authors supplement the main paper with an extensive appendix that explains many of the details. Overall, a very interesting paper that took a while to digest and understand, but the depth is appreciated along with a practical algorithm and reasonable benchmarks.
Critical weaknesses

The paper has two main weaknesses:

  • The derivations and intuition are difficult to follow. There are multiple places where a few transitional explanation phrases would help out immensely. A few cases:
  • There is a rather big gap in reasoning from lines 202-207 to equation (10). First the equation uses DOPDturnD_{OPD}^{turn} which looks like a divergence, but this is really more like a directional derivative (DD to ΔD{\Delta D}?). Second, there's a big leap from the statement "the directional derivative of a functional ..." to the first inner product in (10) -- the functional is Dturn(π)D_{turn}(\pi) and the tangent vector is gradF(Ja(π)){\rm grad}_F(J_a(\pi)) I assume. AOPD(a,s)=−L(a∣s)A^{OPD}(a,s) = -L(a|s) from (8), but now both are in the equation as different quantities (maybe some intuition why you wrote it this way).
  • Equation (15) is not well motivated and appears out of nowhere.
  • Somehow the weights in (16) which are dependent on both t and i are broadcast to wtw_t which is only dependent on tt; this also appears in the algorithm as wt=wi,tDw_t= w_{i,t}^D (?).
  • The reader persists to the end only to be greeted by experiments that seem weak. For instance, in Figure 1, it looks like the authors cherry-picked the stopping point to ensure DT-OPD had a better metric.
  • Similarly Table 1 is all over the place for best results.

Overall, the paper is a difficult read with only moderately convincing results.

Nits:

  • There is a constant definition of new quantities -- some are also not standard. E.g., equation (5) is called a "divergence score", but why not just say likelihood ratio.
  • Having Appendix A.1 seems unnecessary -- this is a very standard result
  • As mentioned earlier, the level of detail does not match the paper's audience -- e.g., lines 136-186 cover standard terminology and could be condensed to make room for more explanation about the information geometry.
Questions for authors

Given the weaknesses, questions for the authors:

  1. The authors need to make the paper more accessible. Can the authors provide more intuition for various steps in their approach? (e.g., (15))
  2. Is there a way to make the experiments more compelling or explain the behavior observed (see critical weaknesses section)
5 comments
AuthorsOpenReview ↗

Weaknesses 1-5. Thank you so much for your review.

W1:Gap from lines 202–207 to Eq. (10); DturnOPDD^{OPD}_{turn} looks like a divergence

Call back: The derivation of Eq. (10) contains a leap, and DturnOPDD^{OPD}_{turn} looks like a divergence.

Answer: DturnOPD(a;s)D^{OPD}_{turn}(a;s) is a divergence drift — the turn-level teacher–student divergence drift of Theorem 4.1.2: it measures in which direction, and by how much, one OPD update on response aa moves the teacher–student divergence. The subscript names the tracked divergence, the superscript the update rule; it is not a new divergence.

We introduce it because DT-OPD must answer one question: does training on a response shrink or enlarge the teacher–student gap? Eq. (10) does exactly one thing — it writes the rate of change of the divergence along the OPD update direction as the Fisher–Rao inner product of two natural gradients: precisely the quoted fact about directional derivatives, with no further leap.

The two factors match intuition: the strength and direction of the update itself, carried by the stop-gradient coefficient AOPDA^{OPD} of the OPD objective, and the excess of this response's divergence over the turn average, L(a∣s)−Dturn(s)L(a|s)-D_{turn}(s). Their product says higher-relative-divergence responses couple more strongly to the change of the divergence — the basis for allocating the budget by relative divergence.


W2:Eq. (15) is not well motivated

Call back: Eq. (15) is not well motivated and appears out of nowhere.

Answer: Eq. (15) follows directly from the criterion established by Theorem 4.1.2. The theorem ties turn importance to the relative divergence ρ(a∣s)=L(a∣s)−Dturn(s)\rho(a|s)=L(a|s)-D_{turn}(s) — an ordering-and-allocation criterion: what matters is which turns within a prompt have relatively higher divergence, not their absolute values.

DT-OPD therefore never estimates LL and DturnD_{turn} separately; instead, it compares each response's divergence score with those of the other responses in its group and lets the within-group relative magnitude set the turn's weight. Eq. (15) is built on this group-relative principle: it maps the raw scores onto a common [0,1][0,1] scale so that the weights carry only the within-group relative order.

This rescaling is needed because divergence scores are unbounded above and their scale varies across prompts and students — only on a common scale does the temperature λD\lambda_D of Eq. (16) have a uniform meaning across settings, and no extreme score can dominate a group's budget.

This is the same principle as GRPO normalizing advantages within a group: all rollouts and turns of one prompt form a group, and the weights should reflect the within-group order rather than the group's overall level; Appendix B.1 registers the corresponding properties, namely shift-invariance, order preservation, and boundedness.


W3:wi,tDw_{i,t}^D is broadcast to wtw_t

Call back: Why are the weights of Eq. (16), dependent on both ii and tt, broadcast to wtw_t, which depends only on tt?

Answer: This is a notational slip, not a conceptual ambiguity. DT-OPD's turn weights are always defined per rollout–turn pair, wi,tDw_{i,t}^D: the response of rollout ii at turn tt receives its own weight by normalizing its score D~i,t\tilde D_{i,t} within the prompt group via Eq. (16), and different rollouts generally have different weights at the same turn — exactly the intent of the within-group relative comparison. The wtw_t in Section 4.3 and the algorithm is shorthand within a fixed rollout, where the turn index tt uniquely identifies wi,tDw_{i,t}^D. The revision will unify the notation to wi,tDw_{i,t}^D throughout.


W4:Figure 1 looks cherry-picked

Call back: Figure 1's stopping point seems cherry-picked to favor DT-OPD.

Answer: There is no cherry-picked stopping point. Following the protocol of prior turn-level distillation work such as SDAR and ATOD, we select step 150 as the evaluation point: all methods — DT-OPD, OPD, SOD, TCOD — run under exactly the same fixed 150-step budget.

  1. SDAR:Self-Distilled Agentic Reinforcement Learning (https://arxiv.org/abs/2605.15155);
  2. ATOD:Annealed Turn-Aware On-Policy Distillation for Multi-Turn Agentic Tasks (https://arxiv.org/abs/2606.27814)

W5:Table 1's best results are scattered

Call back: Table 1's best results are scattered across categories, showing no consistent advantage.

Answer: The per-category breakdown is a deliberate choice: the six ALFWorld categories and seven Search-QA subsets differ greatly in difficulty and interaction pattern, so methods naturally trade wins on individual categories. At the aggregate level, DT-OPD's advantage is consistent and stable: the highest ALFWorld average at every student size and the highest Search-QA average in every block. DT-OPD thus delivers the best overall performance at every scale, rather than winning on isolated categories.

N1:There is a constant definition of new quantities

Call back: Why coin "divergence score" instead of the standard "likelihood ratio"?

Answer: Thank you for pointing this out. The naming was a deliberate choice on our part: we call it the "divergence score" because its expectation under the student equals the turn-level KL divergence E[L]=Dturn(s)E[L] = D_{turn(s)}, and the name is meant to highlight this direct link between a sample-level quantity and the turn-level divergence, echoing the "divergence-calibrated" theme of the paper. That said, you are right that it is essentially the log likelihood ratio between the student and the teacher, and standard terminology is more readable. In the revision, we will rename it "log likelihood ratio" and note the expectation relation above at its first occurrence to preserve this clue; we will also audit the whole paper to merge or remove non-essential new definitions in favor of established terms.


N2:Having Appendix A.1 seems unnecessary?

Call back: Appendix A.1 proves a standard result — is it necessary?

Answer: Thank you for pointing this out. We agree that this identity is the standard score-function gradient result; that said, keeping A.1 is a deliberate choice on our part: it lets the stop-gradient construction in Proposition 4.1.1 be fully checked step by step and serves as the fixed starting point for the derivation of Theorem 4.1.2, and it sits in the appendix at no cost to the main text, so we prefer to keep it. If you still find it redundant, we will remove A.1 and replace it with a one-sentence note in the main text.


N3:Lines 136-186 cover standard terminology and could be condensed to make room for more explanation about the information geometry.

Call back: Can the standard background (L136–186) be condensed to make room for the information geometry?

Answer: Thank you for this constructive suggestion. We agree that Section 3 can and should be condensed, and we commit to doing so. At the same time, we would like to keep one consideration on record: our intended readers also include colleagues from neighboring areas who may not share this subfield's background, and the setup and notation in Section 3 serve as a self-contained entry point for them, so it should not be removed wholesale.

In the revision, we will:

(i) compress Section 3 by about half — purely standard content (the MDP assumption, the token-level decomposition, etc.) moves to the appendix, keeping only the definitions needed for our notation plus one sentence of intuition;

(ii) devote the freed space to expanding the information-geometry exposition in Section 4.1: giving the explicit form of the Fisher–Rao inner product, unfolding the step-by-step substitution from the directional derivative to Eq. (10), and explaining the intuitive roles of AOPDA_{OPD} and the relative divergence ρρ.

AuthorsOpenReview ↗

Questions 1. More intuition for the steps of the approach, especially Eq. (15)

Q1: More intuition for the steps of the approach, especially Eq. (15)

Call back: You asked for a more accessible paper, with more intuition for the various steps of the approach, e.g., Eq. (15).

Answer: Thank you for the suggestion — we fully agree that the derivation in Eqs. (13)–(17) is written too compactly. The revision will merge the following step-by-step intuition into Sections 4.2–4.3, so that readers can see why each step is designed the way it is without consulting the appendix:

1.Eq. (13), token-level divergence: This directly measures how far the student is from the teacher at every position. The positions with large divergence are exactly where the student has not yet learned the teacher's behavior — the most primitive signal of "where more learning is needed."

  1. Eq. (14), length normalization: Longer responses naturally accumulate more divergence terms. Dividing by length makes short and long turns comparable, so that the weight is determined by the degree of disagreement rather than by response length.

3.Eq. (15), within-group truncation + min-max normalization: The intuition is that we only care about the relative divergence ranking among the turns of the same prompt, in two steps: (i) within-group quantile truncation prevents a few extreme values from compressing the scores of all remaining turns into a flat cluster and erasing their differences; (ii) within-group min-max normalization maps scores whose magnitudes differ widely across prompts onto a common relative scale of [0, 1], with ε only guarding against division by zero when all scores in a group are identical. From this point on, the weights depend only on the within-group ordering.

4.Eq. (16), self-calibrated softmax: This maps the relative scores to strictly positive weights whose group mean is approximately 1 — the total distillation budget of each prompt is unchanged; the budget is only redistributed: turns with larger relative divergence receive more supervision, and turns with smaller relative divergence receive less.

5.Eq. (17), weighted objective: The weight is broadcast to every token of the turn and stopped from receiving gradients, entering the original OPD objective as a constant multiplier — zero additional overhead, with no extra rollouts or model evaluations.

Summary: Eqs. (13)–(17) form a single chain: first measure "how far apart" the student and teacher are (13), then remove the interference of length and scale (14)–(15), and finally redistribute supervision according to relative divergence while keeping the budget unchanged (16)–(17). The revision will place an intuition paragraph before each of these equations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.