acceptodds
Under review as a conference paper at ICLR 2027

Natural-Gradient Calibration of Multi-Turn On-Policy Distillation

Abstract

On-policy distillation trains a language model by matching a teacher on the student's own rollouts, and the dense token-level signal complements the sparse terminal reward of reinforcement learning. Rollouts of multi-turn agents divide into turns at tool-call boundaries. Recent methods reweight the teacher-matching loss turn by turn from a gap between teacher and student probabilities, from the terminal reward, or from a depth schedule, without estimating how the teacher supervision at a turn contributes to task success. We view on-policy distillation as a policy-gradient step whose advantage is a proxy, the weighted distillation signal, so a turn-level weight scales that proxy toward the true advantage. The weight that fits the proxy best under the student's action distribution is the covariance of the signal with the return divided by the variance of the signal, which is the projection coefficient of the reward gradient onto the distillation direction in the Fisher metric. The projection splits the policy gradient into a calibrated distillation direction and a residual identified by the return alone, so calibrated distillation is the part of reinforcement learning that the teacher supplies. The nonnegative parts of the per-turn projections are also the assignment that gains the most reward in the second-order KL geometry of one update. We instantiate the calibration as *Turn-Level Advantage Calibration in the Trust Region* (**TACT**). The loss already computes the divergence at each position, and adding it to the token log-ratio makes one rollout an unbiased sample of both the covariance and the variance of every turn it visits. TACT therefore estimates the weights from the training batch, pooled over classes of turns, and adds a reinforcement step on the residual in the classes where the signal explains little of the return. Experiments on ALFWorld, ScienceWorld, WebShop, Search-QA, and -bench, on the same rollouts for every method, show that TACT exceeds the strongest existing rule on every environment, by 4.9 points on average over the rule that uses the return, at the same cost as on-policy distillation. The code is available at [TACT Codebase](https://anonymous.4open.science/r/TACT_ICLR-BD0D/README.md).

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, On-Policy Distillation, Multi-turn Agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZhiVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on improving the turn-level weight assignment for multi-turn OPD. The authors view OPD from the perspective of policy gradients and show that the gradient decomposes into a calibrated distillation component and an orthogonal return-driven residual. The proposed method, TACT, uses conditionally centered signals to provide unbiased single-sample estimates from the batch, pools statistics across up to 13 discrete turn classes, and gates a residual RL step when the signal fails to explain reward variance. Evaluated across ALFWorld, ScienceWorld, WebShop, Search-QA, and τ\tau-bench, TACT effectively outperforms various baselines.

Critical strengths
  1. The proof is sound and the resulting formulation is elegant.
  2. Solid experimental results comparing well-chosen baselines and showing consistent empirical gains
Critical weaknesses
  1. TACT needs to pool estimates into 13 coarse, hand-crafted buckets, which can be suboptimal and may require extra engineering effort; this manual design might also affect performance when applied to other tasks.
  2. Missing efficiency analysis. The number of turns and reasoning tokens can be essential for multi-turn reasoning tasks. Without these results, it is hard to judge fairly whether the method is really better than the baselines.
Questions for authors
  1. The derivation does not seem limited to multi-turn tasks. Is it possible to extend it to more general tasks?
  2. Instead of implicitly inferring the advantage, how about directly learning a value model? Can you discuss the potential trade-offs between the two paradigms?
Reviewer_FJSjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TACT, a method for calibrating turn-level supervision in on-policy distillation for multi-turn agents. It treats the teacher–student distillation signal as a proxy for the student’s reward advantage and derives a calibration coefficient based on the signal–return covariance divided by the signal variance. The method estimates these quantities from training batches, pools observations over predefined turn classes, and adds a residual reinforcement-learning update where the teacher signal explains little of the return. Experiments cover five agent environments, two Qwen student sizes, and an additional Llama teacher–student pair.

Critical strengths
  1. The paper addresses a meaningful problem. Teacher preference does not necessarily align with task success. Calibrating teacher supervision using environmental outcomes is a sensible improvement over weighting rules based only on policy disagreement or turn depth.

  2. The method has a coherent theoretical motivation. The least-squares projection gives an interpretable calibration coefficient, and the trust-region analysis provides a useful local justification for allocating supervision across turns. The decomposition into calibrated distillation and residual reinforcement learning is also conceptually interesting.

  3. The empirical evaluation is reasonably broad. The reported improvements are consistent across environments and model configurations, including an average gain of approximately 4.9 points over RLSD on the primary metrics. The ablations examine important components, including centering, turn classes, the variance floor, and residual updates.

Critical weaknesses
  1. The comparison with directly related reward-aware distillation methods is incomplete. RLAD/TRRD, proposed in Reinforcement-aware Knowledge Distillation for LLM Reasoning (https://arxiv.org/abs/2602.22495), also addresses teacher–advantage misalignment through advantage-aware, trust-region-style updates on student rollouts. This makes it directly relevant to the paper's motivation. Comparisons with RLSD and a fixed OPD-plus-RL ablation do not fully establish TACT's advantage over this approach. A direct comparison on representative environments, with matched teacher information and training budgets, would strengthen the empirical contribution. The paper should also clearly distinguish its covariance-based calibration and residual decomposition from RLAD's teacher-anchored ratio objective.

  2. The presentation makes the method unnecessarily difficult to follow. The paper moves quickly between state-specific theoretical coefficients, class-level estimates, and clipped execution weights. Similarly, the transition from token-level residual advantages to the implemented turn-level residual estimator is insufficiently explained. Important implementation details, particularly the exact turn-class partition, appear only in the appendix. A worked numerical example and a clearer separation between theoretical identities, local approximations, and implementation choices would substantially improve accessibility and verifiability.

  3. The shared-rollout protocol needs clarification. The paper repeatedly states that all methods train on the same rollouts and tool calls. Since different updates produce different student policies, their on-policy rollout distributions would generally diverge. It is unclear whether "same rollouts" means identical generated trajectories or matched tasks, seeds, and sampling budgets. This distinction matters for interpreting the comparison.

  4. The description of negative-gain classes appears inconsistent with Algorithm 1. Section 4.2 states that negative-gain classes receive residual RL updates, but Algorithm 1 gates the residual using an explained-share statistic that squares the estimated gain. A strongly negative relationship can therefore have a high explained share, potentially disabling both distillation and residual RL. The intended behavior should be clarified.

Questions for authors
  1. Can the authors provide a controlled comparison with RLAD/TRRD, or explain the obstacles to adapting it to the multi-turn agent setting?

  2. What exactly is shared across methods in the "same rollouts" protocol? If identical trajectories are reused, which policy generates them, and how is the on-policy interpretation maintained?

  3. Does the residual gate contain an additional rule for negative-gain classes that is omitted from Algorithm 1?

  4. How should the unbiasedness claims be interpreted when the batch baseline includes the current rollout and the calibration weights are estimated from the same data used for the update?

Reviewer_XjCvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TACT, a method that adjusts how much a student learns from its teacher at each turn in multi-turn on-policy distillation. It estimates turn weights from existing rollouts using the covariance between a centered teacher–student signal and the final reward, divided by the signal’s variance. The authors give these weights a natural-gradient interpretation and add a residual reinforcement learning update where the teacher signal explains little of the reward. Results across five agent benchmarks show consistent improvements over the tested baselines.

Critical strengths
  1. TACT connects the strength of teacher guidance to task reward, rather than relying only on teacher–student disagreement or turn depth. Its centered signal allows the required statistics to be estimated from existing rollouts, making the idea practical without extra generation. 4223_Natural_Gradient_Calibrat2.

  2. TACT outperforms the tested baselines across five environments at both Qwen student sizes. The ablations help identify which components matter: removing centering or the variance floor causes clear drops, while calibrated distillation without the residual update still performs well on ALFWorld.

Critical weaknesses
  1. The natural-gradient and trust-region analysis studies an idealized update to action-block distributions, but the implementation uses AdamW on shared model parameters. This leaves an important gap between the theoretical claims and the implemented method.

  2. Algorithm 1 scales and clips the distillation weight but subtracts the original weight when forming the residual. Its gate also uses squared covariance, so a strong negative relationship between the teacher signal and reward can switch off both updates. It needs clarification or correction.

Questions for authors

Why does Algorithm 1 scale and clip the distillation weight but use the original weight to compute the residual?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.