acceptodds
Under review as a conference paper at ICLR 2027

From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation

31 views

Abstract

Multi-teacher on-policy distillation (MOPD) aims to combine the strengths of RL-trained teachers in a single student, but how teacher signals affect parameter changes remains underexplored. We study Qwen3-1.7B with four domain teachers trained with RL from the same initialization as the student, comparing gradients, optimizer updates, and task learning curves, with additional SmolLM3-3B diagnostics. We find that several factors influence teacher signals. First, loss averaging implicitly weights responses: token averaging favors longer responses, and equalizing domain contributions retains this weighting within domains. Second, Adam's first moment reduces differences in parameter updates: the cosine similarity is 0.83 between teachers and 0.96 between averaging rules, despite differences in raw gradients. Third, BF16 rounding hides small changes: about 97% of FP32 master weights differ from initialization, but only 7–11% of BF16 weights do. Finally, the top-64 intersection KL gradient closely matches Qwen's full-vocabulary gradient, but the effect on task performance depends on averaging: mathematics accuracy is 2.6 points higher than with sampled-token policy-gradient (PG) under response averaging and 2.1 points lower under global token averaging.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, LLM post-training, multi-teacher distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cxfxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper did an empirical study on Multi-teacher on-policy distillation. Specifically, Qwen3-1.7B/SmolLM3-3B with four domain teachers trained with RL from the same initialization as the student, and they investigated different loss averaging schemes, optimizer updates, and task learning curves. The authors claimed that they found token-level averaging favors longer responses, Adam first moment reduces differences, and BF16 rounding hides many small parameter changes visible in FP32 during training. The authors did experiments across four downstream benchmarks with five seeds.

Critical strengths
  • The experiments are solid, as the authors did across different benchmarks and multiple seeds.
  • The results provided several useful insights from loss aggregation, optimizer state, numerical precision, and vocabulary supervision.
Critical weaknesses
  • My main concern is that i don't find a clear and coherent research narrative. While the title says "From Gradients to Capabilities," the paper presents several largely disconnected empirical observations. The paper reads more like a collection of different experiments.

  • Although the authors show that shared Adam momentum substantially increases update similarity across teachers, they do not establish whether this alignment actually causes any corresponding effect. This is also applied to BF16/better gradient approximation.

  • The model scale is another concern.

Questions for authors

What is the central scientific hypothesis of this paper?

Reviewer_3B5cOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper examines different factors which affect the final model produced from MOPD from multiple teachers. The factors explored are: 1) the affect of token lengths and different averaging methods in the loss function, 2) different first-order gradient methods (Adam, SGD), and 3) under quantization. The authors do extensive experiments across multiple domain benchmarks (IF, math, coding, Science) with a small Qwen3 1.7B model for MOPD and show the effect of different first-order gradient methods (Table 1). They also look at how sparsity affects the update under varying precision.

Critical strengths

The paper is a systematic exploration of the various issues facing MOPD:

  1. Varying token lengths per domain as well as convergence rates affects how teachers are folded into the final model. The authors thoroughly explain as well as experiment on these aspects. Table 1 is a good summary of the various effects.
  2. The authors refer to past work and expand upon it (e.g., Open MOPD)
  3. The authors thoroughly explore different hyper-parameters of the process and their interactions (top-k, adam, quantization).
Critical weaknesses

Weaknesses of the paper:

  1. The paper does not add anything substantially novel to the MOPD literature. This paper does not propose new algorithms or methods, it explores current methods in the literature and how they affect the optimization.
  2. The paper does not have any method that "pops." Table 1 shows a fair amount of variation, but given the error bars there no clear winner that the authors have found (at least at the mean level).
  3. There's no real discussion of parameter transfer and slower computation for bf16 and fp32. Sure, it might change the properties of optimization to go to fp32. But, many frameworks use delta transfer of models. I.e., for large models the transfer time and compute savings from bf16 may be critical -- fp32 is not an option.
Questions for authors

None -- authors should see if they can address weaknesses.

Reviewer_4eMdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how multi-teacher on-policy distillation (MOPD) transfers specialized capabilities into a shared student. Using Qwen3-1.7B with four reinforcement-learning-trained teachers, plus supplementary SmolLM3-3B diagnostics, the authors examine loss weighting, optimizer dynamics, numerical precision, and vocabulary supervision.

They show that token-level loss averaging implicitly favors longer responses; balancing domains alone does not remove this bias within domains. More notably, apparently aligned teacher updates largely reflect shared Adam momentum: clearing the first moment or subtracting a zero-current-gradient update reveals much less aligned contributions. BF16 rounding also makes widespread parameter changes appear sparse: approximately 97% of FP32 master weights change, compared with only 7–11% after BF16 rounding. Finally, top-64 intersection KL closely approximates full-vocabulary gradients in Qwen, yet its task gains depend on loss averaging.

The paper’s main contribution is a controlled diagnosis of MOPD rather than a universally superior algorithm. It demonstrates that update alignment, apparent sparsity, and gradient fidelity require careful interpretation and do not independently establish successful capability transfer.

Critical strengths
  1. Careful controls distinguish teacher supervision from optimizer history. By holding student parameters and Adam state fixed, then resetting the first moment or subtracting the zero-current-gradient update, the authors show that apparently aligned teacher updates largely reflect shared momentum. This is consequential because it challenges the use of update similarity as evidence of compatible teacher signals and provides a concrete methodology for making such analyses more reliable.

  2. The paper connects local diagnostics to end-to-end task outcomes. It demonstrates that highly accurate gradient approximation does not consistently translate into better capabilities: top-64 intersection KL closely matches full-vocabulary gradients in Qwen, yet its mathematics gains reverse sign across loss-averaging rules. This materially strengthens the paper by testing the practical implications of its diagnostics rather than assuming them, motivating joint evaluation of loss weighting, supervision breadth, and task performance.

Critical weaknesses
  1. The optimizer analysis establishes a diagnostic effect but does not isolate its consequences for learning. Resetting Adam’s first moment convincingly shows that shared momentum explains much of the observed update alignment. However, it does not establish whether this alignment impairs capability integration. The SGD comparison simultaneously changes momentum, adaptive preconditioning, and learning rate, so its performance advantage cannot be attributed specifically to removing historical momentum. This limits the paper’s ability to connect its strongest mechanistic observation to actionable improvements in MOPD.

  2. The vocabulary-supervision experiments leave the explanation for capability differences unresolved. End-to-end training compares PG with renormalized top-k intersection KL, without a full-vocabulary KL training baseline. These alternatives differ in both sampling noise and objective formulation. Consequently, the experiments demonstrate that local gradient fidelity alone does not predict task gains, but do not determine whether the observed differences arise from variance, truncation, renormalization, or their interaction with optimization. This gap is consequential because the proposed connection from gradient approximation to capability outcomes remains primarily observational.

Questions for authors

Clarification on the optimizer comparison: Were Adam and SGD given comparable learning-rate tuning budgets? Do you have results from varying Adam’s β₁ while keeping other settings fixed? These details would help determine whether SGD’s advantage supports the proposed explanation involving historical momentum or reflects other optimizer differences. If such controls are unavailable, please clarify the intended scope of the causal claim.

Clarification on vocabulary supervision: Do you have end-to-end full-vocabulary KL results under both domain-response and global-token averaging, or other evidence separating sampling variance from truncation and renormalization effects? Such evidence would materially strengthen the interpretation of the opposite mathematics gains observed with top-64. If unavailable, is the intended conclusion limited to showing that local gradient fidelity is insufficient to predict downstream performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.