acceptodds
Withdrawn from ICLR 2027

B-MOPD: Data and Gradient–Gap Balancing for Multi-Teacher On-Policy Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-teacher on-policy distillation trains one student from several specialized teachers on responses generated by the student itself. A natural mixture of training data can give domains unequal exposure, while equal loss weights can still produce unequal learning signals. We introduce B-MOPD, a two-stage allocation method that first balances input-token exposure across domains, then adjusts teacher loss weights using gradient strength and the current teacher–student distributional gap. Gradient normalization compensates for unequal signal magnitudes; inverse-gap weighting gives greater relative emphasis to teachers closer to the student. A causal controller estimates both quantities from completed training updates, smooths the estimates, and bounds the weight ratios. Across mathematics, code, and knowledge, B-MOPD achieves macro accuracies of 46.84% and 72.70% for Qwen2.5-0.5B-Instruct and Qwen2.5-1.5B-Instruct on a 753-question diagnostic evaluation. Natural-data uniform baselines obtain 42.79% and 71.50%, respectively. Component comparisons favor the combined gradient–inverse rule over strength-only and gradient–direct weighting in both configurations. We further examine teacher-relative recovery and per-domain trade-offs to characterize the resulting allocation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, multi-teacher distillation, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vGRGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes B-MOPD, a two-stage method for MOPD. It balances input-token exposure across domains, then adjusts teacher weights using estimated gradient strength and the teacher-student reverse KL gap, giving greater emphasis to teachers with smaller gaps.

Critical strengths
  • The idea of balancing input exposure and gradient strength is intuitive, and the reported improvements across domains partly support the method's motivation.
Critical weaknesses
  • For B-MOPD, in stage 2, it includes several hyperparameters, such as the weight-ratio cap, estimation window, and EMA coefficient, but it does no provide any sensitivity analysis of these settings. The cap and window are described as "fixed empirical choices." This leaves unclear whether the method works consistently on different MOPD scenarios, such as including more domains, highly imbalanced data mixtures, and teachers from different model architectures.
  • The experiments are limited to two small students from the Qwen2.5 family and three domains. Comparisons with existing MOPD methods, such as [1], are missing. The related-work discussion also gives limited coverage of MOPD, making it difficult to assess the method's contribution relative to prior approaches.

[1] Gao, Huan-ang, et al. "Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation." arXiv preprint arXiv:2608.19098 (2026).

Questions for authors
  • Do the same hyperparameter settings remain effective across different MOPD configurations?
  • How does B-MOPD compare with existing MOPD methods such as [1]?
Reviewer_E5YKOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper is focusing on multi-teacher on-policy distillation,especially the issue of data imbalance. This paper proposes a two stage method named B-MOPD. In the first stage,input-token exposure is balanced across domains and in the second stage, this method adjusts loss weights with gradient and the gap between teacher and student distributions.

Critical strengths

1 This paper is logically clear and accurately describes the proposed method.

2 this paper has a strong motivation.

Critical weaknesses

1.The paper lacks discussion and citations regarding certain related works,like multi-teacher approaches for refining AE-KD[1] and the relevant GOVERN[2]. The author needs to be presented more comprehensively.

2.The authors conducted only two sets of experiments covering only three domains, which is insufficient to effectively demonstrate the proposed method's performance.

3.The proposed method appears to focus on balancing when multiple teacher in distillation, yet it does not discuss scenarios involving conflicts between multiple domains.

[1]Agree to Disagree: Adaptive Ensemble Knowledge Distillation in Gradient Space,neurips 2020

[2]GOVERN: Gradient Orientation Vote Ensemble for Multi-Teacher Reinforced Distillation,EMNLP 2024

Questions for authors

1.Regarding the first stage that balancing input exposure, however, what if a specific domain naturally requires more input due to inherent complexity or difficulty in learning? Has this possibility been considered?

2.What happens if there are two domains are similar or mutually exclusive? Would the proposed method fail in such cases?

Reviewer_B3jEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes B-MOPD, a two-stage allocation method for multi-teacher on-policy distillation. First, it balances training exposure by the number of raw input tokens across domains. Second, it adjusts teacher-specific loss coefficients using the norm of each domain's unweighted mean gradient and the conditional reverse KL between the student and its domain teacher. The proposed inverse-gap rule gives relatively greater weight to teachers whose distributions are closer to the student, after correcting for planned exposure and gradient magnitude.

The method is evaluated with two Qwen2.5 student configurations and three domains: mathematics, code, and general knowledge. It reports macro-accuracy improvements over natural-data uniform weighting of 4.05 percentage points for the 0.5B student and 1.20 points for the 1.5B student.

Critical strengths
  1. A clearly motivated problem. The paper distinguishes prompt exposure, gradient magnitude, and teacher–student distributional disagreement.

  2. Reasonable experimental decomposition. The authors compare natural versus input-token-balanced schedules, strength-only weighting, direct-gap weighting, and inverse-gap weighting.

Critical weaknesses
  1. Limited evidence for the central methodological claim. The main result is promising, but the experimental evidence is narrow. The study uses only two student sizes, three domains, one training seed, and a fixed 128-update horizon. The diagnostic evaluation contains 753 questions, and the authors explicitly state that the panel was inspected during method development.

The improvements are modest in the larger configuration: B-MOPD achieves 72.70% versus 71.50% for natural-data Uniform, and 72.70% versus 72.34% for the matched-gradient-engine Uniform control.

Without independent training seeds, uncertainty estimates, or an untouched evaluation set, it is difficult to determine whether the smaller gains are robust or specific to this training trajectory and diagnostic panel.

  1. The method's novelty relative to concurrent work needs clarification. The paper cites concurrent work on multi-teacher on-policy distillation, token balancing, and gap-aware prioritization, but explicitly does not provide a head-to-head comparison.

The proposed combination of input-token balancing, gradient normalization, and inverse-gap weighting is understandable. However, the distinction between a genuinely new allocation principle and a particular combination of existing balancing heuristics remains insufficiently established.

  1. Gradient magnitude is an imperfect proxy for learning contribution. The method uses the norm of each domain's unweighted mean gradient to normalize its contribution. This is a reasonable diagnostic, but gradient magnitude alone does not capture whether gradients from different domains align or conflict.

Two domains can have similar gradient norms but point in opposing directions. In that case, equalizing their scalar contributions does not necessarily improve optimization. Moreover, AdamW's adaptive preconditioning and the final global gradient clipping mean that the resulting parameter updates need not follow the intended gradient-magnitude allocation.

  1. Additional technical questions
  • Loss formulation: Equation 2 is described as a local sampled-action surrogate rather than an exact trajectory gradient. The authors should clarify more explicitly which objective the resulting optimization approximates and how response-length normalization affects the intended distillation behavior.

  • Weight saturation: In the 1.5B configuration, the inverse weights reach the maximum allowed ratio after the first admitted window and then remain fixed. This raises the question of whether the method's performance is primarily driven by an early allocation decision rather than sustained online adaptation.

  • Data-schedule alternatives: Input-token balancing is compared with natural sampling, but not with equal prompt counts or other principled domain-sampling strategies. The proposed schedule's advantage over these alternatives remains unknown.

  • Evaluation metric: Counting only correct answers that naturally terminate is a defensible operational metric, but it combines answer correctness and generation termination. The paper reports these separately in the appendix; the conclusions should make clear whether gains reflect improved reasoning, better termination behavior, or both.

Questions for authors
  1. Can the authors demonstrate that the gains from inverse-gap weighting persist across independent training seeds and an untouched evaluation set?

  2. What empirical evidence supports the hypothesis that smaller teacher–student KL indicates a more useful teacher signal than larger KL?

  3. How does B-MOPD compare with the most relevant concurrent multi-teacher distillation method under a matched compute budget?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.