acceptodds
Under review as a conference paper at ICLR 2027

Balancing Specialists with Safety: Lagrangian Anchored Multi-Teacher On-Policy Distillation for Large Language Models

Abstract

Multi-Teacher On-Policy Distillation (MOPD) merges diverse expert capabilities into a single student model. However, current methods rely on rigid routing or heuristics, making them highly vulnerable to multi-domain gradient conflicts. These unconstrained updates often cause a severe “see-saw” effect, where acquiring new skills catastrophically erodes previously established foundational capabilities. To address this, we propose Lagrangian Anchored Multi-Teacher Post-Training (**LAMP**), a framework that protects foundational knowledge by dynamically constraining policy updates. LAMP anchors the student to an initial reference model and employs a self-adjusting penalty mechanism: when aggressive specialist updates threaten to distort core capabilities, LAMP automatically scales up an anchoring penalty to pull the student back toward the reference distribution. Under explicit regularity and step-size assumptions, a theoretical analysis establishes subsequential primal stationarity and a Polyak-Lojasiewicz-based tracking rate. Experiments across model families with different architectures and scales show that LAMP improves aggregate multi-domain performance over strong MOPD baselines under different settings.

Primary area:
generative models
Keywords:
Knowledge Distillation, Large Language Model Post-Training, Constrained Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_gnLJOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LAMP for multi-teacher on-policy distillation. It regularizes the student toward a frozen SFT checkpoint and adjusts the regularization weight according to a predefined KL budget. Experiments across several model families and domain mixtures show improved aggregate performance, particularly in instruction following. The paper also provides conditional theoretical results on stationarity and optimization of the regularized objective.

Critical strengths
  1. Combining specialist capabilities while preserving existing abilities is a relevant practical problem. The proposed mechanism is simple and can be incorporated into existing distillation pipelines.

  2. The ablation includes fixed KL penalties and fixed teacher–anchor mixtures, providing useful comparisons for assessing adaptive anchoring.

Critical weaknesses
  1. The main results and ablations rely heavily on older models. The main comparisons use Qwen2, Qwen2.5, and Gemma 2, while the ablation comparing adaptive anchoring with fixed KL penalties and FixedMix is limited to Qwen2-1.5B. Although the paper includes Qwen3-14B-to-8B distillation, these ablation comparisons are not repeated in that setting. Extending the main experiments and ablations to newer models, such as Qwen3-30B-A3B, would strengthen the evidence that both LAMP’s overall gains and the benefit of its adaptive mechanism generalize beyond the older experimental setups.
  2. The relationship between anchor KL and capability preservation needs stronger justification. A task can admit multiple correct solutions, and teachers may share capabilities while preferring different reasoning paths or response styles. Learning these alternatives can increase KL from the anchor without reducing task performance. Anchoring may therefore restrict useful changes as well as harmful ones. The paper does not sufficiently justify the budget of 0.15 or examine how different budgets affect performance across domains.
  3. Baselines like rehearsal are missing. The paper includes fixed KL penalties and fixed teacher–anchor mixtures, but does not compare against rehearsal. Since the original SFT data are available, a straightforward baseline is to interleave MOPD with supervised updates on those examples. A comparison with tuned rehearsal weights and comparable training compute would help establish whether LAMP offers an advantage over this simple approach to preserving existing capabilities.
  4. The theory does not guarantee satisfaction of the KL constraint. Theorem 5.1 establishes subsequential primal stationarity, while Theorem 5.2 analyzes optimization of the regularized objective at the current penalty weight. Neither result establishes that anchor KL eventually satisfies the prescribed budget or bounds the remaining constraint violation. These results therefore provide limited theoretical support for the ability of the dual update to enforce the KL constraint.
Questions for authors

See weaknesses.

Reviewer_TTbBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LAMP to mitigate capability degradation during multi-teacher on-policy distillation. The method introduces a frozen reference model and employs primal-dual optimization to dynamically adjust a KL regularization penalty based on a predefined divergence budget. The authors provide theoretical analyses of the optimization properties and evaluate the method across multiple model families and domains, demonstrating improvements over existing MOPD baselines. While the method is simple and empirically effective, its novelty relative to existing reference-based distillation methods and the alignment between its central claims and experimental validation warrant further examination.

Critical strengths
  1. Simple and effective method. LAMP introduces a straightforward adaptive KL regularization mechanism that can be easily integrated into existing on-policy distillation frameworks. Despite its simplicity, the method demonstrates consistent improvements in aggregate performance across multiple model families and domains, supporting its practical value.

  2. Solid theoretical analysis. The paper provides a relatively thorough theoretical analysis of the proposed primal-dual optimization framework, including results on primal stationarity and convergence under explicit assumptions. This strengthens the methodological foundation and provides useful insights into the optimization behavior of adaptive anchoring.

Critical weaknesses
  1. Limited methodological novelty and insufficient justification for adaptive Lagrangian control. The proposed method is closely related to existing reference-based on-policy distillation approaches, particularly G-OPD, which already incorporates reference KL regularization. The main methodological distinction appears to be the introduction of a KL budget and a primal-dual update to dynamically adjust the regularization coefficient. While the authors demonstrate improvements over fixed coefficients, it remains unclear whether the proposed feedback-based adaptation offers meaningful advantages over simpler time-dependent regularization schedules. This limits the strength of the paper's methodological contribution.

  2. Insufficient experimental validation of the central claims. The paper motivates LAMP by emphasizing multi-teacher gradient conflicts and catastrophic forgetting of foundational capabilities. However, the experiments primarily demonstrate improvements in aggregate downstream performance, without directly measuring gradient interference or retention of the pre-distillation student's capabilities. Although the imbalanced-mixture experiments provide some evidence of improved cross-domain performance retention, they do not directly establish the proposed mechanisms. This gap between the motivation and experimental evidence weakens the support for the paper's central claims.

Questions for authors
  1. Effectiveness of the adaptive KL constraint. Could the authors provide training trajectories of the Lagrangian multiplier λ\lambda and the Student–Anchor KL divergence relative to the predefined budget ϵ\epsilon? In particular, does LAMP effectively maintain KL near the target budget, and how quickly does λ\lambda respond to constraint violations? How do these behaviors compare with fixed-λ\lambda baselines? These results would help establish whether the proposed primal-dual controller functions as intended.

  2. Necessity of primal-dual adaptation. Given that the primary methodological contribution is dynamically adjusting the KL regularization coefficient, could the authors compare LAMP with simple time-dependent schedules, such as linear or cosine scheduling of λ\lambda? It is important to determine whether the observed improvements arise specifically from KL-feedback-based adaptation or merely from varying regularization strength throughout training.

  3. Validation of gradient conflict mitigation and capability preservation. Could the authors provide direct evidence that LAMP mitigates inter-teacher gradient conflicts and preserves the foundational capabilities of the pre-distillation SFT model? For example, analyses of cross-domain gradient interference and evaluations on held-out general capabilities before and after distillation would help substantiate the central motivation. Notably, the reported MOPD results already outperform the initial SFT checkpoint across the evaluated domain averages, making it difficult to identify where catastrophic forgetting actually occurs.

  4. Clarification of offline teacher log-probability computation. Appendix C.4 states that teacher log-probabilities are pre-computed offline, whereas Algorithm 2 requires evaluating teacher probabilities on student-generated trajectories that change throughout training. Could the authors clarify how these probabilities are computed, cached, or refreshed while maintaining the on-policy nature of the proposed method?

  5. Computational overhead and practical efficiency. Could the authors report the additional computational and memory costs introduced by LAMP compared with standard MOPD, including training time, GPU hours, and peak GPU memory usage? Since the performance improvements over fixed-regularization baselines are relatively modest, understanding the associated computational overhead is important for assessing the method's practical value.

Reviewer_Gt7LOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Lagrangian Anchored Multi-Teacher Post-Training (LAMP) for multi-teacher on-policy distillation. LAMP combines domain-specialist supervision with a KL constraint relative to the frozen initial student, using an adaptive Lagrange multiplier to balance specialist learning against reference-policy drift. The paper provides conditional stationarity and tracking results for an idealized optimization procedure. Experiments across Qwen and Gemma models, including balanced and imbalanced instruction-following, mathematics, and coding mixtures, report improved aggregate performance over MOPD and ExOPD. Ablations compare adaptive anchoring with fixed penalties and fixed teacher–reference mixtures, supplemented by training-trajectory and loss-landscape analyses.

Critical strengths
  1. LAMP improves all three domain averages over MOPD across the Qwen2, Qwen2.5, and Gemma 2 settings. The larger Qwen3 experiments also show advantages under imbalanced mixtures, particularly in instruction following.

  2. LAMP adds a frozen reference model and a scalar adaptive penalty to standard MOPD. Comparisons against fixed penalties and fixed teacher–reference mixtures help distinguish the contribution of anchoring from that of adaptation.

Critical weaknesses
  1. Mismatch between the motivation and the proposed mechanism. The paper motivates LAMP through inter-teacher gradient conflicts, but the method controls only the student’s KL divergence from a frozen reference model. Distributional drift and gradient conflict are distinct, and no direct measurements establish that LAMP reduces the latter. The authors should provide supporting evidence or narrow the claim to drift control.

  2. Insufficient evidence for the adaptive mechanism. In Table 2, LAMP improves the overall score over the best tested fixed penalty and FixedMix by only 1.0 and 0.7 points, respectively, without reporting variability across independent training runs. Moreover, no trajectories of the multiplier, anchor KL, or constraint violation are provided. These diagnostics and broader controlled comparisons are needed to establish the benefit of adaptation.

  3. The motivating forgetting phenomenon is not adequately demonstrated. In Table 1, MOPD improves all three domain averages over the SFT initialization across every model family. The imbalance experiments show degradation relative to balanced training, rather than relative to the initial model. Although aggregate improvements do not rule out localized forgetting, the paper should identify which existing capabilities MOPD actually degrades and demonstrate that LAMP preserves them.

Questions for authors
  1. What evidence establishes that inter-teacher gradient conflict drives the reported degradation, and that LAMP mitigates this conflict rather than simply regularizing distributional drift?

  2. Which capabilities fall below their pre-distillation SFT levels under MOPD, and does LAMP reduce those losses?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.