- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes LAMP for multi-teacher on-policy distillation. It regularizes the student toward a frozen SFT checkpoint and adjusts the regularization weight according to a predefined KL budget. Experiments across several model families and domain mixtures show improved aggregate performance, particularly in instruction following. The paper also provides conditional theoretical results on stationarity and optimization of the regularized objective.
Critical strengths
-
Combining specialist capabilities while preserving existing abilities is a relevant practical problem. The proposed mechanism is simple and can be incorporated into existing distillation pipelines.
-
The ablation includes fixed KL penalties and fixed teacher–anchor mixtures, providing useful comparisons for assessing adaptive anchoring.
Critical weaknesses
- The main results and ablations rely heavily on older models. The main comparisons use Qwen2, Qwen2.5, and Gemma 2, while the ablation comparing adaptive anchoring with fixed KL penalties and FixedMix is limited to Qwen2-1.5B. Although the paper includes Qwen3-14B-to-8B distillation, these ablation comparisons are not repeated in that setting. Extending the main experiments and ablations to newer models, such as Qwen3-30B-A3B, would strengthen the evidence that both LAMP’s overall gains and the benefit of its adaptive mechanism generalize beyond the older experimental setups.
- The relationship between anchor KL and capability preservation needs stronger justification. A task can admit multiple correct solutions, and teachers may share capabilities while preferring different reasoning paths or response styles. Learning these alternatives can increase KL from the anchor without reducing task performance. Anchoring may therefore restrict useful changes as well as harmful ones. The paper does not sufficiently justify the budget of 0.15 or examine how different budgets affect performance across domains.
- Baselines like rehearsal are missing. The paper includes fixed KL penalties and fixed teacher–anchor mixtures, but does not compare against rehearsal. Since the original SFT data are available, a straightforward baseline is to interleave MOPD with supervised updates on those examples. A comparison with tuned rehearsal weights and comparable training compute would help establish whether LAMP offers an advantage over this simple approach to preserving existing capabilities.
- The theory does not guarantee satisfaction of the KL constraint. Theorem 5.1 establishes subsequential primal stationarity, while Theorem 5.2 analyzes optimization of the regularized objective at the current penalty weight. Neither result establishes that anchor KL eventually satisfies the prescribed budget or bounds the remaining constraint violation. These results therefore provide limited theoretical support for the ability of the dual update to enforce the KL constraint.
Questions for authors
See weaknesses.