acceptodds
Under review as a conference paper at ICLR 2027

WEIGHTED ON-POLICY DISTILLATION FOR MULTI- DOMAIN CAPABILITY INTEGRATION

Abstract

Multi-domain post-training seeks to combine specialist skills while preserving capabilities of the starting model. We propose Weighted On-Policy Distillation (WOPD), which evaluates specialists on shared verifiable training prompts and assigns domain-specific teacher weights in the OPD loss based on their success rates. The starting model receives a fixed weight as a reference teacher. All teachers supervise the same student-generated tokens through a weighted reverse-KL objective. In a five-domain study with Qwen3-4B, WOPD exceeds equal weighting on all five in-domain benchmarks by \(1.6\)–\(15.8\) percentage points under the same six-teacher pool. Compared with domain-routing Multi-Teacher On-Policy Distillation (MOPD), it improves the in-domain mean by \(1.1\) points and the out-of-domain mean by \(3.8\) points. WOPD matches the reference model on the out-of-domain math benchmark and exceeds it on the other four. Removing the reference teacher lowers every out-of-domain score by \(3.0\)–\(6.6\) points. At each student prefix, the weighted objective targets a geometric mixture of the teacher distributions, and reference supervision adds an explicit KL anchor to the reference model. Combining score-based specialist weights with reference supervision improves in-domain performance without out-of-domain losses relative to the reference model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on policy distillation, post training

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_juDgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to integrate specialist LLM capabilities while preserving the reference model’s general performance. WOPD combines verifier-derived, domain-specific specialist weights with a fixed reference weight, using a clamped, sampled-token surrogate of weighted reverse KL on student-generated responses. Experiments on Qwen3-4B show improvements over uniform weighting across five in-domain benchmarks, while reference-removal ablations support improved out-of-domain retention. The analysis characterizes the effective teacher as a geometric pool and reference supervision as KL regularization.

Critical strengths

Substantial gains over uniform weighting. With the same teacher pool and training configuration, WOPD improves all five in-domain scores, including Chemistry by 15.80 points and FinQA by 15.00 points (Table 1).

Evidence for reference anchoring. Reference inclusion improves all five out-of-domain scores by 3.0 to 6.6 points, with in-domain changes within 1.8 points (Table 2). This supports retention without separate preservation prompts.

Critical weaknesses

The allocation benefit is not isolated. Joint five-domain training lacks comparisons against reference-regularized routing and uniform specialist weighting with the same reference share. Tables 3, 4, and 8 use different domains, teacher pools, or training settings and do not resolve this gap. The additional teacher-inference cost is also unquantified.

Reproducibility and reliability remain limited. The main training configuration, advantage clamp, verifier implementations, and in-domain test sizes need clarification. Single-run results without uncertainty estimates weaken the smaller gains and retention claims.

Questions for authors

Clarification: Please specify the main five-domain training hyperparameters, advantage clamp, verifier implementations, and in-domain test-set sizes.

Clarification: How were α=0.1\alpha=0.1 and τ=0.3\tau=0.3 selected, and using what data? Table 9’s single-domain, two-teacher study does not establish robustness around the main configuration.

Suggested experiment: Does WOPD outperform modified routing with own-domain specialist/reference weights of 0.9/0.10.9/0.1, and uniform specialist weighting with 0.180.18 per specialist and 0.10.1 for the reference, under matched five-domain training settings?

Suggested experiment: Do the main gains and reference-removal results persist across training seeds, with training variability and paired confidence intervals reported?

Reviewer_XQamOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper suggests using multiple teacher models to perform knowledge distilaltion. The baseline teacher model is trained by a verifier model via GRPO to form five different teacher models. Then, the trained teacher model and their corresponding verifier is used to produce a weight on the distillation loss in the main knowledge distillation process.

Critical strengths

I can't see a strength to accept this paper

Critical weaknesses

The paper does not provide any detail of verifier design at all. The domains of specialist are diverse, so the verifier needs to be diversified as well. There is no mention of such details.

This method seems to have a huge overhead. The main purpose of knowledge distillation is inference on student model with more complex teacher model. This model requires the teacher model inference as well. Often, the teacher model is given without any additional training, but this method is spending the learning time of teacher, which require evaluation on teacher and verifier networks.

There is no specification on number of function evaluation, inference/training cost, etc. This method may spends huge computation cost for the experimental performance gain.

There is no speific reason of model details. why use reverse-KL loss? why softmax or min-max normalization, etc? there are many model designs without any justification or experimental examinations.

Which is the main performance gain reason? Particularly, can you compare "MOPD + reference model teacher" and "MOPD + KL divergence loss with reference model"? Or, are these the same model? In that case, does the performance gain only come from adding the reference model to the teacher?

Questions for authors

see the weakness section

Reviewer_hegoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Weighted On-Policy Distillation (WOPD) to combine specialized language models into one student. It evaluates each specialist across domains, gives stronger models greater influence, and keeps the original model as a teacher to reduce forgetting.

Critical strengths

The paper addresses the important challenge of combining specialized LLM capabilities while preserving the original model's general knowledge. The experiments show promising improvements over existing baselines.

Critical weaknesses

Regardless of the problem novelty and the performance improvement, I have the following concerns about the paper.

  1. Limited contributions in terms of technical perspective: most of the components of the proposed methods have been studied and proposed by others. Specifically, the idea of multiple teachers with special domain knowledge to teach a single student has been developed. Therefore, the contributions are incremental.
  2. Limited experimental validation, only on Qwen3-B, which does not guarantee the technique works on other LLMs. This can raise the issue of cherry-picking. In addition, the baseline simulations are very limited; what if we have a list of specialists and we consider uniform weighting instead of weighted on-policy distillation?
  3. The additional computation requires multiple teachers to evaluate the student-generated token.
Questions for authors

Please answer all the weakness points that I raise in the weakness part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.