acceptodds
Under review as a conference paper at ICLR 2027

UP-MOPD: Update Projection in Multi-Teacher On-Policy Distillation

Abstract

On-policy distillation from multiple teachers combines expertise from different domains in a single student, but conflicting gradients can hinder this integration. Gradient corrections directly constrain parameter updates under plain SGD. With optimizers such as AdamW, however, momentum, adaptive scaling, and weight decay can turn a corrected gradient into an update that increases a domain loss to first order. To address this gap, we propose Update Projection for Multi-Teacher On-Policy Distillation (UP-MOPD). UP-MOPD lets the original mixed gradient update the optimizer state and generate a candidate displacement, then projects only violating candidates before they are committed to the parameters. The projection gives the unique feasible update closest to the candidate in Euclidean distance. In experiments combining medical and general domains, UP-MOPD improves IFEval-loose accuracy late in training by 2.96 points over vanilla M-OPD. It achieves an average score of 60.03 across eight metrics, compared with 59.00 for gradient projection and 59.15 for update rejection. On a public benchmark covering mathematics, code, and instruction following, it achieves the best average across six tasks (32.67), leads on LiveCodeBench v5, and ties for the best IFEval result. These results support projecting optimizer updates to reduce interference between domains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Teacher On-Policy Distillation, Knowledge Distillation, Multi-Objective Optimization, Gradient Conflict, Update Projection

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_xaoiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UP-MOPD, which projects AdamW’s candidate parameter update onto a "feasible cone". Unlike gradient projection, UP-MOPD applies the correction after AdamW produces a candidate update, while retaining the optimizer state computed from the original mixed gradient. Experiments cover medical-general integration and a public math-code-instruction benchmark.

Critical strengths
  1. The paper identifies a meaningful limitation of gradient-level conflict handling, namely, AdamW can transform a seemingly safe gradient into a harmful parameter update.
  2. The projection is clearly formulated, has a unique solution, and admits a low-dimensional dual formulation.
  3. The evaluation covers two-/three-domain settings and includes comparisons with gradient projection, update rejection, and reweighting.
Critical weaknesses
  1. The empirical evidence is limited by the lack of multiple training seeds. The paper also reports that none of the eight paired metric differences are significant after multiple-comparison correction.
  2. The Euclidean projection metric is not clearly justified for AdamW.
  3. First-order feasibility does not guarantee that the actual finite update decreases each domain’s loss.
Questions for authors
  1. Are the gains consistent across multiple independent training seeds?
  2. Are domain-gradient passes executed serially or in parallel, and can the authors provide a detailed runtime breakdown?
  3. What is the effect of retaining the optimizer state computed from the original mixed gradient?
  4. Personally, I think that the mixed gradient can be obtained by merging domain-specific gradients, which raises a question: why do you perform repeated computation?
3 comments
AuthorsOpenReview ↗

Response to Reviewer xaoi

Thank you for your careful review and specific suggestions. We have added results from three training seeds, along with loss and runtime analyses, to address your concerns.

W1 / Q1. Training seeds and consistency of results. Under the same evaluation protocol, UP-MOPD achieves six-task average scores of 34.33, 31.47, and 32.16 across three seeds, all exceeding the reported Open-MOPD score of 30.85. The mean and sample standard deviation across the three seeds are reported below:

MetricMean ± Standard Deviation
AIME2426.67 ± 1.11
AIME2524.44 ± 2.94
LCB v525.35 ± 1.92
LCB v620.95 ± 1.19
IFEval76.28 ± 1.14
IFBench22.22 ± 0.69
Overall32.65 ± 1.49

These repeated training runs provide additional evidence of consistency across seeds in the public benchmark setting. The conclusions of the BH-adjusted analysis in Table 12 for the medical setting remain unchanged.

W2. Euclidean metric. We adopt Euclidean distance to implement the design objective of minimally modifying the actual parameter displacement already proposed by AdamW. The candidate displacement incorporates momentum, adaptive scaling, and weight decay; projection preserves this update as closely as possible while satisfying the domain constraints. Proposition 2 guarantees a unique correction, leaves feasible candidates unchanged, and establishes that the exact projection does not increase the displacement’s Euclidean norm. Appendix C.4 presents an extension to positive-definite weighted metrics, with the dual Gram matrix. Thus, the Euclidean metric is the minimum-correction choice adopted and evaluated in this work, rather than a claim about the optimal metric for AdamW.

W3. First-order guarantees and finite-step behavior. Corollary 1 guarantees first-order non-increase on the current batch and characterizes finite-step loss changes through a second-order remainder. The new loss figure compares the original updates and their offline UP projections for the same 645 rejected AdamW candidates from the Update Rejection trajectory. A median relative correction of 1.77% removes the predicted increase in the anchor loss while retaining a median of 96.26% of the predicted medical loss decrease. The improvement in aggregate score over Update Rejection in Table 3 (59.15 → 60.03) further supports the value of this local constraint for capability integration in practice. Our theoretical claim is first-order non-increase, rather than domain-wise monotonic loss decrease under finite updates.

Q2. Runtime breakdown. In the public verl implementation, domain gradients are computed sequentially across domains, with distributed parallelism within each pass. The newly added timing figure for the medical setting shows the cost of each stage. The mean timings corresponding to Table 9 are provided below. Actor time includes forward and backward computation, domain-gradient processing, and optimizer updates.

Stage (seconds per step)M-OPDUP-MOPD
Rollout47.2347.32
Reference log-probabilities3.053.06
Student log-probabilities3.063.07
Actor training11.5739.69
Other1.641.62
Logged end-to-end step66.5594.76

Actor training takes 3.43 times as long as the baseline, while the end-to-end time per step is 1.42 times that of the baseline. The additional actor computation primarily comes from the domain-gradient computations required for parameter projection.

Q3. Optimizer state. AdamW continues to update its momentum using the original mixed gradient. Specifically, we retain the first and second moments computed from that gradient, so AdamW’s historical statistics continue to correspond to the original distillation objective, while the current domain constraints act directly on the committed parameter displacement. The projection correction does not directly enter the moment estimates at the current step. M-OPD and UP-MOPD therefore use the same gradient-computation and state-update rules, and their ablation comparison adds only parameter-displacement projection (59.13 → 60.03 in Table 3). Update Rejection further tests the value of correcting updates relative to skipping them entirely.

Q4. Repeated computation. Thank you for this suggestion. Mathematically, the mixed gradient satisfies g0=∑dqdgdg_0=\sum_d q_d g_d and can be reconstructed from the domain gradients. However, under finite-precision computation such as BF16, gradients aggregated from separate backward passes may differ numerically from those obtained by directly backpropagating the mixed objective. To preserve the original M-OPD gradient-input path, our current implementation passes the result of the mixed backward pass to AdamW and monitors discrepancies through a decomposition-consistency check (Appendix C.3). After assessing the numerical impact, gradient reuse can eliminate this additional backward pass.

Thank you for the detailed response and the additional experiments. The clarifications address my questions about the rationale for Euclidean projection, the scope of the first-order guarantee, and the execution and computational cost of the domain-gradient passes. The explanation of the separate mixed-gradient backward pass also clarifies the current implementation choice. The additional training runs provide encouraging evidence of consistency on the public benchmark.

Some minor limitations remain. In particular, repeated runs of the main baselines would strengthen the comparison, and the new loss analysis uses first-order estimates rather than directly measured post-update losses. The benefit of retaining the original optimizer moments could also be investigated more explicitly. Nevertheless, I do not consider these remaining limitations to undermine the paper’s central contribution or its appropriately scoped conclusions. Overall, the rebuttal has sufficiently addressed my main concerns.

Thank you for taking the time to review our response and the additional experiments. We’re glad they helped address your main concerns, and we appreciate your helpful suggestions for strengthening the work further.

Reviewer_3VzzOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Update Projection for Multi-Teacher On-Policy Distillation (UP-MOPD) to mitigate gradient interference during multi-teacher capability integration. The paper provides theoretical analysis of the projection formulation and evaluates the method in medical–general capability integration and a public mathematics–code–instruction benchmark.

Critical strengths
  1. Well-motivated optimizer-level intervention. The paper identifies a meaningful limitation of gradient-level conflict mitigation under adaptive optimizers. Proposition 1 demonstrates that diagonal preconditioning can invalidate gradient-space feasibility, while Section 3.3 reports disagreements between gradient-based and update-based feasibility checks. These findings provide a concrete motivation for performing projection directly on optimizer-generated parameter updates.
  2. Principled formulation with practical implementation. UP-MOPD formulates conflict mitigation as a convex projection problem that yields a unique minimum-distance feasible update. The domain-dimensional dual formulation reduces the optimization problem to the number of active domains, making the approach practical when the number of teachers is small.
  3. Evaluation across different capability integration settings. The experiments cover both medical–general capability integration and mathematics–code–instruction consolidation using different model families and training frameworks. UP-MOPD achieves the highest aggregate scores in both settings.
Critical weaknesses

1.Substantial computational overhead and limited scalability validation. UP-MOPD requires additional domain-specific gradient computations and projection operations. According to Appendix D.3 (Table 9), its actor training time increases from 11.57 to 39.69 seconds per step compared with M-OPD, approximately a 3.43-fold increase. Moreover, experiments involve only two or three teachers and models of 3B–4B parameters. The performance gains may not sufficiently justify this overhead, particularly in larger-scale settings. 3. Incomplete isolation of the proposed mechanism's effectiveness. The comparison between GP-MOPD and UP-MOPD changes both the projection location and the optimizer-state evolution: GP-MOPD updates AdamW using projected gradients, whereas UP-MOPD retains the moments computed from the original mixed gradient. Without an ablation isolating these factors, it is difficult to attribute the observed improvements specifically to update-space projection. Comparisons with additional multi-objective optimization baselines would further strengthen the methodological contribution.

Questions for authors

See Weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 3Vzz

Thank you for your careful review and constructive suggestions.

W1&2. Computational overhead and scalability. Actor training time includes the student model’s forward and backward passes, domain-gradient processing, and optimizer updates. To provide a clearer account of the actual training cost, we have added a runtime breakdown figure to the main text. Although actor training takes 3.43 times as long as the baseline, the end-to-end time per step, including rollouts, is 1.42 times that of the baseline (66.55 → 94.76 seconds). The additional computation comes from the domain-gradient computations and update corrections required for parameter projection. In exploring scalability, we focus on rollout generation as a key bottleneck and are actively investigating more efficient rollout generation and larger-scale training.

W3. Mechanism attribution and baselines. Thank you for raising this question. GP-MOPD is a practical gradient-intervention baseline inspired by PCGrad. The direct ablation baseline for UP-MOPD is M-OPD: both use the same distillation objective, mixed-gradient computation, and AdamW state-update rule, with UP-MOPD adding only the projection of the candidate parameter displacement. Thus, the comparison between M-OPD and UP-MOPD (59.13 versus 60.03 in Table 3) evaluates the effect of adding projection, whereas the comparison between GP-MOPD and UP-MOPD evaluates two complete intervention procedures. We will clarify the roles of these comparisons in the revision. Table 3 also includes gradient clipping, domain reweighting, and Update Rejection as controls.

Q1&Q2. Thank you for these further questions. The diff-localized DPO, text-correction projector, and edit horizon mentioned in the questions are not part of our setting. To clarify the meaning of “projection” in our work, UP-MOPD constructs constraints from domain gradients and AdamW’s candidate parameter displacement, then corrects the update through low-dimensional numerical optimization. It therefore requires no additional generative projector, API calls, or gold answers for projection. The relevant theoretical properties are presented in Proposition 2 and Corollary 1, and the computational cost is further detailed in the newly added runtime analysis.

Thank you for the authors' detailed explanations and clarifications, which have largely addressed my concerns.

I would also like to apologize for an oversight in my original review. The Questions For Authors section mistakenly included questions intended for another submission. The appropriate questions for this paper should have been largely aligned with the concerns raised in the Critical Weaknesses section. I sincerely apologize for this error and any confusion it may have caused.

Considering the authors' responses and the overall technical merits of the work, I have decided to maintain my original rating of 4: Clear acceptance.

Thank you very much for your positive evaluation of our work and for taking the time to review our manuscript and response.

Reviewer_j8shOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper considers the effect of AdamW on MOPD. The basic observation is that per domain gradients, gig_i, individually produce improvements in domain performance, but when mixed and transformed through AdamW, the resulting model update does not strictly improve performance in all domains and can, in fact, degrade some domain's performance. The authors demonstrate the basic mechanism in proposition 1. Their solution to this issue is to consider half-spaces induced from the various gig_i and project the AdamW update onto these half-spaces to ensure feasibility. The authors then proceed to test the method on various benchmarks (including a medical benchmark); the results show average improvement over alternate methods such as Open MOPD. The authors also verify assumptions of the method such as gradient alignment and update conflict. Overall, a nicely motivated method with a clear solution and good results.

Critical strengths

The paper has the following strengths:

  • The motivation for the paper is clear -- after mixing and transforming domain gradients with AdamW, the combined model update could potentially degrade domain performance (related to observations made in some prior works on conflict in multi-task optimization).
  • The paper shows that modifications can be made after the AdamW step by projecting on half-spaces induced by the domain gradients to obtain a feasible upate
  • Experiments show the method works and provides gains over standard methods
  • The paper refers to prior work on continual learning and multi-task learning -- both of these areas have related, but different methods, than the current work. Overall, the paper is clear, well-motivated, has good derivations, and results demonstrate the viability of the approach.
Critical weaknesses

Overall, the paper has no critical weaknesses.

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Response to Reviewer j8sh

Thank you for carefully reading our paper and for your positive assessment of its motivation, methodological formulation, and experimental results. We greatly appreciate your recognition.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Area Chair and Reviewers

Dear Area Chair and Reviewers:

Thank you for your careful reviews and constructive discussion. All three reviewers gave ratings in the acceptance range, with j8sh and 3Vzz awarding the highest rating of 4 (Clear acceptance). Reviewer j8sh highlighted the motivation, derivations, and empirical results. Reviewer 3Vzz recognized the rationale for intervening at the optimizer-update level and the practical value of the low-dimensional formulation. Reviewer xaoi gave a rating of 3 (Weak acceptance) and responded positively to our additional experiments and clarifications in their follow-up.

UP-MOPD’s key innovation is to constrain the parameter update produced by AdamW directly. AdamW can turn a mixed gradient that satisfies the domain-wise first-order constraints into an update that violates them. UP-MOPD applies the smallest Euclidean correction needed to satisfy those constraints. In the ablation, adding projection alone raised the medical–general aggregate score from 59.13 to 60.03. All three independent runs on the public benchmark also exceeded Open-MOPD’s reported score.

By checking and constraining interference at each parameter update, UP-MOPD provides a reusable optimization principle for combining the expertise of multiple teachers in a single foundation model. We have open-sourced our implementation so the community can reproduce the results and extend the method to more domains and larger training settings. We believe this work will make a substantial contribution to the community’s efforts to build more versatile foundation models. Thank you again for your careful reviews and constructive discussion!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.