acceptodds
Under review as a conference paper at ICLR 2027

Training-free Multi-teacher Distillation as a Continual Learning Problem

Abstract

Multi-teacher distillation (MTD) has achieved notable success in integrating diverse capabilities into a single model; however, existing approaches suffer from immense data and computational costs. While model merging offers a training-free alternative, existing techniques largely rely on empirical heuristics and lack a rigorous theoretical foundation. In this paper, we novelly formulate MTD as a continual learning problem. Specifically, we conceptualize independent teachers fine-tuned from a shared base model as having been trained sequentially, where acquiring a "new" task induces catastrophic forgetting on "previous" ones. Consequently, we propose **Distill**-by-**Pro**jection (Distill-Pro), a post-hoc parameter rectification technique that projects and corrects model updates to eliminate components that interfere with previous task performance. Crucially, Distill-Pro remains entirely data-free and training-free, while being grounded in a theoretical foundation of first-order performance invariance. Extensive evaluations on two large language model multi-teacher distillation benchmarks—encompassing two model architectures and three diverse tasks—demonstrate that our method consistently outperforms or matches state-of-the-art model merging and on-policy distillation baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-teacher distillation, Catastrophic forgetting, Model merging

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VpmnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of merging multiple language models fine-tuned on different tasks into a single model that preserves their capabilities, without additional training or access to task data. The authors propose Distill-Pro, a training-free model-merging method that applies orthogonal projections to task-specific parameter updates to reduce interference between tasks. The method is evaluated on two multi-teacher benchmarks using SmolLM3-3B and Qwen3-4B, covering mathematical reasoning, code generation, and instruction following.

Critical strengths
  1. Simple and efficient approach.

  2. The authors evaluate Distill-Pro against several model merging methods.

Critical weaknesses
  1. The method assumes that components shared between task-specific parameter updates may cause interference. However, overlapping directions can also represent useful shared knowledge, particularly for closely related tasks. The paper does not sufficiently justify this assumption or investigate when projection helps versus harms performance.

  2. Limited improvements.

Questions for authors
  1. See Weaknesses.

  2. The experiments consider at most three teachers. Could the authors extend the evaluation to a larger number of tasks, such as 5 or 7, to investigate whether Distill-Pro remains effective as the number of merged models increases? This is particularly important because repeated projections may progressively remove useful parameter-update directions.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Vpmn

Overall response

We thank the reviewer for recognizing the simplicity and efficiency of Distill-Pro and our comparisons with existing model-merging methods. Below, we clarify the scope of our projection formulation, discuss the performance and computational advantages of our method, and present a dimensionality analysis and empirical update-norm statistics to address the concern about repeated projections.

W1 & Q1: Insufficient justification for removing overlapping update directions

Thank you for the suggestion. Our current formulation considers only the preservation of performance on previous tasks. We leave projection strategies that distinguish between overlapping update directions with the same and opposite orientations to future work.

W2 & Q1: Limited performance improvements

Our method consistently achieves strong performance across all five experimental settings, outperforming or matching state-of-the-art model-merging and MOPD baselines. In contrast, existing model-merging methods exhibit inconsistent performance. Although MOPD methods outperform our method in some cases, they require substantial amounts of data and computational resources for training.

Q2: Scalability to more teachers and tasks

Repeated projections do not overly constrain the space of parameter updates. Consider a linear layer with weights of shape (d,d)(d,d). Both ΔW1\Delta W_1 and ΔW1→2\Delta W_{1\to2} are elements of Rd×d\mathbb{R}^{d\times d}. The projection in Equation 6 maps ΔW1→2\Delta W_{1\to2} onto the subspace orthogonal to ΔW1\Delta W_1. Since the subspace spanned by ΔW1\Delta W_1 in Rd×d\mathbb{R}^{d\times d} is one-dimensional, its orthogonal complement has dimension d2−1d^2-1. Even in the row-wise fine-grained variant, the orthogonal complement for each row has dimension d−1d-1. Thus, this projection places only a small restriction on ΔW1→2\Delta W_{1\to2}. Even repeated projections do not excessively reduce its degrees of freedom.

Current multi-teacher distillation studies generally consider only two or three tasks, so we retain the existing experimental settings. To empirically demonstrate that projection has only a small effect on ΔW1→2\Delta W_{1\to2}, we report the ratio between the norms of ΔW1→2,proj\Delta W_{1\to2,\text{proj}} and ΔW1→2\Delta W_{1\to2} during the actual rectification process. For the three-task setting, Task Combination 4 in the Open-MOPD benchmark, we summarize the distributions of all ∥ΔW1→2,proj∥∥ΔW1→2∥\frac{\Vert\Delta W_{1\to2,\text{proj}}\Vert}{\Vert\Delta W_{1\to2}\Vert} values during the first and second projections of Distill-Pro-FG-BR. We also summarize the distribution of this ratio during the projection of Distill-Pro-FG-BR on Task Combination 5 in the G-OPD benchmark.

∥ΔW1→2,proj∥∥ΔW1→2∥\frac{\Vert\Delta W_{1\to2,\text{proj}}\Vert}{\Vert\Delta W_{1\to2}\Vert}25th percentile50th percentile75th percentile
TC 4: first projection0.9995240.9996580.999712
TC 4: second projection0.9965150.9981550.998975
TC 50.9929160.9973900.998882

The results show that projection changes the norm of ΔW1→2\Delta W_{1\to2} only slightly in most layers.

Thank you for the response. However, my main concern regarding the projection mechanism remains unresolved. Overlapping directions between task-specific updates do not necessarily represent harmful interference; they may also encode useful shared knowledge. Regarding scalability, I specifically requested experiments with a larger number of teachers (e.g., five or seven). The dimensionality argument and update-norm statistics are useful but not enough. In particular, preserving most of the update norm does not necessarily imply preserving task-relevant information. Additionally, no revised manuscript has been uploaded to incorporate the new analyses and clarifications. Overall, the rebuttal does not resolve my main concerns, and I therefore maintain my original score.

Reviewer_aDH6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how to combine several fine-tuned expert LLMs that are trained from the same base model, into one model without any training. The authors reframe the problem as continual learning by treating the independently trained teachers as trained sequentially. They proposed Distill-Pro, which removes the part of each new expert's weight change that points along the previous expert's weight change. A first-order argument says this should keep the earlier task's performance intact. The previous expert's weight change serves as a cheap, data-free stand-in for its gradient. The authors also propose a finer-grained variant that projects each row of a weight matrix separately, and a "base-referenced" variant that they show is closely related to, and in the two-task case algebraically identical to, standard Task Arithmetic plus the projection. They evaluate on two recent multi-teacher distillation benchmarks and report that Distill-Pro matches or beats both training-free merging baselines and training-based on-policy distillation methods.

Critical strengths
  1. Training-free distillation is an important and interesting direction. The proposed method is simple yet effective.
  2. It shows that training-free merging can match or beat multi-teacher on-policy distillation.
Critical weaknesses
  1. Some theoretical analysis are not solid enough. For example, FG does not has enough theoretical justification. FG imposes d_out row-wise constraints whose benefit is not clear to me. Besides, the authors do not justify rows over columns or blocks

  2. The evaluation results are confusing

a. Eq.10 proves that Distill-Pro and Distill-Pro-BR give the same merged model for two tasks and so do Distill-Pro-FG and FG-BR. However, in the evaluation, they can differ a lot in performance. It appears that numerical differences or other factors could matter in the performance but the authors do not provide adequate analysis for this.

b. From Sec 4.2, Distill-Pro-FG is proposed to better handle 2-D weight matrices, which implies that it should be a better fit for LLM. However, in practice, Distill-Pro-FG does not always outperform Distill-Pro (e.g. Task Combination 1) and Distill-Pro-FG-BR does not always outperform Distill-Pro-BR (e.g. Task Combination 2)

Questions for authors
  1. By Eq. 10, Distill-Pro = Distill-Pro-BR and FG = FG-BR. Why are they evaluated separately and shows different results?
  2. For designing FR, what justifies d_out row-wise constraints? Why rows rather than columns?
  3. The evaluation results show different results for 4 Distill-Pro variants. Given a model, how to choose which variant in practice?
3 comments
AuthorsOpenReview ↗

Response to Reviewer aDH6

Overall response

We thank the reviewer for recognizing the importance of training-free distillation, the simplicity and effectiveness of our method, and its performance relative to multi-teacher on-policy distillation. Below, we clarify the motivation for row-wise projection, discuss numerical precision and evaluation randomness in the comparison between equivalent formulations, explain the trade-offs of fine-grained projection, and provide practical guidance for selecting a variant.

W1 & Q2: Insufficient justification for row-wise fine-grained projection

Since <A,B>F=Tr(A⊤B)=∑aijbij=∑i∑jaijbij=∑j∑iaijbij,\left<A,B\right>_F =\text{Tr}(A^\top B)=\sum a_{ij}b_{ij}=\sum_i\sum_j a_{ij}b_{ij} =\sum_j\sum_i a_{ij}b_{ij}, there is no fundamental distinction between row-wise and column-wise formulations. We adopt the row-wise formulation because it follows the convention in continual learning, where the orthogonality constraint usually takes the form ΔWM=0\Delta W M=0, requiring each row of ΔW\Delta W to be orthogonal to the column space of a memory matrix MM.

W2(a) & Q1: Discrepancy between theoretical equivalence and experimental results

We acknowledge that the results as currently presented can be confusing, and thank the reviewer for pointing this out. Due to finite numerical precision, the model weights produced by Distill-Pro and Distill-Pro-BR are not exactly identical in practice. Evaluation randomness also contributes to differences in the results. To address the latter factor, we conducted evaluations with multiple seeds to obtain more statistically meaningful results; please see our response to W4 & Q6 from Reviewer NG4t. After averaging across seeds, the differences between Distill-Pro and Distill-Pro-BR are reduced.

W2(b): Inconsistent empirical benefits of fine-grained projection

Row-wise orthogonality (∑jaijbij=0,∀i\sum_j a_{ij}b_{ij}=0,\forall i) is a sufficient condition for full-matrix orthogonality (∑ijaijbij=0\sum_{ij} a_{ij}b_{ij}=0) and imposes stronger constraints on parameter updates. On the one hand, it can prevent very small projection corrections from being lost due to finite numerical precision, potentially better preserving performance on previous tasks. On the other hand, it may cause a larger loss in performance on new tasks. Therefore, neither the FG nor the non-FG version is necessarily superior. Since model integration is extremely inexpensive, the choice between these two versions can be based on evaluation results.

Q3: Practical guidance for choosing among the four variants

Our method has a very low computational cost: it produces the merged weights in just 10 minutes on a single NVIDIA H20 GPU, with a peak GPU memory usage of only 8 GB. Evaluation is also substantially less expensive than MOPD training. For example, using two H20 GPUs, evaluating the math tasks in the Open-MOPD benchmark takes only 3 hours, the code tasks take 2 hours, and the instruction-following tasks take just 15 minutes. We therefore consider it reasonable and practical to try all variants and select the best one based on evaluation results.

Thanks for your response! However, several concerns still remains.

  1. from your table above, BR and non-BR version still differ a lot (e.g. Distill-Pro-FG and Distill-Pro-FG-BR in TC3 and TC4). This phenomenon still lacks a clear explanation.

  2. Choosing the best variant on the evaluation benchmarks is currently done on the test set. But it should be done in the validation set and then report the best variant on the test set, just like the training-based method for tuning their training hyper-parameters.

Thank you for your follow-up and for highlighting these remaining concerns!

Regarding the differences between BR and non-BR formulations.

In practice, finite numerical precision introduces differences in the resulting model weights because the two formulations involve different sequences of arithmetic operations. When these differences become substantial in a particular setting—for example, when one formulation provides greater numerical accuracy than the other—it is reasonable to consider them as distinct implementations and evaluate their performance separately, despite their algebraic equivalence.

From a practical perspective, we view this flexibility as an advantage: our method offers alternative implementations at very low computational cost, providing a broader set of choices for model integration. Their mathematical equivalence therefore does not eliminate the practical value of comparing them under finite-precision computation.

Regarding variant comparison versus hyperparameter tuning.

We would like to clarify that the variants are explicitly defined algorithmic formulations, rather than different values of a tunable hyperparameter. We evaluate and report each formulation separately, including those that do not achieve the best performance. In this respect, the comparison is analogous to evaluating several algorithms on the same benchmarks: identifying which algorithm achieves the highest score does not, by itself, constitute hyperparameter tuning on the test set.

Our intention is therefore to present the performance of each method and compare their respective strengths, rather than to introduce a single adaptive method that selects its configuration using test results. A validation set would be necessary if the goal were to construct such a selection procedure and report its selected model’s test performance. However, that is a different claim from reporting the individual benchmark results of several explicitly defined methods.

Reviewer_NG4tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper reformulates multi-teacher distillation (MTD) as a continual-learning problem: independently fine-tuned teachers are viewed as if they had been trained sequentially, so integrating a new teacher’s update can “forget” earlier teachers. Distill-Pro rectifies each new parameter update by projecting it away from the previous task update, using \Delta W_i as a proxy for the gradient direction of task i. The method is data-free and training-free, requiring only parameter arithmetic. Experiments on Open-MOPD and G-OPD benchmarks compare Distill-Pro against model-merging and MOPD baselines.

Critical strengths

The core operation is lightweight. This requires no data, no gradients, and no student training. For practitioners who already have several domain experts fine-tuned from a shared base, this is an appealing alternative to MOPD.

Interpreting independently fine-tuned teachers as a conceptual sequence \mathcal{T}_1\to\mathcal{T}_2\to\cdots\to\mathcal{T}_T gives a clean narrative for why naive task-vector summation suffers from interference.

The row-wise variant is a natural way to avoid discarding too much of the new update. Since row-wise orthogonality implies full-matrix orthogonality, so the refinement is mathematically consistent.

The paper evaluates on two MTD benchmarks, multiple task combinations, and compares against AvgMerge, ExPO, TA, DARE, Naive MOPD, ExOPD, and Open-MOPD. On Open-MOPD Task Combination 4, Distill-Pro-FG reaches Total 32.69 and Rec. Rate 113.12%, outperforming Open-MOPD’s 31.09. The projection ablation in Figure 3 is also useful: removing projection collapses to the last teacher, while projection recovers much of the earlier teacher’s performance.

Because the method is framed sequentially, checking task order is important. Table 3 reports only a 0.46-point spread for vanilla Distill-Pro across three orders on Math+Code+IF, which supports the practical stability of the approach.

Critical weaknesses

The theoretical guarantee is weaker than the paper suggests. Setting the first-order term to zero does not guarantee performance invariance for large fine-tuning updates. More importantly, the method uses \Delta W_1 as a proxy for G_1. The paper cites optimization literature for why \Delta W approximates a gradient direction, but it does not empirically verify this proxy. The phrase “first-order performance invariance” should be qualified.

The multi-task recursive projection is not clearly guaranteed for all prior tasks. The recursive scheme produces W_{1:T}=W_1+\Delta W_{1\to(2:T)}^{\mathrm{proj}}, where the final update is projected against \Delta W_1. The earlier step projects \Delta W_{T-1\to T} against \Delta W_{T-1}, and so on. But it is not obvious that the final update satisfies \langle \Delta W_i,\Delta W_{1:T}-\Delta W_i\rangle\approx 0 for every earlier task i. The paper should clarify whether the procedure enforces orthogonality to all prior task updates or only to the immediately preceding one at each recursion.

On Open-MOPD, Naive MOPD and Open-MOPD appear only in Task Combination 4; Task Combinations 1–3 compare Distill-Pro mainly against model merging. Therefore, the claim that Distill-Pro “consistently outperforms” MOPD is not supported across all Open-MOPD combinations. On G-OPD, Distill-Pro-FG-BR reaches Total 52.64, while Naive MOPD reaches 53.81 and ExOPD reaches 53.88. The conclusion should explicitly say that Distill-Pro is competitive but not uniformly better on G-OPD.

The tables report single numbers, with no seeds, confidence intervals, or paired tests. Given the small margins in several places—e.g., 52.64vs 53.81 on G-OPD, and 0.46-point task-order spread—uncertainty estimates are necessary to judge whether differences are meaningful.

The paper reports Distill-Pro, Distill-Pro-BR, Distill-Pro-FG, and Distill-Pro-FGBR, but the main text does not clearly define what “BR” and “FG” mean. Since different variants win in different tables, readers cannot assess whether the best variant was selected post hoc. Please define each variant and state whether variant selection uses any benchmark data.

The paper introduces row-wise projection as “more fine-grained,” but does not report full-matrix projection as an ablation. It would be useful to know whether row-wise projection is actually better or simply different.

Distill-Pro does not use task data for training or projection. However, if the best variant or projection granularity is chosen by benchmark performance, then model selection is not entirely data-free. The paper should distinguish “training-free/data-free optimization” from “benchmark-tuned variant selection.”

Questions for authors

How reliable is \Delta W_i as a proxy for G_i?

Does the recursive multi-task projection enforce orthogonality to all prior tasks?

How does full-matrix projection compare with row-wise projection?

What exactly are Distill-Pro-BR, Distill-Pro-FG, and Distill-Pro-FGBR? Define the variants in the main text and state whether the best variant is selected using validation data, benchmark results, or fixed a priori.

Can you run MOPD baselines on all Open-MOPD task combinations? Currently, Naive MOPD and Open-MOPD appear only in Task Combination 4. A full comparison would substantiate or soften the “consistently outperforms MOPD” claim.

Can you report variance over seeds and statistical tests? Please add confidence intervals or paired tests for the main Total and Rec. Rate comparisons, especially for G-OPD where Distill-Pro is close to but below MOPD.

What is the actual wall-clock and memory cost? The paper says “a few minutes,” but does not quantify. Please report projection time and peak memory for Distill-Pro versus MOPD training under the same hardware.

How robust is the method to task order beyond one combination? Table 3 studies only Math+Code+IF. Please test task-order sensitivity for other combinations, especially the two-task settings and G-OPD.

What happens if teachers are not fine-tuned from the same base or have different architectures? The method assumes a shared base and same parameter shapes. Please state this as an explicit limitation and discuss whether projection is meaningful when teachers come from different bases.

3 comments
AuthorsOpenReview ↗

Response to Reviewer NG4t (1/3)

Overall response

We thank the reviewer for recognizing the lightweight, data-free and training-free nature of Distill-Pro, its continual-learning perspective, and the value of our empirical comparisons and ablations. Below, we address the gradient proxy and recursive projection, provide additional baseline comparisons, multi-seed evaluations, resource measurements, and task-order experiments, and clarify the variant definitions, selection procedure, and shared-base assumption.

W1 & Q1: Unverified gradient proxy

Using ΔW\Delta W as a proxy for GG has been widely adopted in the literature. For example, a series of studies on mitigating catastrophic forgetting [1–7] use ΔW\Delta W as a substitute for the gradient of a previous task. Moreover, as discussed in lines 191–197, this substitution has been verified both empirically [8–9] and theoretically [10] in prior work.

W2 & Q2: Unclear preservation of all prior tasks under recursive projection

In the multi-task setting, the final integration step combines an expert W1W_1 for task 1 with an expert W2:TW_{2:T} that performs well on tasks 2 through TT. Thus, ΔW1→(2:T)=W2:T−W1\Delta W_{1\to(2:T)}=W_{2:T}-W_1 enables the model to acquire the capabilities of tasks 2 through TT while forgetting task 1. At this stage, we therefore only need to project ΔW1→(2:T)\Delta W_{1\to(2:T)} to be orthogonal to ΔW1\Delta W_1. For each task i∈[2,T]i\in[2,T], its knowledge is already contained in ΔW1→(2:T)\Delta W_{1\to(2:T)}, rather than being preserved by enforcing orthogonality to ΔWi\Delta W_i.

W3 & Q5: Incomplete MOPD comparisons and overstated performance claims

We have added results for Naive MOPD and Open-MOPD on Task Combinations 1, 2, and 3. Please see our response to W4 & Q6.

The results show that our method outperforms the baselines on Task Combinations 2, 3, and 4, while achieving performance comparable to MOPD methods on Task Combinations 1 and 5. These findings are consistent with our statement:

... our method consistently outperforms or matches state-of-the-art model merging and on-policy distillation baselines. (line 24-26)

W5 & Q4: Unclear variant definitions and selection protocol

We introduce and define Distill-Pro-FG in Section 4.2 (line 235) and Distill-Pro-BR in Section 4.3 (line 251), respectively:

We term this variant as Fine-Grained Distill-Pro (Distill-Pro-FG).

We term this variant Base-Referenced Distill-Pro (Distill-Pro-BR).

The best variant can be selected based on evaluation results, as both weight merging and evaluation incur little computational cost.

W6 & Q3: Missing full-matrix versus row-wise projection ablation

We report the results of all Distill-Pro variants in every experiment. Variants with the “-FG” suffix use row-wise projection, whereas those without it use full-matrix projection.

W7 & Q4: Scope of the data-free claim under variant selection

By “data-free,” we mean that no training data are required. Access to evaluation data is naturally expected for evaluation.

References

[1] Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, Xuanjing Huang: Orthogonal Subspace Learning for Language Model Continual Learning. EMNLP (Findings) 2023: 10658-10671

[2] Haoyu Wang, Tianci Liu, Ruirui Li, Monica Xiao Cheng, Tuo Zhao, Jing Gao: RoseLoRA: Row and Column-wise Sparse Low-rank Adaptation of Pre-trained Language Model for Knowledge Editing and Fine-tuning. EMNLP 2024: 996-1008

[3] Shuo Yang, Kun-Peng Ning, Yu-Yang Liu, Jia-Yu Yao, Yong-Hong Tian, Yi-Bing Song, Li Yuan: Is Parameter Collision Hindering Continual Learning in LLMs? COLING 2025: 4243-4259

[4] Shristi Das Biswas, Yue Zhang, Anwesan Pal, Radhika Bhargava, Kaushik Roy: ELLA: Efficient Lifelong Learning for Adapters in Large Language Models. EACL (Volume 1: Long Papers) 2026: 1907-1924

[5] Alexandra Dragomir, Ioana Pintilie, Antonio Barbalau, Marius Dragoi, Florin Brad, Cristian Daniel Paduraru, Alexandru Tifrea, Elena Burceanu, Radu Tudor Ionescu: JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models. CoRR abs/2604.16171 (2026)

[6] Kefan Cao, Shuaicheng Wu: Orthogonal Low-rank Adaptation in Lie Groups for Continual Learning of Large Language Models. CoRR abs/2509.06100 (2025)

[7] Chenxu Wang, Yilin Lyu, Zicheng Sun, Liping Jing: Continual Gradient Low-Rank Projection Fine-Tuning for LLMs. ACL (1) 2025: 14815-14829

[8] Alex Nichol, Joshua Achiam, John Schulman: On First-Order Meta-Learning Algorithms. CoRR abs/1803.02999 (2018)

[9] Guy Gur-Ari, Daniel A. Roberts, Ethan Dyer: Gradient Descent Happens in a Tiny Subspace. CoRR abs/1812.04754 (2018)

[10] Luca Zhou, Daniele Solombrino, Donato Crisostomi, Maria Sofia Bucarelli, Giuseppe Alessio D'Inverno, Fabrizio Silvestri, Emanuele Rodolà: On Task Vectors and Gradients. CoRR abs/2508.16082 (2025)

AuthorsOpenReview ↗

Response to Reviewer NG4t (2/3)

W4 & Q6: Missing uncertainty estimates and statistical tests

We ran two additional seeds for Distill-Pro and all its variants, as well as for selected top-performing baselines within each task combination. The results show that our method outperforms the baselines on Task Combinations 2, 3, and 4, while achieving performance comparable to MOPD methods on Task Combinations 1 and 5.

TC 1: Math + Code

MethodsAIME24AIME25MathLCB v5LCB v6CodeTotalRecovery Rate
TA+DARE23.92±0.3824.29±0.4424.11±0.0421.98±0.4821.98±0.1221.98±0.2923.04±0.1599.58±2.24
Naive MOPD23.40±1.1724.29±0.3723.85±0.7622.89±0.4422.08±0.5722.49±0.1723.17±0.46101.45±7.00
Open-MOPD24.36±0.1825.38±0.4424.87±0.2023.61±1.1222.13±0.7522.87±0.9223.87±0.36112.25±5.52
Distill-Pro24.08±0.5724.27±0.7424.18±0.6422.32±0.2722.40±0.3622.36±0.2523.27±0.31103.00±4.77
Distill-Pro-BR24.08±0.2524.03±0.4424.06±0.3422.74±1.3721.96±0.3122.35±0.7723.20±0.48102.01±7.27
Distill-Pro-FG24.33±0.5924.13±0.2124.23±0.2222.55±0.5021.91±0.6922.23±0.4123.23±0.24102.44±3.61
Distill-Pro-FG-BR24.34±0.58‾\underline{24.34±0.58}24.58±0.98‾\underline{24.58±0.98}24.46±0.29‾\underline{24.46±0.29}23.10±0.70‾\underline{23.10±0.70}22.32±0.13‾\underline{22.32±0.13}22.71±0.41‾\underline{22.71±0.41}23.59±0.27‾\underline{23.59±0.27}107.91±4.07‾\underline{107.91±4.07}

TC 2: Math + IF

MethodsAIME24AIME25MathIFEvalIFBenchIFTotalRecovery Rate
TA23.82±0.8824.27±0.1924.04±0.3573.81±1.5823.55±1.0748.68±0.90‾\underline{48.68±0.90}36.36±0.4797.50±6.27
Naive MOPD24.25±1.0024.72±0.4824.49±0.3573.20±0.9822.56±0.5147.88±0.7236.18±0.4995.10±6.56
Open-MOPD22.93±0.3025.00±0.24‾\underline{25.00±0.24}23.97±0.2073.63±1.1122.67±0.6748.15±0.5136.06±0.3593.41±4.69
Distill-Pro24.18±0.77‾\underline{24.18±0.77}24.48±0.5024.33±0.2074.18±0.11‾\underline{74.18±0.11}22.33±1.2148.26±0.5636.29±0.2496.58±3.22
Distill-Pro-BR23.66±0.1625.14±0.3724.40±0.22‾\underline{24.40±0.22}74.12±1.4525.78±1.3549.95±1.0937.18±0.61108.37±8.20
Distill-Pro-FG23.75±0.9624.46±0.2924.11±0.5274.24±2.0423.11±0.5148.68±0.79‾\underline{48.68±0.79}36.39±0.3397.90±4.42
Distill-Pro-FG-BR24.08±0.6024.90±0.5824.49±0.5973.44±1.8523.44±1.26‾\underline{23.44±1.26}48.44±1.5436.47±0.63‾\underline{36.47±0.63}98.89±8.49‾\underline{98.89±8.49}

TC 3: Code + IF

MethodsLCB v5LCB v6CodeIFEvalIFBenchIFTotalRecovery Rate
TA23.63±0.7422.70±0.3523.17±0.5173.01±0.3723.78±0.1948.39±0.11‾\underline{48.39±0.11}35.78±0.30‾\underline{35.78±0.30}100.21±4.58‾\underline{100.21±4.58}
TA+DARE23.20±0.1322.82±0.4323.01±0.1671.84±0.7724.22±1.5048.03±1.0735.52±0.5296.18±8.03
Naive MOPD23.41±0.6322.53±0.1822.97±0.3974.06±1.11‾\underline{74.06±1.11}22.00±1.2048.03±0.8935.50±0.6295.87±9.65
Open-MOPD23.13±0.7222.53±0.4322.83±0.1674.31±0.9821.89±0.5148.10±0.7435.47±0.3095.33±4.71
Distill-Pro23.79±0.6623.07±0.23‾\underline{23.07±0.23}23.43±0.43‾\underline{23.43±0.43}71.28±0.8524.55±2.04‾\underline{24.55±2.04}47.92±0.6335.67±0.1798.54±2.60
Distill-Pro-BR23.83±0.53‾\underline{23.83±0.53}22.63±1.0023.23±0.2572.95±0.9123.67±1.1548.31±0.8735.77±0.49100.03±7.54
Distill-Pro-FG23.41±0.7122.82±0.2223.11±0.2572.09±0.8124.11±1.7148.10±0.6735.61±0.3697.51±5.58
Distill-Pro-FG-BR24.19±0.9423.31±0.5123.75±0.2773.45±0.7725.22±1.5749.34±1.1536.54±0.71112.03±10.94

TC 4: Math + Code + IF

MethodsAIME24AIME25MathLCB v5LCB v6CodeIFEvalIFBenchIFTotalRecovery Rate
TA23.80±0.4724.58±0.68‾\underline{24.58±0.68}24.19±0.17‾\underline{24.19±0.17}22.89±0.2822.36±0.2622.63±0.1373.07±0.2822.33±2.6047.70±1.2131.51±0.4395.77±6.27
Distill-Pro24.27±0.2424.34±0.0824.31±0.1623.57±0.1522.97±0.3123.27±0.2373.57±2.0024.33±1.76‾\underline{24.33±1.76}48.95±0.15‾\underline{48.95±0.15}32.18±0.07‾\underline{32.18±0.07}105.57±1.02‾\underline{105.57±1.02}
Distill-Pro-BR23.46±0.2824.32±0.6123.89±0.4423.99±0.7122.80±0.7723.40±0.30‾\underline{23.40±0.30}72.70±0.1124.33±1.77‾\underline{24.33±1.77}48.52±0.9031.93±0.51102.04±7.50
Distill-Pro-FG24.18±0.81‾\underline{24.18±0.81}24.78±0.3424.48±0.2524.19±0.6322.55±0.4823.37±0.2273.32±1.19‾\underline{73.32±1.19}26.44±1.3949.88±0.6032.58±0.14111.47±2.01
Distill-Pro-FG-BR23.73±0.3924.34±0.4924.04±0.0724.15±0.03‾\underline{24.15±0.03}22.88±0.41‾\underline{22.88±0.41}23.51±0.2272.02±0.7724.33±1.21‾\underline{24.33±1.21}48.18±0.3331.91±0.11101.69±1.65

(Our response to this question continues in the next comment.)

AuthorsOpenReview ↗

Response to Reviewer NG4t (3/3)

W4 & Q6 (continued)

TC 5: Math + Code

MethodsAIME24AIME25HMMT FebHMMT NovMathHumanEval+MBPP+LCBCodeTotalRecovery Rate
AvgMerge+ExPO54.72±0.4352.02±1.3928.61±0.9534.86±0.5742.55±0.4886.03±0.8470.62±1.2328.95±0.58‾\underline{28.95±0.58}61.87±0.8252.21±0.3994.64±2.07
TA+DARE57.15±0.7952.01±0.6329.27±0.1836.11±0.3343.64±0.1382.06±0.3270.74±0.39‾\underline{70.74±0.39}28.62±0.3060.48±0.2552.06±0.1893.82±0.97
Naive MOPD59.27±0.90‾\underline{59.27±0.90}55.83±0.8532.12±0.30‾\underline{32.12±0.30}37.01±0.84‾\underline{37.01±0.84}46.06±0.21‾\underline{46.06±0.21}84.50±1.1470.42±1.5428.29±1.3861.07±0.6553.56±0.28‾\underline{53.56±0.28}101.91±1.53‾\underline{101.91±1.53}
ExOPD61.35±0.1855.80±0.84‾\underline{55.80±0.84}32.19±0.8138.44±0.3746.94±0.4485.36±0.27‾\underline{85.36±0.27}69.91±0.6329.05±0.7961.44±0.11‾\underline{61.44±0.11}54.19±0.27105.29±1.44
Distill-Pro56.70±0.8951.22±0.2229.93±1.0135.55±0.6743.35±0.3082.62±0.7070.66±0.5528.71±0.1560.66±0.3852.01±0.3493.55±1.85
Distill-Pro-BR57.88±0.5851.22±0.7329.27±0.3136.46±0.3643.71±0.3082.42±2.1370.64±0.8928.52±0.1660.53±1.0652.12±0.4694.15±2.45
Distill-Pro-FG57.50±0.4251.63±1.1430.03±0.7936.39±0.9543.89±0.1382.77±1.1970.33±0.3928.67±0.4360.59±0.5252.24±0.3294.80±1.70
Distill-Pro-FG-BR57.01±0.8552.36±0.4231.01±0.4236.46±0.8544.21±0.4883.49±0.8971.05±0.4428.76±0.5061.10±0.0952.66±0.2897.04±1.51

Q7: Unquantified wall-clock time and memory cost

For Distill-Pro and its variants, we used a single NVIDIA H20 GPU. The process took approximately 10 minutes, with a peak GPU memory usage of 8 GB.

For Naive MOPD and Open-MOPD on Task Combinations 1, 2, and 3, we used 8 NVIDIA H20 GPUs and trained for 80 steps. Training with Naive MOPD took approximately 12 hours, with a peak GPU memory usage of 52 GB. Training with Open-MOPD took approximately 46 hours, with a peak GPU memory usage of 52 GB.

Q8: Limited task-order robustness evaluation

We have added experiments with the task order reversed for Task Combinations 1, 2, 3, and 5.

TC 1: code->math

MethodsAIME24AIME25MathLCB v5LCB v6CodeTotalRecovery Rate
Distill-Pro24.1123.8523.9823.0521.8322.4423.21102.14
Distill-Pro-BR24.7423.2323.9922.8721.0921.9822.9898.66
Distill-Pro-FG24.1724.4324.3023.1721.7122.4423.37104.58
Distill-Pro-FG-BR23.9124.4324.1723.5322.2322.8823.53106.95

TC 2: math->if

MethodsAIME24AIME25MathIFEvalIFBenchIFTotalRecovery Rate
Distill-Pro23.6524.6424.1575.2322.3348.7836.4698.83
Distill-Pro-BR23.2824.4323.8674.1224.3349.2336.5499.87
Distill-Pro-FG24.7924.3824.5974.1222.6748.4036.4999.20
Distill-Pro-FG-BR23.1324.5823.8676.5222.6749.6036.73102.34

TC 3: code->if

MethodsLCB v5LCB v6CodeIFEvalIFBenchIFTotalRecovery Rate
Distill-Pro22.8123.7723.2972.6422.6747.6635.4795.43
Distill-Pro-BR24.6722.8623.7774.6822.6748.6836.22107.02
Distill-Pro-FG23.4122.4622.9472.0924.0048.0535.4995.70
Distill-Pro-FG-BR22.9923.6023.3073.0124.0048.5135.90102.06

TC 5: math->code

MethodsAIME24AIME25HMMT FebHMMT NovMathHumanEval+MBPP+LCBCodeTotalRecovery Rate
Distill-Pro57.0852.2929.2736.1543.7079.8770.1128.1459.3751.5491.02
Distill-Pro-BR57.8151.5629.2735.1043.4481.8669.9728.2960.0451.7492.11
Distill-Pro-FG56.5651.7730.1036.2543.6781.8670.5728.8660.4352.0593.79
Distill-Pro-FG-BR57.9251.0429.5835.7343.5782.7769.9729.1460.6352.1094.04

The results are similar to those obtained with the task orders already reported in the paper, demonstrating robustness to task order.

Q9: Shared-base and architecture limitations

Multi-teacher distillation typically considers teachers with the same base model and architecture. The shared-base assumption is common in both model-merging and MOPD methods.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.