- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the problem of merging multiple language models fine-tuned on different tasks into a single model that preserves their capabilities, without additional training or access to task data. The authors propose Distill-Pro, a training-free model-merging method that applies orthogonal projections to task-specific parameter updates to reduce interference between tasks. The method is evaluated on two multi-teacher benchmarks using SmolLM3-3B and Qwen3-4B, covering mathematical reasoning, code generation, and instruction following.
Critical strengths
-
Simple and efficient approach.
-
The authors evaluate Distill-Pro against several model merging methods.
Critical weaknesses
-
The method assumes that components shared between task-specific parameter updates may cause interference. However, overlapping directions can also represent useful shared knowledge, particularly for closely related tasks. The paper does not sufficiently justify this assumption or investigate when projection helps versus harms performance.
-
Limited improvements.
Questions for authors
-
See Weaknesses.
-
The experiments consider at most three teachers. Could the authors extend the evaluation to a larger number of tasks, such as 5 or 7, to investigate whether Distill-Pro remains effective as the number of merged models increases? This is particularly important because repeated projections may progressively remove useful parameter-update directions.
2 comments
Thank you for the response. However, my main concern regarding the projection mechanism remains unresolved. Overlapping directions between task-specific updates do not necessarily represent harmful interference; they may also encode useful shared knowledge. Regarding scalability, I specifically requested experiments with a larger number of teachers (e.g., five or seven). The dimensionality argument and update-norm statistics are useful but not enough. In particular, preserving most of the update norm does not necessarily imply preserving task-relevant information. Additionally, no revised manuscript has been uploaded to incorporate the new analyses and clarifications. Overall, the rebuttal does not resolve my main concerns, and I therefore maintain my original score.
Response to Reviewer Vpmn
Overall response
We thank the reviewer for recognizing the simplicity and efficiency of Distill-Pro and our comparisons with existing model-merging methods. Below, we clarify the scope of our projection formulation, discuss the performance and computational advantages of our method, and present a dimensionality analysis and empirical update-norm statistics to address the concern about repeated projections.
W1 & Q1: Insufficient justification for removing overlapping update directions
Thank you for the suggestion. Our current formulation considers only the preservation of performance on previous tasks. We leave projection strategies that distinguish between overlapping update directions with the same and opposite orientations to future work.
W2 & Q1: Limited performance improvements
Our method consistently achieves strong performance across all five experimental settings, outperforming or matching state-of-the-art model-merging and MOPD baselines. In contrast, existing model-merging methods exhibit inconsistent performance. Although MOPD methods outperform our method in some cases, they require substantial amounts of data and computational resources for training.
Q2: Scalability to more teachers and tasks
Repeated projections do not overly constrain the space of parameter updates. Consider a linear layer with weights of shape (d,d). Both ΔW1 and ΔW1→2 are elements of Rd×d. The projection in Equation 6 maps ΔW1→2 onto the subspace orthogonal to ΔW1. Since the subspace spanned by ΔW1 in Rd×d is one-dimensional, its orthogonal complement has dimension d2−1. Even in the row-wise fine-grained variant, the orthogonal complement for each row has dimension d−1. Thus, this projection places only a small restriction on ΔW1→2. Even repeated projections do not excessively reduce its degrees of freedom.
Current multi-teacher distillation studies generally consider only two or three tasks, so we retain the existing experimental settings. To empirically demonstrate that projection has only a small effect on ΔW1→2, we report the ratio between the norms of ΔW1→2,proj and ΔW1→2 during the actual rectification process. For the three-task setting, Task Combination 4 in the Open-MOPD benchmark, we summarize the distributions of all ∥ΔW1→2∥∥ΔW1→2,proj∥ values during the first and second projections of Distill-Pro-FG-BR. We also summarize the distribution of this ratio during the projection of Distill-Pro-FG-BR on Task Combination 5 in the G-OPD benchmark.
The results show that projection changes the norm of ΔW1→2 only slightly in most layers.