acceptodds
Under review as a conference paper at ICLR 2027

Merge by Teaching: Consolidating a LoRA Library by Distillation

Abstract

Fine-tuned adapters accumulate. A team that has collected a dozen LoRA modules, one per behavior, trained by different people at different times, eventually wants them in a single model, and the field’s default answer is to add their weight deltas together. We show that this answer quietly stops working. As a library grows, summed adapters do not merely blur. At thirty-two skills the merged model scores zero on every one of them, and on a knowledge benchmark none of the adapters ever touched it answers no better than a coin, because the deltas disagree in sign about as often as coin flips. Yet nothing is actually lost. Recent post-training work has established the remedy in a different setting — let each specialist teach the student on the student’s own outputs rather than donate its weights — and we ask what happens when that operator is taken out of the lab that trained the specialists and into a library someone inherited. Three things change, and each is a result. Nobody knows which adapter owns which prompt, so the router has to be learned. The library is far larger than the language-model studies of this operator, which is exactly the range where weight arithmetic dies. And the base model’s own abilities, which that work protects but never measures, turn out to be collateral damage for weight arithmetic and imitation alike. Teaching leaves them standing, and its one loss has a nameable cause. Where the sum scores zero on all thirty-two skills, the student keeps 92% of what the teachers reach alone. We call the recipe Merge by Teaching, and we report where it holds, over three model families, five sizes, and adapters we did not train, along with the one failure mode we can characterize but have not yet learned to prevent.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, LoRA, adapter consolidation, model merging, on-policy distillation, multi-teacher distillation, parameter-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_GLV6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper shows that merging many LoRA adapters leads to significant performance degradation on many expert contract tasks (32 tasks to be exact). To solve this, they make use of multi-teacher on-policy distillation. They show that this can retain performance across the 32 tasks.

Critical strengths
  1. The paper is well thought through - it seems well put together and sound for the most part.

  2. Scaling up the number of experts is good. These many contract tasks can be of utility to the community as long as the authors open source them after acceptance - the data to train them as well as the expert adapters themselves. And showing that performance is retained with 32 experts is nice as well.

Critical weaknesses
  1. I can't help but think this work is slightly incremental. Both MOPD and CollectionLoRA (50 experts) follow the same approach but the former is focused on non-PEFT language setting while the latter is focused on vision + PEFT. So this papers core contribution is something similar in the language + PEFT setting. While this paper seems generally sound to me, I'm unsure if the contribution is significant enough to warrant acceptance. I am curious to hear what the authors have to say about this and am open to discussion. Is routing their primary novelty - on the surface it seems like engineering? Or is it the 32 experts in language?

  2. The lack of task categorization is slightly hurtful to the papers claims. 32 tasks seems like an arbitrary number and its unclear how well these tasks transfer from one to another. Perhaps a useful addition to this work would be some more detail on this but not a big problem.

  3. Smaller models are a problem. I'd like to understand if merging continues to cause significant loss of ability at the 12B+ scale, on the tasks the authors train and evaluate. I cannot deduct points for this because of compute constraints but would urge the authors to include such results, if feasible.

  4. How much of the 0% failure after merging comes from lack of ability to follow output format instructions? It seems unclear in the paper and perhaps the author could simply add some more details in the appendix.

  5. The paper motivates its problem statement by bringing up inherited LoRA libraries but train their own for the most part across most important experiments. I would tone down the inherited LoRA libraries claim.

My primary concern still is the lack of novelty or insights in this work. Degradation with merging is known. Multi-teach on/off policy distillation has been attempted, with success. I'm not entirely sure I've learned something very new or fascinating but I might have missed something. Curious to hear the authors‘ thoughts.

Questions for authors

I've included the relevant questions within the weakness section.

Reviewer_nY5xOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the problem of consolidating multiple LoRAs into a single model. The authors demonstrate that existing methods for merging LoRAs, such as task arithmetic and TIES, struggle as the number of adapters increases. Merging 32 LoRAs with task arithmetic in particular causes significant degradation in the model's general capabilities as well as failures on task benchmarks. To rectify this, they propose merging the LoRAs by using on-policy distillation to train a student adapter from using the trained teachers as supervision. They additionally include a learned prompt-based router for teacher selection, an anchor to preserve base model capabilities, and a method for adding future adapters on top of the trained student. The authors show that their method can retain much of the individual teachers' performance while also preserving base model capabilities better than baselines.

Critical strengths

The main strength of the paper is the initial experiment, which shows that simply merging all adapters drops the performance of the base model and does not work, as well as the additional analysis to explain this phenomenon.

Critical weaknesses

First, I found the AI generated text in the paper to be difficult to read and confusing.

My main concern with the paper, however, is that the method seems to not be particularly novel. The method is very similar to MOPD, which they cite, and the difference is not substantial enough to warrant acceptance. In addition to a lack of novelty, it appears that the proposed method is very computationally expensive compared to competitive baselines.

Finally, a review of the supplementary code leaves some questions regarding the methodology for the baselines. It seems that the K=32 task arithmetic baseline uses an unnormalized sum, which may impact results. It also appears that the "best weight curve" K=32 point does not have an associated coefficient sweep.

Questions for authors

From the supplementary code, it does not appear that there was a full sweep for K=32. Was that conducted? If so, what were the results?

From the supplementary code, it appears that scale for task arithmetic was not set and therefore defaults to 1. Is that correct? Did you sweep other values for scale? This may severely effect the initial results.

Reviewer_c597OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper considers the problem of merging many (specifically, thirty two) LoRA adapters into a single model without conflict. They show that a naive approach of adding together the LoRA adapters doesn't work as the number of LoRA modules gets too large. Instead, they show that distilling the adaptors into the weights preserves performance.

Critical strengths
  • Their method beats somewhat reasonable baselines, and generalizes to new adapters.
Critical weaknesses
  • The paper is incredibly hard to read, Although the AI Use Statement doesn't talk about whether AI was used to write the paper, the abstract alone (if not just its first sentence) suggests that there was little human involvement in the writing.
  • Important acronyms, such as knowledge distillation (KD), are never defined. Likewise, important baselines (offline KD) are never explained in the paper.
  • It's unclear what problem the paper is trying to solve. The point of LoRA adapters is to edit model weights in a lightweight and non-permanent manner (so they could easily be applied / taken off for different inputs). Distilling these edits into the model weights seems to defeat the point of using LoRA to begin with.
  • The paper doesn't compare reasonable baselines, such as training a stronger router without performing knowledge distillation. However, it could be the case that the paper does include such baselines, but they weren't reported clearly enough for me to identify them.
Questions for authors
  • Training a probe to predict routing (if I understand correctly) is a neat approach! Is there a token position / layer where probing was more effective, and how did it compare to full fine-tuning (or training a LoRA adapter for routing)?
Reviewer_HcygOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes a method for combining many LoRA adapters (not new as such, but new for LLM LoRA adapters), which addresses three practical challenges. (i) Methods based on weight arithmetic fail when number of adapters is high. They address this by using (on-policy) distillation from adapters (teachers) to a combined student. (ii) Combining many adapters may hurt the base model performance. They address this by adding base model as one of the teachers. (iii) LoRA adapters obtained from others may not come with metadata. They propose training routers to map each prompt to an expert adapter. Within their custom setup, they show that their method outperforms weight arithmetic based methods, and also off-policy distillation in some cases. They also suggest a method of continually adding more adapters.

Critical strengths

The suggested limitation of weight space arithmetic for merging numerous LoRA adapters is genuine, and the method used in the paper is more sound, as evidenced by their experiments. Notwithstanding some issues with their task setup, the paper does thorough empirical evaluations of different methods, and almost thorough baseline comparisons to substantiate their claims.

Adding base model as a teacher (akin to replay in continual learning) is a useful idea, which addresses an important pitfall of adding many adapters. To my knowledge, this is new. This utility of this idea extends beyond the specific method (on-policy distillation) proposed here.

Critical weaknesses

The authors recognize that their suite of tasks are already solved by instruct models without LoRA adapters -- so they decide to make the task harder by removing instructions from the prompts, and demanding a specific format for answers. This adds confounders to their evaluations, since failure on the task can simply be a failure of identification or failure of answer formatting, rather than a genuine failure of capability. Such a task setup is of limited practical use, and biased towards methods from the paper. This makes the overall claims of the paper less convincing.

The primary method of the paper is on-policy distillation, which is authors claim, works substantially better than other methods including off-policy distillation. However the evidence for the benefit over off-policy distillation is mixed, as pointed out in the paper. An important baseline comparison that is missing from the paper is adding the base-model as a teacher in off-policy distillation (and perhaps other methods too).

Many assertions made in the paper are surface-level with limited justification. The idea that weight arithmetic fails due to opposite signs is an example of this. There are many possible explanations for the failure, and the paper does not provide enough evidence for theirs. As a naive example, it could be that many additions take the model too far from the base. (Also note that a random walk with + and - entries has a norm N\sqrt{N} not 0.) This limitation can be addressed with further baseline experiments.

Questions for authors
  • How important is on-policy distillation vs just the replay of base model (anochoring)? Is there experimental results isolating the effects of the two?

  • How much do the results depend on the lack of instructions in the prompts. Is it possible to address this confounder by making stronger tasks that base models cannot perfectly solve even with instructions?

  • Can you be more precise about how sign disagreements lead to merging failure (as opposed to other mechanisms of degradation)?

  • How important is initializing on-policy distillation from TIES merge. Can you instead start from the base model?

  • Suggestion: The authors should clarify/emphasize the novelty of their methods compared to existing works such as CollectionLoRA and SAMerging, which explore similar ideas.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.