acceptodds
Under review as a conference paper at ICLR 2027

Rescuers Matter: Rethinking Conflict in LoRA Merging

Abstract

When merging LoRA adapters, a pair that violates a task-loss cap can become feasible after another adapter is added. We call this a rescue. We ask when rescuers can be abundant and how often they occur in trained adapter banks. An exact softmax construction realizes rescue with rank-one LoRA: opposing useful-token preferences can cancel in a pair and be restored by a third adapter. In a robust two-type regime, every opposite-type pair is rescued by every remaining adapter; under balanced iid types, the certified fraction of distinct rescued triples tends to three quarters. A rank-one quadratic construction also gives a sharp cardinality ratio for sound downward-closed policies. Exhaustive audits of six new CLIP adapter banks find rescuers for most bad pairs overall, with rescue in every bank even under stricter margins. Earlier complete-cube audits show that permanent exclusion can reduce merge accuracy. The theory concerns explicit constructions and sampling laws; the empirical results do not provide population-risk certificates.

Primary area:
learning theory
Keywords:
Model merging, LoRA, Contextual rescue, Downward closure

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_HYMaOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

It is known that merging several LoRA adaptors can result in worse per-task performance due to, for example, cancellation of weights. This paper broadly studies when an infeasible combined pair of LoRA adaptors can be "rescued" by merging a third adaptor. The authors provide a particular rank-one regime where rescuers can be easily found. The experiments look at several banks of LoRA adaptors and show that most bad pairs can be rescued by merging a third LoRA adaptor.

Critical strengths

None

Critical weaknesses
  • I found this paper to be extremely hard to parse in terms of both its motivation and its main theorems. There are many times when terms are used before definitions are given, if a definition is given at all. This makes its very difficult to understand what is actually being proved. For example, I can't find a definition of a "opposite-type pair" or "feasibility". The related works seems to just list out papers and their main contributions without addressing how they relate to the work at hand. I also find it strange that the main body is only 6.5 pages long, yet the total paper is 60 pages. It would have been nice to use the extra space in the main body to flesh out the general motivation, definitions, and notation further.
Questions for authors

None

1 comment

Thank you for pointing out where the motivation, definitions, and connections to related work could be clearer. We will draw on your feedback as we continue to refine the paper after this review cycle.

Reviewer_mhzaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies subset selection for LoRA merging and points out that a pair of adapters that fails a task-loss cap can become acceptable once a third adapter is added, which the authors call a rescue. They give a softmax-attention construction where rescue is abundant, show that banning bad pairs can be arbitrarily costly in a worst-case example, and find through exhaustive evaluation of six CLIP adapter banks that most bad pairs have at least one rescuer. Finally, they propose a method to predict rescuers by using the interaction scores from the pairwise results.

Critical strengths
  1. The core observation is simple and clearly stated, and the toy attention mechanism gives an intuitive explanation of why rescue can happen.

  2. The analysis experiments are well set up: every pair and triple is evaluated, and adapters are tested on separate discovery and audit data.

Critical weaknesses
  1. Gap between theory and experiments. The theory characterizes adapters by their influence on the attention scores, but there is no analysis showing that real-world trained adapters follow this behavior.

  2. Unclear practical benefit. The paper argues against banning merges containing bad pairs, but this selection rule doesn’t seem to be an established practice. As a result, it’s hard to judge if evaluating all pairs and predicting rescuers is more practical than other approaches, such as model merging methods or random search over all merge subsets. Since the proposed approach requires evaluating all pairs, the gains would be more convincing if shown as a function of the evaluation budget, compared against simple alternatives like random search using the same number of evaluations.

Questions for authors

Does rescue also happen in larger merges e.g. with > 2 candidates and/or multiple rescuers?

1 comment

Thank you for the careful review and the specific feedback. We will draw on it as we continue to refine this work after the current review cycle.

Our work studies how low-rank updates can interfere with each other or compensate for each other. The longer-term goal is a low-cost way to anticipate these interactions before choosing which updates to combine or apply. As a research direction, such an estimate could eventually help ReLoRA-style training, which periodically merges low-rank updates into the model, choose its update directions.

We use permanent exclusion as a controlled comparison under a fixed merge rule. It measures what is lost when every larger merge containing a failing subset is ruled out. Appendix G, Table 9 shows mixed results against random and greedy search at a single matched budget of 37 evaluations.

The controlled attention example shows one way rescue can arise. Our CLIP results measure task errors in trained adapters.


Q: Does rescue also happen in larger merges e.g. with > 2 candidates and/or multiple rescuers?

A: Yes. In our earlier exhaustive rank-8 evaluations of six CLIP adapter banks, 195 triples failed at least one task limit. Adding one adapter rescued 132 of them. In 41 of those cases, at least two different adapters could each rescue the triple on their own. Another 61 triples could be rescued only by adding two or more adapters together.

Reviewer_FMgXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies rescue in LoRA adapter merging setting, a situation in which a pair of adapters violates a task-performance requirement, but adding a third adapter makes the combined model satisfy all requirements.

Critical strengths
  • The problem of weight conflicts in model merging might be an interesting problem, but the paper's convoluted presentation prevents the reader from appreciating the paper's contribution (see weaknesses below).
  • The paper presents some theory to support the claims and runs experiments on Transformer-based vision models.
Critical weaknesses
  • The most glaring flaw of the paper is presentation. The paper directly jumps to the problem, without providing the reader enough background about model merging, what are the issues of model merging, and the problem that this paper is trying to solve. It takes several passes to have a good grip on the paper's contribution. While this is not a flaw from a methodological point of view, but presentation is very vital in understanding a paper, and the authors did not do any justice. The main paper finishes within 7 pages, thus there were enough space available to provide a reader with sufficient introductory knowledge about model merging and motivation behind solving the task at hand.
  • In the introduction, the authors use terms such as "coalition", "rescue", etc. -- non-standard terms in the deep learning literature. It is recommended to first introduce these terms sufficiently well before using them, or write the introduction in a more generic manner, and then gradually introduce these technical terms in the problem formulation part.
  • The methodology section is also unreadable. Instead of reporting the theorems and proofs, it would be good to prioritize a more intuitive explanation of rescue adapters, and some empirical analyses.
  • It would be good to provide a graphical representation of the conflict graph when adapters are merged and the role that the rescue plays.
  • The motivation of this work is not clear to the reviewer. What is the value of studying in depth a special case scenario, where a rescue (say adapter C) can lower the task-loss with respect to merging two tasks (say adapters A+B)? Will such a scenario ever occur in practice? If so, can you please provide an intuitive example?
Questions for authors
  • My earnest request to the authors is to please improve the presentation of the paper. You might be solving a cool problem, but if not conveyed to the readers in a digestible way, with enough background and intuitive explanation, you risk losing the reader halfway into the paper.
  • Please consider adding some graphical representation or schematic diagrams to better illustrate the problem.
1 comment

Thank you for your practical suggestions on the presentation of our paper. We will draw on them as we continue to refine this work after the current review cycle.

LoRA weight updates add like vectors. In a nonlinear model, the effect of one update on a task can depend on which other updates are present. Merging A and B can therefore make a task miss its performance limit. Adding C can bring the merge back within the limit. We study where this interference comes from, which structures produce it, whether trained adapters show it, and how much is lost when every larger merge containing a failing subset is ruled out.

In the paper's controlled attention example, two adapters cancel each other's preferences for useful inputs, which draws attention toward a distractor. A third adapter restores a useful preference. Whether the same mechanism explains the results in trained CLIP adapters remains an open question.

With CLIP ViT-B/16 and the paper's rank-8 merge, MNIST+SVHN raises EuroSAT error by 6.05 percentage points over the base model, above our 2-point limit. Adding RESISC45 gives a triple that keeps the error increase within the 2-point limit on all eight tasks.

In future work, we hope to find a low-cost way to estimate how candidate low-rank updates will interact before we apply them. ReLoRA trains with low-rank updates and periodically merges them into the model. Such an estimate could help a ReLoRA-style optimizer choose update directions by weighing how successive updates work together alongside their immediate loss decrease.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.