Rescuers Matter: Rethinking Conflict in LoRA Merging
Abstract
When merging LoRA adapters, a pair that violates a task-loss cap can become feasible after another adapter is added. We call this a rescue. We ask when rescuers can be abundant and how often they occur in trained adapter banks. An exact softmax construction realizes rescue with rank-one LoRA: opposing useful-token preferences can cancel in a pair and be restored by a third adapter. In a robust two-type regime, every opposite-type pair is rescued by every remaining adapter; under balanced iid types, the certified fraction of distinct rescued triples tends to three quarters. A rank-one quadratic construction also gives a sharp cardinality ratio for sound downward-closed policies. Exhaustive audits of six new CLIP adapter banks find rescuers for most bad pairs overall, with rescue in every bank even under stricter margins. Earlier complete-cube audits show that permanent exclusion can reduce merge accuracy. The theory concerns explicit constructions and sampling laws; the empirical results do not provide population-risk certificates.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.