acceptodds
Under review as a conference paper at ICLR 2027

New LoRA Skills Should Read but Never Write

Abstract

Low-rank adapters (LoRA) make it cheap to fine-tune a large language model once per task, but combining several independently trained adapters into one model remains difficult: merging the updates in weight space causes interference, retraining on all task data is expensive, and routing between separate adapters gives up the goal of a single combined model. We trace the difficulty to two choices that every composition method makes implicitly. A LoRA update admits infinitely many equivalent factorizations; the choice among them is invisible while an adapter serves alone, but it determines what a learned interaction between adapters can see. A coupling between an old skill and a new one can likewise point in either direction, and the direction decides whether the old skills keep computing what they computed before. We introduce READ (Read-only Expansion of Adapter Deltas), which fixes both choices: each adapter is rewritten into a balanced canonical form that preserves its update exactly, and the coupling grows in one direction only, so a new skill can read the input subspaces of old skills but cannot write into their output subspaces. The only trainable object at each append is the new skill's row of the coupling matrix, and the composed update folds into the base weights with no inference cost, routing, or task-specific rules. We evaluate READ across four benchmark suites and two model families, adding skills one at a time. Across several families, READ improves every suite average over the strongest published baselines built from the same adapters—by more than twenty points on SuperGLUE and more than seven points on the domain suite—and nearly all complete addition sequences end above every direct baseline. Factor coordinates and coupling direction, which a lone adapter never exposes, are what decide whether composed skills survive.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
parameter-efficient fine-tuning, LoRA composition, model merging, sequential skill growth

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_w2ZnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes READ (Read-only Expansion of Adapter Deltas), a method for sequentially composing independently trained LoRA adapters into a single model. READ addresses two aspects of LoRA composition: the non-uniqueness of adapter factorization and the directionality of interactions between existing and incoming skills. The evaluation covers Llama-3.2-3B-Instruct and Qwen3-4B on GLUE-6, SuperGLUE-4, Domain-3, and BBH-6.

Critical strengths
  1. The paper identifies two distinct issues: equivalent factorizations of an individual update can induce different interaction coordinates, and the direction of cross-skill coupling determines which components of the composed update can change.
  2. Many evaluation results on append-wise analysis, factorization-sensitivity experiment, etc.
Critical weaknesses
  1. The read-only constraint does not guarantee preservation of old-task functions.
  2. Analyse the main reason of improvement on these experiments. Eg. why are the canonicalization and directional constraints design useful?
Questions for authors
  • What exactly does the read-only constraint preserve?

  • To what extent are READ's performance gains attributable to its structural constraints rather than training on the union of old and new task data?

  • How is incoming closure defined, and what criteria determine whether a transition is eligible?

  • What evidence supports the authors' interpretation of the negative GLUE macro scores, and how were the reported confidence intervals computed?

  • How does READ differ from recent methods such as LoRAGen (ICLR 2026) and DWF (ICLR 2026), particularly in terms of the composition mechanism?

Reviewer_gY6TOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes READ, a framework for sequential LoRA composition. It combines SVD-based canonicalization with a read-only coupling mechanism that allows new adapters to reuse previous skills while keeping existing parameters frozen. Only the new coupling parameters are trained using both old and new task data. The resulting model requires no additional inference overhead and outperforms various LoRA merging baselines on several benchmarks.

Critical strengths
  • READ achieves strong performance across multiple benchmarks, consistently outperforming existing LoRA merging baselines.
Critical weaknesses
  • Many classical merging methods perform surprisingly poorly, with some obtaining near-zero or negative scores on GLUE. More importantly, READ requires additional supervised training on the union of all task datasets, whereas many merging baselines are training-free. This raises concerns about the fairness of the comparisons and makes it difficult to determine whether the improvements come from the proposed read-only coupling mechanism or simply from additional training.
Questions for authors
  • The definitions of Eligible / Observed and the eligibility criteria for sequential append evaluation are insufficiently explained. For example, only 7 out of 20 transitions are considered eligible for Llama on BBH. Why are the remaining transitions excluded, and how does this filtering affect the reported pass rates? The authors should also clarify the definition and calculation of Incoming Closure.
  • Can the authors provide theoretical justification or additional experiments demonstrating that the read-only constraint itself effectively reduces forgetting, independently of the use of old task data during training?
Reviewer_GBAqOpenReview ↗
Rating
3
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes READ to sequentially combine independently trained LoRA skills into a single model whose updates can be folded into its weights. The method first rewrites the adapters into balanced canonical factors, then freezes the existing factors and couplings and trains only the new skill’s coupling row on the union of old and new task data. Experiments cover two model families and four task suites; READ outperforms the strongest merging baseline for each sequence in 24 of 32 final sequences, with all eight losses occurring on BBH. The results show practical promise, but do not sufficiently support the guarantee of preserving old behavior or the claim that the two core design choices determine composition success.

Critical strengths
  1. The folding results are consistent and have little impact on model outputs or inference latency.
  2. The experiments cover multiple models, tasks, and settings, which makes the evaluation reasonably broad.
Critical weaknesses
  1. Freezing old coupling blocks does not guarantee unchanged behavior on old tasks. The added term in Eq. (3) can still change outputs on old inputs, so the claim in §3.3 seems too strong.
  2. The paper does not clearly separate the effects of the two design choices from replay training. It is also unclear whether the baselines use comparable data and training budgets. More targeted ablations are needed for canonicalization and the read-only constraint.
Questions for authors
  1. What exactly does §3.3 preserve: the old parameter components for a fixed layer input, or the full model’s outputs on old tasks? If it is the latter, what assumption guarantees that Eq. (3) has no effect on old inputs or downstream computations?
  2. With the same replay data and optimization steps, what are the per-task retention results, acquisition success rates, and confidence intervals for paired differences across the six coordinate–direction configurations? How much data and what optimization budget did CAT and LoRAHub each use in Table 1?
  3. How are refit, incoming closure, eligible transition, and retained-share defined? Why were 28 of the 120 transitions excluded? Which tasks, data splits, and label-parsing rules are used in GLUE-6, SuperGLUE-4, BBH-6, and Domain-3?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.