acceptodds
Under review as a conference paper at ICLR 2027

Adaptation Is Not Retention: Reframing Continual Low-Rank Adaptation through Response Coordinates

Abstract

Low-rank adaptation (LoRA) enables parameter-efficient continual learning, but good current-task fit does not ensure subsequent retention. We identify a controlled *fit–inheritance mismatch*: changing only the task-start down-projection, with the up-projection initialized at zero, preserves the initial model function, yet can yield similar acquisition and substantially different later retention. We study this phenomenon through a response-coordinate view of LoRA, with the down-projection as a *reader* and the up-projection as a *writer*. Under a fixed local response model, we formalize a *response-coordinate bottleneck* in factorized LoRA states: the reader determines the task-response space accessible to adaptation and the local protection boundary carried into subsequent learning, while the writer can only form responses within the reader-exposed coordinates and is optimized in the resulting reader-conditioned geometry. This response-coordinate analysis yields two design principles: *response-supported reader selection* and *reader-conditioned writer optimization*. Building on these principles, we develop **InheritLoRA**, a replay-free continual adaptation method that constructs task-informed readers from response-supported coordinates, optimizes their writers in reader-conditioned geometry, and retains completed low-rank blocks. We evaluate InheritLoRA on T5-large and LLaMA 3.1–8B across the Standard and Long-Sequence continual-learning benchmarks. On the 15-task Long-Sequence benchmark, InheritLoRA matches the leading replay-free method on T5-large with substantially higher adaptation efficiency; on LLaMA 3.1–8B, it surpasses the strongest previously reported replay-free LoRA-based method and even its replay-assisted variant at markedly lower adaptation cost. Controlled interventions further probe the inheritance mechanism, showing that unprotected update components contribute to old-task interference and that inherited reader protection can alter subsequent interference even with the learned function held fixed.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Low-Rank Adaptation, Parameter-Efficient Fine-Tuning, Large Language Models, Subspace Tracking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CzN6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a method for alleviating forgetting in LoRA-based learning. They propose thinking of LoRA's weight update BAx as a reader (up projection A) and a writer (down projection B) for input X. Here, A is a feature selector of X, and then B is the writer that turns the selected features into updates. This interpretation helps explain forgetting in LoRA-based learning. Based on this, the authors propose Inherit LoRA, which constructs A using the task's gradients, filters it against the previous task's reader direction, and freezes it while learning B. This shows a better retention-acquisition tradeoff

Critical strengths
  • The papers show competitive results on task retention with fewer trainable parameters (LoRA rank 8)
  • I like the reader-writer interpretation. It shows a useful framework for understanding LoRA updates (and the authors have provided a good rationale for it)
  • The authors have done a good job with extensive ablations, including reader construction, preconditioning, AdamW reset, etc. These help assess how different components contribute to the performance
Critical weaknesses
  • The reader ablation Table 1's claims are a bit confusing. The results don't necessarily show that this produces better acquisition accuracy than a randomly initialized. Better final accuracy indicates methods like O-Lora seem to work better with a task-specific reader. But the reasoning behind this is not clear
  • The reader construction ablation (Table 4) shows GradResp is not necessarily the best reader constructor
  • The pre-conditioning ablation (Table 5 and 30) is a bit unclear. Table 5 shows degradation in Final ACC w/o preconditioning. But for different lr, Table 30 shows not much difference.
Questions for authors
  • How general is the effect for the reader? Did the authors find that the result (table 1 and 16) hold with llama models too?
  • Table 5 shows a pre-conditioning benefit, but Table 30 for some learning rates produces nearly similar final ACC. What is authors' takeaway from this? Is pre-conditioning improving robustness across different learning rates or showing a better task retention trade-off, rather than improving the best achievable final accuracy?
  • Resetting the optimizer state every update vs resetting every 4 updates shows very similar performance. What led to the choice of resetting every update?
Reviewer_H8KpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper argues that LoRA states with similar current-task fit can differ substantially in later retention. It interprets the down-projection (matrix A) as a “reader” of task-response coordinates and the up-projection as a “writer”, deriving the output from the features produced by the reader. InheritLoRA initializes task-informed readers using gradients on a small number of examples, filters them against historical reader subspaces, freezes them, and trains writers with Gram preconditioning and per-update optimizer resets. The authors demonstrate a reader deficit which impacts later retention, along with empirical improvements on downstream performance in a continual learning setting.

Critical strengths
  1. Interesting controlled finding. Changing only the initial reader, with the writer starting at zero, can substantially affect later forgetting even when initial task performance changes little (Table 1). Task-informed readers outperform random ones in the matched tests (Table 4; Appendix D.3).
  2. Good experimental coverage. The method achieves strong results without replay. The authors provide useful ablations, results for individual seeds, details of the baselines, and negative results.
Critical weaknesses
  1. The theory does not clearly explain the retention gains. Theorem 3.1 and Corollaries 3.2–3.4 concern fitting at a fixed model state and local sensitivity, not forgetting across tasks. Reader deficit does not reliably predict forgetting (D.6, D.9), and swapping the reader used for future protection gives mixed results (D.8). A simpler explanation is that task-informed readers select useful subspaces, which later training protects through orthogonalization, as in GPM (Saha et al., 2021). This fits Table 1: GradResp substantially improves final accuracy and forgetting in O-LoRA and GORP, which use historical subspace constraints, but much less in IncLoRA, which does not. Furthermore, changing the initial reader changes the writer learned later, so the gains cannot be attributed to reader protection alone.
  2. Novelty and the need for each component are unclear. InfLoRA (Liang & Li, 2024) already uses fixed, task-specific down-projections and protection against past tasks. EVA (Paischer et al., 2025) uses activation-based initialization, and Zhang & Pilanci (2024) study preconditioning for LoRA. Gradient-SVD slightly outperforms GradResp on T5 (Table 4), while diagonal and full-Gram preconditioning perform similarly (Table 35). The one-step least-squares result also does not directly apply to the actual AdamW update with resets (Eq. 54). Finally, the baseline comparisons need more context: the authors' local LLaMA O-LoRA reaches 80.8% on the 15-task benchmark, compared with the third-party 71.4% in Table 2. InheritLoRA also scores below the local IncLoRA baseline on the 4-task benchmark with LLaMa (79.9% vs 81.2%). These runs are not all from identical pipelines (Tables 2, 10). Training-step timings exclude reader initialization (Table 12).
Questions for authors
  1. Given the mixed results in D.6-D.9, what evidence, beyond your initial motivating ablation, shows that reader deficit or reader protection explains the retention gains, rather than the established idea of protecting task-relevant subspaces?
  2. What does InheritLoRA add beyond InfLoRA? Please compare against it in the matched pipeline. On LLaMA, please also compare Gradient-SVD and Activation-SVD readers; currently only GradResp and Random are compared.
  3. With sign descent (Eq. 54), a positive diagonal preconditioner should have no effect beyond the epsilon term. Yet Diagonal and Identity differ by 3.2 pp on one seed (Tab. 36). What explains this, and what is the run-to-run variance at a fixed seed?
  4. Can Tab. 2 report matched local baselines for both backbones, with paired tests or confidence intervals? Can the writer ablations be compared at each configuration's best learning rate?
  5. How were λ, τ, the probe budgets, learning rates, and reset frequencies chosen?
Reviewer_wa2KOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines why LoRA states with similar current-task accuracy can differ in their retention after subsequent tasks. It views the down-projection as a reader that exposes response coordinates and the up-projection as a writer that acts on them. A fixed-state analysis decomposes response-fitting error into a rank limit, a reader deficit, and a writer gap. Motivated by this analysis, InheritLoRA constructs a task-informed reader, filters it against historical readers, and trains a fixed-reader writer using Gram preconditioning and per-update AdamW-state reset. It freezes and retains a new low-rank block after each task without historical-example replay. Its reported results are on the 15-task sequences.

Critical strengths
  • The reader intervention exposes an interesting continual-learning effect. Changing only the initial reader while setting the new writer to zero preserves the task-start function. For O-LoRA and GORP, this changes final accuracy substantially more than acquisition accuracy (Table 1).

  • The analysis gives the method a coherent rationale. It distinguishes response directions a reader cannot expose from error the writer can still reduce. The paper states the fixed-state scope of the results and separately ablates reader selection and writer optimization.

  • The long-sequence results are promising. InheritLoRA reports 75.9% final accuracy on T5-large, close to GORP’s reported 76.0%, and 82.6% on LLaMA 3.1–8B, above SPARTA’s reported 80.0% (Table 2). Seed-level results and diagnostic interventions help qualify these findings.

Critical weaknesses
  • The theory only partially explains long-term retention. Its exact results concern a fixed-state response-fitting objective, whereas the method trains with cross-entropy, sampled Gram estimates, and AdamW. The local protection result does not guarantee less forgetting during subsequent tasks; the measured reader deficit also fails to consistently predict later forgetting (§5.2).

  • The need for the specific reader and writer choices remains unclear. On T5-large/O4, Gradient-SVD slightly outperforms the default GradResp reader (76.89% versus 76.39% final accuracy; Table 4), while diagonal and full Gram conditioning achieve similar mean final accuracy (Appendix D.4). Per-update AdamW reset also produces an approximately sign-based step, without a directly tuned sign-update control. The ablations establish that task-informed readers and writer updates matter, but do not clearly establish the added value of these particular implementations.

  • The resource evaluation does not reflect an ongoing task stream. Each task adds a retained reader–writer block, so adapter storage and unmerged forward cost grow with the number of tasks. The appendix reports 51.1 million retained adapter coefficients after 15 tasks on LLaMA 3.1–8B. The headline timing and throughput measurements cover only the first task, and step timings exclude reader construction and Gram setup. No consolidation mechanism is evaluated for continued learning.

Questions for authors
  • How much of the current task’s leading response subspace remains after historical-reader filtering, and how does the installed reader’s effective rank evolve as more tasks are added?

  • Reader construction probes the original base model rather than the cumulative model at task start. How would reader selection and final retention change if the cumulative model were used?

  • How does a learning-rate-tuned sign-update baseline, with and without Gram preconditioning, compare with the proposed writer update?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.