acceptodds
Withdrawn from ICLR 2027

What Should Continual LoRA Remember? Historical Activation Geometry for Fixed-Capacity Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continual LoRA updates a shared low-rank adapter across tasks, but preserving previous behavior under fixed capacity depends on what information is retained from past data. We first show that adapter updates alone cannot identify their historical layer-response drift uniformly over past input distributions: the same update can induce zero or maximal drift on different histories. Guided by this result, we compare two uses of few-shot history, replaying labeled examples in the training loss and encoding prompt-only examples as activation geometry. We propose HARD AG-LORA, which stores a capped reservoir of historical layer activations, freezes the LoRA output factor, and projects subsequent input-factor displacements away from the leading historical activation subspace. This preserves direct adapter responses on the protected coordinates while maintaining a single fixed-rank adapter; a controlled variant further shows that storing activations when tasks are learned is more effective than recomputing them under the current model. Across three orders of four text-classification tasks on Qwen2.5-7B, HARD AG- LORA achieves 80.96% final average accuracy with 0.95 points of forgetting, compared with 77.82%/5.36 for Replay-64 and 80.72%/1.20 for growing-rank O-LoRA. On Meta-Llama-3.1-8B, it achieves 78.51%/4.58 versus 75.14%/8.20 for rank-32 O-LoRA. HARD AG-LORA keeps one rank-8 adapter and a task-count- independent activation sketch, providing an actionable fixed-capacity representation of historical information for continual adaptation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Low-Rank Adaptation, Parameter-Efficient Fine-Tuning, Continual Learning, Catastrophic Forgetting, Functional Preservation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_Le8vOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies continual LoRA tuning of LLMs where one fixed-rank adapter is shared across a task sequence. It argues that an adapter update alone cannot determine how much it perturbs a layer’s responses on past inputs. It also argues that the uncentered second moment of historical layer inputs is enough to measure this layer-local drift. HARD AG-LoRA freezes the output factor after Task 1, stores a capped reservoir of prompt-only activations at each task boundary, and projects later input-factor displacements away from the reservoir’s leading subspace. A Refresh variant re-encodes stored prompts under the current model instead. Evaluation covers four classification tasks in three orders on Qwen2.5-7B and Llama-3.1-8B, plus a mechanism ablation, a 15-task extension and a forgetting-diagnostic study. The authors conclude that a compact stored activation sketch lets one fixed-rank adapter match growing-adapter methods and beat small-buffer replay.

Critical strengths
  1. The mechanism controls are careful (Section 4.3, Table 13, Appendix E.3). These are Freeze-B alone, a rank-matched random projector with energy-coverage diagnostics, a soft penalty, and stored vs refreshed geometry. Together they show that the historical directions, not just the restricted update space, add retention. The three-order Freeze-B gap in Table 11 supports this beyond Order 1.
  2. A rank-8 adapter matches growing-rank O-LoRA on Qwen with low cross-order variance (Table 2), and persistent state is measured directly (Table 3). Unfavorable results are reported candidly: InfLoRA, Llama Order 3, and calibration KL in Table 6.
Critical weaknesses
  1. HARD vs O-LoRA (80.96 vs 80.72) is within O-LoRA’s order std; O-LoRA wins Order 2 and InfLoRA wins all orders (Table 11). The stored-over-refreshed claim rests on Order 1 (Table 1, Figs. 2–3); on Order 3, Refresh has higher AA (81.57 vs 81.09). The random projector is Order 1 only.
  2. Replay-64 is not budget-matched: 0.03 vs 25.71 MB of auxiliary state (Table 3), and 64 examples total vs 256 probes per task.
  3. The 15-task run (Appendix H) compares only against Sequential, on one seed. Algorithm-R dilutes each old task’s reservoir share as tasks accumulate, and forgetting reaches 17.81.
  4. With B frozen, Eq. (9) is GPM/OWM-style input-subspace projection on the A factor. Adam-NSCL, AlphaEdit, CorDA and PGP project updates away from activation covariance but are not compared, and GPM applied to the shared LoRA is missing as a baseline. N-LoRA, SAPT and SLoRA use the same benchmark but are absent from the tables.
  5. Prop. 1 is an elementary null-space construction, and Eq. (25) is Courant–Fischer. Both are layer-local, so they motivate the method rather than contribute theory.
Questions for authors
  1. Can you report three or more seeds per order for HARD, Refresh, O-LoRA and Replay-64 on Qwen? Do the O-LoRA margin and the Order-3 Refresh reversal survive?
  2. How do all methods compare under the scheduled LR used for Llama Sequential, on both backbones?
  3. How does replay do with 256 labeled examples per task, or a byte budget matched to the reservoir?
  4. Do you have results for GPM-style projection on the shared LoRA with B trainable, and for O-LoRA or InfLoRA on the 15-task sequence?
Reviewer_AsjmOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks what a fixed-rank LoRA adapter should retain to limit forgetting. Its theory shows that the same adapter change can cause different layer-local drift under different historical input distributions. HARD AG-LoRA stores a capped activation reservoir, freezes the output factor, and protects leading historical activation directions. Controlled comparisons test stored versus refreshed activations and historical versus random projection directions. On four Qwen classification tasks, it reports 80.96% average accuracy and 0.95 percentage points of forgetting with one rank-eight adapter.

Critical strengths
  • The layer-local theory directly motivates what the method stores and protects. Appendix A separates sampling error from sketch-compression error, connecting the analysis to the finite reservoir.
  • REFRESH, Freeze-B, and matched-rank random projection provide useful controls (Tables 11 and 13). They support the value of historical activation directions beyond freezing the output factor alone.
  • The memory comparison counts persistent state, including the activation reservoir. Table 3 reports 35.82 MB for HARD versus 40.39 MB for O-LoRA, making the capacity tradeoff concrete.
Critical weaknesses
  • The margin over O-LoRA is small: 80.96% versus 80.72% average accuracy (Table 2). The reported spread is across three task orders, not repeated seeds, so the reliability of this margin remains unclear.
  • The main evidence covers four task-aware English classification datasets. Appendix H extends to 15 tasks, but uses one seed and only Sequential as a baseline, leaving long-horizon comparisons with replay and O-LoRA open.
  • The capacity choices remain fixed: 64 reservoir rows, a 95% energy threshold, and a rank cap of 64 (Appendix C). It is unclear how much memory can be removed before retention suffers, or how these choices affect learning new tasks.
Questions for authors
  • Could you sweep reservoir size, retained-energy threshold, and rank cap? An accuracy-and-forgetting versus memory curve would help justify the chosen capacity.
  • Why is retention weaker on Llama, where average forgetting is 4.58 percentage points? Would an identical learning-rate schedule for HARD and Sequential change the Order 3 result in Table 14?
  • How would the method work without known task boundaries or task identity? The current protocol relies on boundary-time activation collection and task-aware prompts.
Reviewer_gZZzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a LoRA-based continual learning method, HARD AG-LoRA, which utilizes one fixed-rank LoRA to preserve old tasks and learn new tasks. Established on an information requirement that the same update can behave different historical layer-response drift effects on histories, HARD AG-LoRA retains historical layer activations instead of LoRA parameters, maintaining old tasks information after immediately training each task. For the update rule, HARD AG-LoRA freezes LoRA output factor and uses orthogonal projection to update input factor. Experiments across Qwen and Llama models on text-classification tasks show that HARD AG-LoRA achieves competitive performance compared to baselines.

Critical strengths
  • HARD AG-LoRA has a good motivation that studying fixed-capacity LoRA continual learning is important and novel for this field.
Critical weaknesses
  1. The paper does not provide clear explanations for the design of HARD AG-LoRA in Section 3.2 and make confused about why designing like that. For example, in Eq(8), how is the second part derived C_l,tQ_l,t=Q_l,t_l,t? And in Eq(9), why is it designed to freeze B rather than freeze A?

  2. The experimental setting only uses four text classification datasets to evaluate the proposed method and baselines, which does not follow the variable and long benchmarks used in other baselines, thus making the experimental results not fully validate the effectiveness of HARD AG-LoRA. The ablation studies do not study several necessary and important factors that may change the performance of HARD AG-LoRA, such as sweeping ranks for HARD AG-LoRA and comparing the performance of freezing A and updating B. The lack of foundational experiments make the results weak to support the proposed method.

Questions for authors
  1. The experiments only use text classification datasets, but other LoRA-based continual learning methods also use text generation, text understanding, math problems, and code benchmarks when using LLMs. Can authors use the SuperNI benchmark in SAPT [1] to evaluate HARD AG-LoRA and compare with baselines?

    [1] “SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Models”, ACL 2024.

  2. The length of the task sequence in the experiment is only four, which is too short for continual learning evaluation, since other baselines often use 15 tasks or even longer. Since HARD AG-LoRA only uses one LoRA to keep old knowledge and learn new knowledge, it is necessary to know whether fixed-capacity LoRA has limitations or restrictions in the length of the task sequence. Can authors evaluate HARD AG-LoRA on longer task sequences?

  3. For the design of HARD AG-LoRA, can authors explain why to freeze B and update A? LoRA itself has asymmetric properties for its two factors, A and B, as discussed in [2]. In different settings, freeze A or freeze B has different performance. If possible, comparing the performance of exchanging the role of A and B in experiments should be better.

[2] “Asymmetry in Low-Rank Adapters of Foundation Models”, ICML 2024.

  1. For the baseline method Replay-64, can authors explain why using 64 rather than 1%1\% samples?

  2. Can authors give clear mathematical definitions of forgetting and BWT metrics?

  3. Tables 7 and 8 in line 1309 are over the page width, and line 1333 is also over the page width.

Reviewer_gVX8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies what historical information should be retained when one fixed-rank LoRA adapter is trained sequentially. It shows that the same adapter displacement can produce different layer-local response drift under different historical input distributions, with the historical second moment determining the quadratic drift. HardAG-LoRA stores a capped reservoir of prompt-only activations, freezes the LoRA output factor after the first task, and projects subsequent input-factor displacements away from the leading historical activation subspace. Experiments cover four text-classification tasks under three orders on Qwen2.5-7B and Llama-3.1-8B, with a supplementary 15-task Qwen study. The method achieves competitive retention with one rank-8 adapter, and the stored-geometry variant outperforms a refreshed-geometry control.

Critical strengths
  1. The intervention has a clear, correctly scoped mathematical interpretation. With the output factor fixed, the task-anchored projection enforces Dℓ,tQℓ,t=0D_{\ell,t}Q_{\ell,t}=0, protecting direct adapter responses on the retained activation directions.

  2. Projecting the accepted displacement after the optimizer step makes this constraint explicit even with Adam. The paper does not incorrectly extend the guarantee to unchanged network predictions.

  3. The experiments include informative mechanism controls. Comparisons with Freeze-B, matched-rank random projection, and refreshed activations help distinguish activation-informed protection from simply reducing trainable capacity.

  4. The four-task Qwen results are competitive with growing-rank O-LoRA, and the second-backbone evaluation provides additional evidence that the implementation is useful beyond one model.

Critical weaknesses
  1. The central replay comparison does not isolate the value of the historical representation under a common memory budget. Table 3 assigns HardAG-LoRA 25.71 MB of required auxiliary state, compared with only 0.03 MB for Replay-64. Their total persistent states are 35.82 MB and 10.14 MB, respectively.

  2. Appendix B.1 acknowledges that these are not byte-matched, but much of the argument still uses this comparison to favor activation geometry over replay. In addition, HardAG-LoRA processes all 256 boundary probes, whereas Replay-64 retains a much smaller historical sample.

  3. The evidence for the method’s incremental contribution and long-horizon robustness remains limited. Protecting historical activation subspaces is established in Gradient Projection Memory, which the paper acknowledges, but there is no directly matched fixed-rank adaptation of that approach.

  4. The detailed mechanism study is one Qwen task order: Freeze-B already reaches 79.37% AA, compared with 80.84% for the full method. The main standard deviations describe task-order variation rather than repeated training seeds.

  5. The 15-task extension is useful, but compares only with unprotected Sequential LoRA and still records 17.81 points of forgetting. It therefore does not establish that the proposed memory representation remains preferable to competitive fixed-budget alternatives over longer sequences.

Questions for authors
  1. Resource-matched comparison: How does HardAG-LoRA compare with replay when total persistent bytes and access to the 256 boundary probes are matched?

  2. Robustness of the incremental gain: Does the improvement over Freeze-B persist across independent training seeds and on Llama?

  3. For the 15-task sequence, how does it compare with at least one strong fixed-budget retention baseline under comparably tuned optimization settings? The existing three-order standard deviations do not answer these questions.

  4. Distinction from existing projection methods: What additional benefit comes from the capped historical reservoir and task-anchored finite-displacement projection relative to a GPM-style implementation on the same fixed-rank adapter, with identical activation sampling, memory capacity, and optimizer?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.