- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies continual LoRA tuning of LLMs where one fixed-rank adapter is shared across a task sequence. It argues that an adapter update alone cannot determine how much it perturbs a layer’s responses on past inputs. It also argues that the uncentered second moment of historical layer inputs is enough to measure this layer-local drift. HARD AG-LoRA freezes the output factor after Task 1, stores a capped reservoir of prompt-only activations at each task boundary, and projects later input-factor displacements away from the reservoir’s leading subspace. A Refresh variant re-encodes stored prompts under the current model instead. Evaluation covers four classification tasks in three orders on Qwen2.5-7B and Llama-3.1-8B, plus a mechanism ablation, a 15-task extension and a forgetting-diagnostic study. The authors conclude that a compact stored activation sketch lets one fixed-rank adapter match growing-adapter methods and beat small-buffer replay.
Critical strengths
- The mechanism controls are careful (Section 4.3, Table 13, Appendix E.3). These are Freeze-B alone, a rank-matched random projector with energy-coverage diagnostics, a soft penalty, and stored vs refreshed geometry. Together they show that the historical directions, not just the restricted update space, add retention. The three-order Freeze-B gap in Table 11 supports this beyond Order 1.
- A rank-8 adapter matches growing-rank O-LoRA on Qwen with low cross-order variance (Table 2), and persistent state is measured directly (Table 3). Unfavorable results are reported candidly: InfLoRA, Llama Order 3, and calibration KL in Table 6.
Critical weaknesses
- HARD vs O-LoRA (80.96 vs 80.72) is within O-LoRA’s order std; O-LoRA wins Order 2 and InfLoRA wins all orders (Table 11). The stored-over-refreshed claim rests on Order 1 (Table 1, Figs. 2–3); on Order 3, Refresh has higher AA (81.57 vs 81.09). The random projector is Order 1 only.
- Replay-64 is not budget-matched: 0.03 vs 25.71 MB of auxiliary state (Table 3), and 64 examples total vs 256 probes per task.
- The 15-task run (Appendix H) compares only against Sequential, on one seed. Algorithm-R dilutes each old task’s reservoir share as tasks accumulate, and forgetting reaches 17.81.
- With B frozen, Eq. (9) is GPM/OWM-style input-subspace projection on the A factor. Adam-NSCL, AlphaEdit, CorDA and PGP project updates away from activation covariance but are not compared, and GPM applied to the shared LoRA is missing as a baseline. N-LoRA, SAPT and SLoRA use the same benchmark but are absent from the tables.
- Prop. 1 is an elementary null-space construction, and Eq. (25) is Courant–Fischer. Both are layer-local, so they motivate the method rather than contribute theory.
Questions for authors
- Can you report three or more seeds per order for HARD, Refresh, O-LoRA and Replay-64 on Qwen? Do the O-LoRA margin and the Order-3 Refresh reversal survive?
- How do all methods compare under the scheduled LR used for Llama Sequential, on both backbones?
- How does replay do with 256 labeled examples per task, or a byte budget matched to the reservoir?
- Do you have results for GPM-style projection on the shared LoRA with B trainable, and for O-LoRA or InfLoRA on the 15-task sequence?