acceptodds
Under review as a conference paper at ICLR 2027

Emergent, Coordinate-Specific Structure in Dense Parameter Updates for Continual Learning

Abstract

Continual learning (CL) requires models to learn new tasks while retaining previous knowledge, yet sequential parameter updates can cause catastrophic forgetting. Parameter-efficient adaptation methods such as LoRA address this challenge by restricting updates to a low-rank subspace, limiting the degrees of freedom of each task update. We ask the converse: what structure emerges when the full parameter space remains unconstrained and is updated continuously across tasks? We freeze a pretrained model and train a same-sized parameter state from zero with standard AdamW. We find that continuous dense updating develops cross-task co-directionality and a contracting effective rank, in contrast to the rank expansion observed in LoRA, and that this structure favours knowledge retention across tasks. Coordinate and trajectory interventions show that retention depends on both where updates are written and whether their accumulation remains continuous: preserving update values while permuting their coordinates sharply reduces retention, while interrupting the trajectory at task boundaries also degrades it. At matched learning amounts, LoRA incurs greater per-task parameter displacement, accounting for much of its additional forgetting, with residual dependence on rank. Across the swept range, the dense adapter achieves better retention than LoRA, while its frontier intersects that of EWC. These results reveal that continuous unconstrained updating itself forms structured parameter dynamics with measurable consequences for continual learning.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, catastrophic forgetting, fast weights, slow weights, memory consolidation, continual adaptation, parameter updates

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_451MOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper compares the updates of dense fine-tuning and LoRA when training sequentially on continual learning tasks, questioning the effectiveness of LoRA for reduced forgetting.

Critical strengths

None.

Critical weaknesses

I consider this manuscript substantially below the standard expected of a scientific submission (let alone publication).

Content-wise, it is very hard to understand the claim of the paper. Firstly and foremost, it presents to depict the "unrestricted" fine-tuning as something novel to study, when in reality it is just ordinary fine-tuning. Secondly, it fails to explain why looking at the rank dynamics is important and useful, and why the inverse rank evolution in unconstrained vs. LoRA is surprising. This latter part lacks context about the rank at initialization, which would clarify the actual dynamics. Overall, the paper is vague, but tries to convince the reader with overconfident wording of the discoveries presented.

The paper also has major editorial problems. Indeed, it is very hard to read and presents multiple classical AI-style behaviors that hinder the comprehension of the content. Below are the details:

  • There are multiple completely empty sections in the appendix, clearly leftovers from previous iterations, e.g. A.10, A.11, A.15, etc.
  • There are multiple meta-comments, e.g. in the title of section A.22 "moved from the main text", but also in A.23, A.24.
  • The text is very hard to read due to what looks like AI cues and convoluted wording. Some examples:
    • unnecessary terminology, e.g. more than 40 "arm", 9 times "rung"
    • long sentences and convoluted syntax, e.g. around 15x em-dash sentences
    • extra-defensive statements and over-explanations (e.g., about what the paper is not doing), the most evident are at lines 94-97, 175-176, Section 4.6, Section A.8, Section A.13. These, instead of clarifying, confuse the reader even more with unnecessary details and nitpicking.

I believe that the submission did not receive sufficient critical scrutiny before submission.

(Minor: the title does not use the usual capital letters font of ICLR)

Questions for authors

None.

Reviewer_biYpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work investigates weight dynamics in vanilla continual learning of tasks in large language models and contrasts them with LoRA. The manuscript, particularly the Appendix, appeared to be unfinished, and the details on models, tasks, and measurements are not presented clearly; thus, I found it difficult to evaluate the soundness and significance of the content as presented.

Critical strengths

None

Critical weaknesses

The manuscript appears to be unfinished. Several Appendix sections, A.10, A.11, A.15, and A.18, do not have any content. The remaining Appendix sections, as well as the main text, are not written in a clear and understandable manner. Missing Appendix sections appear to be crucial because, for instance, L264-265 of the main text state: “The full description of the decomposition, …, is in Appendix A.15”. It is thus impossible to assess the manuscript as presented.

The motivation for studying vanilla continual learning is not well articulated. The manuscript frames the absence of LoRA as novel, but the underlying logic is difficult to comprehend. The overall lack of clarity does not help clarify this.

Questions for authors

None

Reviewer_FpSkOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies continual learning with a frozen pretrained backbone and a same-sized side weight W_A, initialized at zero and trained with plain AdamW across ten tasks. The authors call this the dense adapter. They report that successive per-task updates become co-directional and the effective rank of the update contracts, whereas a continuously updated LoRA shows the opposite trend. At matched forward adaptation, the dense adapter retains more than LoRA over most of the swept range. Using a separate fast/slow decomposition with a scalar writeback at task boundaries, they show that permuting the coordinates of a written update (same energy, different placement) destroys retention, and that interrupting the accumulation at boundaries also hurts. They attribute most of LoRA’s extra forgetting to larger per-task displacement, with a residual rank dependence. Against EWC they report a crossing of frontiers rather than an advantage. All experiments use GPT-2 on 20-Newsgroups with 10 tasks of 60 steps.

Critical strengths

S1. The isometric coordinate intervention is a clean tool. Permuting the coordinates of a written update keeps the multiset of values exactly the same, so the written energy and every symmetric statistic are unchanged. Only the placement changes. This separates “how much is written” from “where it is written” more strictly than adding noise or projecting, which would change the norm. The graded version (coordinate fidelity from 100% to 0%, Appendix A.24) turns placement into a dose axis instead of a two-point contrast. I do not fully agree with how the authors interpret the result, but the tool itself can be reused by other continual learning work. S2. The paper tries to explain why LoRA forgets more, not only that it forgets more. Section 5.1 matches LoRA ranks at the same forward adaptation and shows that most of the extra forgetting tracks the larger per-task displacement that the low-rank constraint forces (R² = 0.79). Adding rank as a second predictor isolates a residual rank effect. Framing the cost of LoRA as a geometric quantity, rather than as rank alone, is a useful idea and could guide new adapter designs that control displacement directly. S3. The related work connects continual learning to selective consolidation in neuroscience (Leimer et al., Lindsey and Litwin-Kumar), fast weights, and complementary learning systems. The authors use these links to ask a specific question: whether placement structure can appear without any selector. This makes the related work section more than a list of methods.

Critical weaknesses

W1. The experiments use only GPT-2, a model released in 2019. All results come from a single backbone family and a single dataset (20-Newsgroups) with 10 tasks of 60 steps. GPT-2 has a different architecture and training recipe from current models (no RoPE, no grouped-query attention, no SwiGLU, no instruction tuning), and its weights are far from the scale and quality that adapters like LoRA are used on today. The paper’s main observation is about the geometry of parameter updates on a pretrained model, and that geometry can depend on how the backbone was trained. So it is not clear that the rank contraction, the co-directionality, or the displacement result would hold on a modern model such as Llama 3, Qwen 2.5, or Gemma 2 at a comparable size. The authors list the single backbone as a limitation (Section 6), but the practical weight of the paper’s claims depends on this choice. One modern backbone at 1B scale, even on the same protocol, would make the results much easier to trust. W2. The appendix is not finished. Sections A.10, A.11, A.15, A.18, and A.21 contain only a heading and no text. Section 4.1 in the main text says the full description of the decomposition is in Appendix A.15, but A.15 is empty and the content appears under A.16 and A.17 without being referenced there. A.21 says the supporting tables are collected “here”, but the tables are placed elsewhere. This makes the paper hard to verify, since several main-text claims point to sections that do not exist. The AI use statement says the paper was drafted and structured with generative AI. Combined with the empty sections, this suggests the submission was not fully checked before upload.

Questions for authors
  1. All results use GPT-2 on 20-Newsgroups. Do you have any result on a modern backbone at a similar size (Llama, Qwen, or Gemma around 1B)? One run on the same protocol would show whether the rank contraction and the retention gap depend on the backbone.
  2. Sections A.10, A.11, A.15, A.18, and A.21 are empty. Section 4.1 refers to A.15 for the full description of the decomposition. What was meant to be there, and will it be added?
  3. Have you tried training the dense adapter and then truncating W_A to a low rank by SVD after training? Figure 1 suggests the update is already low rank. If retention survives truncation, the storage cost problem goes away and the paper would have a practical result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.