acceptodds
Under review as a conference paper at ICLR 2027

Stability Does Not Mirror Plasticity: A Module-Level View of Declarative and Procedural Learning in Transformers

Abstract

Plasticity and stability in continual learning are typically studied at the entire-model level and in a task-agnostic manner, yet prior work has reported divergent conclusions across settings. Inspired by cognitive distinctions between *declarative* and *procedural* memory, we instead study how plasticity and stability vary across transformer modules and task types. In a controlled synthetic environment with clean measurement, we show a double dissociation: query-key (QK) and value-output (VO) projections are the most plastic on procedural tasks, whereas MLPs are the most plastic on declarative memorization. Stability, however, does not simply mirror plasticity: when learning a new procedure, updating QK retains over half of the stored declarative memory, while updating MLPs retains far less and updating VO loses roughly four-fifths. Targeted module reset-and-recovery experiments explain this asymmetry: plasticity tracks which module is best suited to a task's primary computation or storage, whereas stability depends on all ways the existing capability relies on that module, including access to information stored in other modules. Experiments on a realistic model and tasks broadly echo these asymmetric dynamics. Overall, our work shows that plasticity and stability depend on different aspects of a module's functional role, offering a task-dependent, module-level perspective on continual learning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
procedural memory, declarative memory, continual learning, plasticity, stability, Transformers

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_eTZKOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates module-level plasticity and stability in transformers from a continual learning perspective, focusing on declarative and procedural learning. The authors demonstrate that MLPs exhibit greater plasticity for declarative memorization, whereas attention projections are more effective at acquiring procedural skills. Importantly, they find that cross-task stability does not simply mirror plasticity: modules with similar learning capabilities can exhibit substantially different degrees of forgetting. Through targeted reset-and-recovery experiments, the authors suggest that declarative knowledge is preferentially stored in MLPs, while attention projections also contribute to accessing this knowledge. In particular, updating value-output (VO) projections can substantially impair declarative memory retention, despite their relatively limited role in storing such information. These findings are demonstrated through controlled synthetic experiments with a small transformer and further examined on realistic tasks using a pretrained language model.

Critical strengths
  1. Well-controlled experiments and interpretable findings. The paper employs a carefully controlled synthetic setting that enables systematic comparisons of plasticity and stability across transformer modules and task families. The experiments are repeated across three random seeds to improve the reliability of the results. While some observations are consistent with established intuitions about the functional roles of attention and MLP layers, the empirical evidence connecting these roles to different plasticity–stability dynamics is valuable. In particular, the asymmetric forgetting behaviors of QK and VO provide an interesting perspective on the distinction between information storage and access.
  2. Validation beyond synthetic settings. The authors complement their controlled synthetic experiments with evaluations on realistic procedural and declarative tasks using a pretrained language model. The observation of similar module-level asymmetries across these settings strengthens the empirical relevance of the findings beyond the synthetic testbed.
Critical weaknesses
  1. Limited evidence of generalizability to larger models. The controlled synthetic experiments are conducted on a 25M-parameter transformer, while the realistic experiments use only Qwen2.5-0.5B. Although the consistency of the findings across synthetic and realistic settings is encouraging, it remains unclear whether the observed module-level plasticity-stability dynamics generalize to larger, more capable pretrained LLMs. This limits the broader applicability of the conclusions, particularly given the potential differences in how knowledge and computational functions are distributed across model scales.

  2. Incomplete causal explanation of the observed stability asymmetry. While the reset-and-recovery experiments provide useful evidence about the functional roles of different transformer modules, they do not fully establish the causal mechanism underlying the observed forgetting patterns. In particular, the substantially lower declarative retention after VO updates compared with QK updates could arise either because VO plays a more critical role in accessing stored information or because its access-related functions are more vulnerable to procedural adaptation. The current experiments do not clearly distinguish these explanations, leaving the proposed connection between functional roles and stability only partially substantiated.

Questions for authors
  1. Interpretation of reverse-direction stability. While the paper provides evidence that updating VO can interfere with accessing declarative knowledge stored in MLPs, the reverse relationship is less clear. Figure 2(d) shows that updating MLPs also substantially degrades previously learned procedural skills, with sequence-level BFS accuracy dropping to zero across all configurations. How should this observation be reconciled with the claim that procedural skills are primarily implemented in attention projections? Does the relatively higher token-level accuracy after MLP updates indicate preservation of procedural knowledge, or merely less severe degradation compared with updating attention?

  2. Intra-task-family stability. The paper focuses on cross-task-family stability, measuring declarative memory retention after procedural learning and procedural skill retention after declarative learning. I wonder whether the observed module-level asymmetries also hold within the same task family. For example, how much of the previously learned BFS skill is retained after learning DFS, and how much existing declarative memory is preserved after acquiring new declarative information? Such experiments could help clarify whether the observed stability patterns are specific to interference between declarative and procedural learning or reflect more general module-level properties of continual learning.

Reviewer_VCS7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies plasticity and stability from the view of transformer modules. Based on cognitive distinctions between declarative and procedural memory, the authors finding QK/VO learn procedure memory more effectively, while MLPs memorize declarative sequences more effectively. The experiment result show that QK and VO reach nearly same DFS accuracy but retain 54% and 19% of old declarative memory. Similar patterns appear on Qwen2.5-0.5B. And the authors use reset-and-recovery experiments to interpret MLPs as storage and attention projections as access/computation modules.

Critical strengths

1.The synthetic experiment setup is well-designed to disentange sequence memorization from execution on unseen inputs.

2.Including parameter-matched MLP baselines (MLP-S and MLP-LoRA) is also good that rules out trainable parameter count as a confounding factor.

3.The contrast between QK and VO is convincing that clearly shows that modules with similar plasticity can behave very differently in terms of stability.

4.The core findings from the synthetic experiment setup carry be largely transfered to realistic benchmarks on Qwen2.5-0.5B.

Critical weaknesses

1.Switching to teacher-forced token accuracy after sequence-level BFS accuracy fall to zero does not show the model can still execute BFS exactly, and the score is likely inflated by predictable formatting tokens.

2.The reset-and-recovery intervention may measure the recoverability rather than storage/readout directly, as recovery may also depends on reset size (15.7M MLP, 4.2M QK/VO), optimization dynamics, or representation alignment. Moreover, it lacks the parameter-matched MLP-S control and relies on transient peak recovery before replay overfitting, so the storage/access account may be not very convinced.

3.No error bars or variance intervals are plotted despite running 3 seeds. And plotting retention–plasticity trade-offs across a grid of module-specific learning rates would make the comparison much more convincing. (Not must, I think it is resource-intensive.)

4.Testing only on a 25M synthetic model and Qwen2.5-0.5B leaves open whether these patterns scale to larger LLMs. And the realistic procedural tasks still rely on linguistic and factual knowledge (declarative memory). So this paper is a good work on interpretability, though it may lack a little bit of practical utility. I remain open to further discussion on this point.

Questions for authors

1.Do all results use three independent seeds, and can uncertainty be reported?

2.Does the memory recovery result hold with a parameter-matched MLP-S reset?

Reviewer_pZ4zOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how different transformer modules (QK, VO, MLP) exhibit different forgetting and plasticity behaviors throughout a continual learning cycle. The authors adopt a cognitive science perspective in their analysis and distinguish between declarative and procedural memory. Using this lens, the authors develop different scenarios to study transformers. The main empirical claim of this work seems to be that the modules most effective at learning new tasks are not necessarily those that best preserve previously learned information. Mainly, QK and VO learn a new procedure equally well, but updating VO erases far more stored memory. Further experiments investigate the forgetting mechanisms across a few different setups.

Critical strengths
  • The paper is overall well polished and structured.
  • The synthetic setting is reasonable and well motivated. I think it could be also extended to investigate further questions.
  • The QK vs. VO result is probably the most interesting empirical observation of this paper. Here two modules learn a new procedure equally well but differ substantially in how much they disrupt previously stored knowledge.
  • The cognitive science perspective provides a nice way to organize the analysis and distinguish between different types of learning in Transformers.
Critical weaknesses
  • In my opinion, the core insight of this work is largely known. The roles of MLPs in storing facts and attention in routing information are established. In the current version of the manuscript, the authors do not sufficiently establish the novelty of their work compared to Zhu et al. (2025), Kotha et al. (2024), and Zheng et al. (2025), which they cite. It seems that the QK/VO distinction is the main novel finding.

  • The stability ordering of MLP and VO changes between the synthetic and realistic settings too which complicates the proposed functional explanation.

  • The experiments focus on cross-task-family forgetting but do not investigate forgetting within the same family, such as BFS retention after learning DFS. I think investigating this setting is important to establish clearer insights from the empirical observations of this work.

  • The paper could communicate its main message more clearly. Beyond showing that plasticity and stability do not necessarily follow the same patterns, it is unclear what generalizable insight of this work.

  • In some places, the writing is too convoluted, even though the core message is somewhat simple. For instance, in the Introduction, the authors write: “plasticity tracks a module’s suitability for implementing a task’s primary mechanics, whereas stability additionally reflects auxiliary roles such as access to storage held elsewhere, rather than a strict task-modular division.” There are several examples of such convoluted constructions, and I think the paper would benefit from rewriting certain passages.

Questions for authors
  • What does the module-level view predict that a standard importance measures (e.g., Fisher information or similar information-theoretic measures) would not?

  • Does the QK vs. VO stability gap hold when each module is tuned with its own optimal learning rate? More generally, how sensitive are the observed differences in plasticity and stability to the choice of learning rate?

  • On the realistic tasks, what does QK-only tuning add over tuning all attention projections, as in Zhu et al. (2025)? In particular, does separating QK from VO provide a better trade-off between learning new tasks and preserving previously learned knowledge?

Reviewer_AvnMOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper is an interesting cognitive science based take on an well-known problem (learning new capabilities vs. retaining old ones). In synthetic testing, updating one module at a time shows that attention projections learn procedures best and MLPs learn facts best. The trend for stability is a bit different. QK and VO circuits learn a new procedure equally well, but updating VO erases far more stored memory. The paper produces an explanation. Procedures live mainly in attention, facts are stored in MLPs and attention also provides access to those facts. Thus, we can say updating VO disrupts memories it doesn’t store, whereas QK likely plays a less critical role in access (hence why updates leave its memory more intact). Fine-tuning Qwen2.5-0.5B on realistic tasks largely reproduces these patterns.

Critical strengths
  • I really enjoy how clean and well controlled the testing is, allowing for great degree of separation between the two types of learning and confidence in the results. The synthetic setup measures memorization in bits on random sequences and procedural skill by exact execution on held-out graphs. This removes the confounds of pretrained models (which is very common a confound in this field). There is lots of testing to rule out simpler explanations such as parameter count or capacity saturation. This makes the finding seem very sound.
  • The explanation is both clear and thought-provoking. Procedures live mainly in attention, facts in MLPs, and attention provides access to those facts. It is that clear. This explains conflicting results in past work and should prompt discussion about actionable future steps.
Critical weaknesses
  • The paper never pins down why VO harms memory far more than QK. It sort of leaves it at VO may have a more critical access role or may just be more vulnerable to the updates. This is, in the end, the central conceptual contribution of the paper.
Questions for authors
  • In the realistic experiments, were learning rates and training budgets tuned per module? Full fine-tuning underperformed single modules on some procedural tasks (take 18.0% vs. 29.4% on CRUXEval-O). This suggests the shared learning rate may favor some setups. Also, are the results single seed w/ untuned hyperparameters?
  • For suggestions of future work please see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.