acceptodds
Under review as a conference paper at ICLR 2027

Can Scale Overcome Plasticity Loss in Large Language Models?

Abstract

The loss of plasticity – the ability of a network to learn new information after having already learned older information – is a fundamental challenge in creating artificial neural networks capable of continual learning. Although this phenomenon has been known for decades, it has mostly been studied in older, relatively small architectures and rarely in natural-language domains. To determine whether loss of plasticity remains a problem in the modern transformer-based LLM paradigm, we study plasticity loss in GPT-style Transformer models trained on a multilingual continual learning problem. Consistent with prior work, we find evidence of plasticity loss across models ranging from 5M to 314M non-embedding parameters, as measured by deterioration on a held-out Vietnamese probing task. We further find that the onset of plasticity loss follows a predictable scaling law, growing sublinearly with model size. These results suggest that larger models may delay the measurable effects of plasticity loss, but that increasing parameter count alone is likely to be insufficient to completely prevent it. We also find evidence of plasticity loss under stationary multilingual training, challenging the view that the phenomenon is exclusive to continual learning with abrupt task changes. Overall, our results suggest that even large Transformer language models trained on natural-language will eventually lose the ability to efficiently adapt to new data after sufficiently long training, in both continual and stationary settings.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Deep Learning, Continual Learning, Loss of Plasticity, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GxgYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tests whether scaling up model size prevents plasticity loss in GPT-style language models. The authors train eight models (5M to 314M non-embedding parameters) on a continual multilingual cycling benchmark and periodically probe adaptability on held-out Vietnamese. All models eventually lose plasticity, though larger models delay the onset. Fitting a power law to these onset points, the authors argue that plasticity scales sublinearly with size—meaning scale alone won't solve the problem—and show similar degradation under stationary multilingual training.

Critical strengths

1. Realistic benchmark.

Unlike prior work on small vision models or synthetic tasks, the CulturaX multilingual cycling setup with a held-out Vietnamese probe (Section 4.1) is a clean way to test long-horizon plasticity in natural language.

2. Controlled comparison.

Fixing the aspect ratio and head dimension across eight scales isolates model size well, and adding stationary training (Section 5.2) rules out abrupt task switches as the only cause.

Critical weaknesses
  1. Sublinear scaling is borderline. In Table 5, the fitted log-log exponent is b = 0.8269 ± 0.1027, so linear scaling (b = 1) is within ~1.7 standard errors. In-sample R² and RMSE are also nearly identical to the linear fit (0.8969 vs. 0.8914), and several models share the exact same onset in Table 4 (e.g., T = 6 for 5M/12M; T = 54 for 53M/83M). That's a bit thin for a strong sublinear claim.
  2. No error bars on onset. Onsets are picked from a 3-point moving average over single-seed runs (Appendix B). Without seed or probing variance, it's hard to tell how stable these points are—especially for 314M, where the curve in Figure 6 is very flat around the minimum (−12.4% to −13.2%).
  3. Unclear generalization at scale. Everything is ≤ 314M parameters and tested only on Vietnamese adaptation. Extrapolating Equation (2) to 1B–7B models (Section 5.1) spans orders of magnitude, and it's unclear if the trend holds for larger LLMs or other adaptation tasks.
Questions for authors
  1. How much variance is there across random seeds or probing runs, especially for the flat 314M curve?
  2. How sensitive is the exponent b to the smoothing window (3) and the choice of extended probing languages (Chinese, German, Russian)?
  3. Given b = 0.8269 ± 0.1027 and similar in-sample R² to the linear model, what really rules out near-linear scaling?
  4. How wide are the confidence intervals when extrapolating Equation (2) to 1B and 7B models in Section 5.1?
Reviewer_wkjUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors of this paper test whether GPT-style language models (5M–314M non-embedding parameters) lose plasticity, meaning their ability to learn new data. Models are trained on repeating cycles of 8 languages. After each cycle, a copy of the model is trained on held-out Vietnamese, and learning speed is measured by the area under the validation-loss curve (AUC).

Critical strengths
  • Asks an important, timely question in a realistic natural-language setting rather than on synthetic or vision tasks.
  • Careful scaling setup: fixed aspect ratio and head dimension, tuned learning rates, and optimizer resets at each task.
  • Large compute budget, with up to ~48 cycles (about 1.9T tokens).
  • The stationary-training result and the per-language transfer analysis (Table 3) are useful.
Critical weaknesses

Small effects, single seed.

  • The AUC changes are around 0.5–1.5%, with no seeds or error bars. Table 3 shows that the language trained just before probing shifts AUC by a similar 1–2%, so noise could drive much of the trend.

Possible confounds.

  • The learning rate is held constant with no decay.
  • Weights grow under LayerNorm, which lowers the effective learning rate. The authors mention this but do not control for it.
  • The number of tokens per parameter differs hugely across model sizes (the 5M model sees ~1000 tokens per parameter per task), so the result may reflect overtraining rather than model scale.

Overclaiming.

  • Calling plasticity loss “a fundamental and extremely general property of neural network training” goes beyond the evidence. The architecture (absolute positional embeddings, LayerNorm, GeLU) is also older than modern LLMs (RoPE, RMSNorm, SwiGLU), and the largest model has 314M parameters.

Necessity and Informativeness.

  • This is my main concern. Loss of plasticity is well studied, and the finding that larger networks lose plasticity later is already established, as the authors acknowledge. Degraded adaptability after long training has also been shown for language models (Springer et al., 2025). While I appreciate the effort to confirm this in GPT-style models, confirming a known phenomenon in a new setting adds limited insight on its own. The genuinely new contribution is the scaling law for the onset of plasticity loss, but it is also the least supported: it is fit to 8 points from a single seed, and the exponent (0.83 ± 0.10) is not clearly below 1, so the central claim that scale alone cannot overcome plasticity loss is not established.

Springer et al., 2025: J. M. Springer, S. Goyal, K. Wen, T. Kumar, X. Yue, S. Malladi, G. Neubig, A. Raghunathan. Overtrained Language Models Are Harder to Fine-Tune. ICML 2025.

Questions for authors

Please see the weaknesses.

Reviewer_FmknOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether increasing model size delays plasticity loss in GPT-style language models. Models with 5M to 314M non-embedding parameters undergo repeated cycles of pretraining on eight languages. At selected checkpoints, copies of the models are trained on held-out Vietnamese data, and adaptation is assessed using the area under the validation-loss curve.

The authors observe that probing performance eventually deteriorates and that larger models generally deteriorate later. They fit a power law relating model size to the estimated onset of deterioration. Additional experiments show similar behavior under stationary multilingual training for three smaller models. Analyses of parameter magnitudes, dormant units, and attention-weight spectra examine possible correlates of plasticity loss.

Critical strengths
  • The paper provides a substantial empirical investigation of adaptation after extended natural-language training. Evaluating eight model sizes over long training horizons goes beyond demonstrating plasticity loss in a single small model or a synthetic task. The copied-checkpoint protocol keeps Vietnamese probe updates out of the pretraining trajectory, and optimizer resets at task boundaries address stale optimizer state as one possible explanation. Keeping attention-head dimension fixed and approximately matching width-to-depth ratios also makes the scale comparison more controlled. The resulting observation, that larger models generally retain favorable probing performance for longer is interesting and relevant to continual LLM training.

  • The stationary experiments show that abrupt language switches are not necessary for deterioration in this setup, although these experiments cover only the three smallest models. The network diagnostics also provide useful negative evidence: weight growth, dormant units, and spectral concentration do not consistently track the onset of deterioration. The paper appropriately treats these as correlates rather than established mechanisms. Overall, the main experimental design is clearly presented, and the distinction between forgetting and plasticity loss is explained accessibly.

Critical weaknesses
  • Table 5 reports an exponent of 0.8269 with standard error 0.1. A conventional 95% regression interval with six residual degrees of freedom is approximately [0.576, 1.078], which includes linear scaling. The point estimate suggests sublinearity, but the evidence does not justify treating diminishing returns as an established result.

  • The estimated onset is sensitive to analysis choices, without quantified uncertainty. Appendix B identifies onset using the minimum of a three-point moving average, after adding probes near previously observed minima and selecting preceding languages with similar transfer behavior. These choices can affect the estimated transition. The paper does not report sensitivity to smoothing or onset definitions, nor variability across independent training seeds. This is particularly important for the largest models, where deterioration is shallow and identifying a precise minimum is difficult.

  • The scaling interpretation depends strongly on the definition of model size. Using the total parameter counts in Table 1 and the same onset values in Table 4, my log-log refit gives an exponent of approximately 1.5, compared with 0.83 using non-embedding parameters. Non-embedding count is a reasonable measure of Transformer-stack capacity, but embeddings are trained and constitute a substantial fraction of these models. The broader resource-efficiency interpretation therefore requires sensitivity analyses using total parameters and training compute.

  • Validation-loss AUC does not separate starting performance from adaptation ability. Absolute AUC combines initial Vietnamese loss, subsequent improvement, and final loss. A later checkpoint could have higher AUC because it starts further from the target distribution, even if its learning rate of improvement is unchanged. Normalizing by first-cycle AUC or Vietnamese-only training does not resolve this ambiguity. Representative probe-learning curves, initial and final losses, and improvement relative to initialization would clarify what deteriorates.

  • All adaptation measurements use Vietnamese, and continual pretraining follows one fixed language order. Table 3 shows that the preceding language materially affects Vietnamese probing performance. The stationary experiments broaden the training setting, but retain the same probe and cover only the three smallest models. These results support delayed deterioration in the tested setting; they do not establish that scaling cannot prevent plasticity loss across larger models, other target distributions, or different training protocols.

Questions for authors

-Given that the reported exponent’s conventional 95% interval includes 1, what evidence supports sublinear rather than approximately linear scaling? How does the exponent change across smoothing windows, onset definitions, and repeated training seeds?

  • How should the diminishing-returns claim be interpreted given the different fit obtained using total parameters? Can you report sensitivity to total parameter count and training compute alongside non-embedding parameter count?

  • Do later checkpoints exhibit slower loss reduction after accounting for their initial Vietnamese loss? Does the deterioration persist when probe learning rates and budgets are varied, and on another held-out language?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.