- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper tests whether scaling up model size prevents plasticity loss in GPT-style language models. The authors train eight models (5M to 314M non-embedding parameters) on a continual multilingual cycling benchmark and periodically probe adaptability on held-out Vietnamese. All models eventually lose plasticity, though larger models delay the onset. Fitting a power law to these onset points, the authors argue that plasticity scales sublinearly with size—meaning scale alone won't solve the problem—and show similar degradation under stationary multilingual training.
Critical strengths
1. Realistic benchmark.
Unlike prior work on small vision models or synthetic tasks, the CulturaX multilingual cycling setup with a held-out Vietnamese probe (Section 4.1) is a clean way to test long-horizon plasticity in natural language.
2. Controlled comparison.
Fixing the aspect ratio and head dimension across eight scales isolates model size well, and adding stationary training (Section 5.2) rules out abrupt task switches as the only cause.
Critical weaknesses
- Sublinear scaling is borderline. In Table 5, the fitted log-log exponent is b = 0.8269 ± 0.1027, so linear scaling (b = 1) is within ~1.7 standard errors. In-sample R² and RMSE are also nearly identical to the linear fit (0.8969 vs. 0.8914), and several models share the exact same onset in Table 4 (e.g., T = 6 for 5M/12M; T = 54 for 53M/83M). That's a bit thin for a strong sublinear claim.
- No error bars on onset. Onsets are picked from a 3-point moving average over single-seed runs (Appendix B). Without seed or probing variance, it's hard to tell how stable these points are—especially for 314M, where the curve in Figure 6 is very flat around the minimum (−12.4% to −13.2%).
- Unclear generalization at scale. Everything is ≤ 314M parameters and tested only on Vietnamese adaptation. Extrapolating Equation (2) to 1B–7B models (Section 5.1) spans orders of magnitude, and it's unclear if the trend holds for larger LLMs or other adaptation tasks.
Questions for authors
- How much variance is there across random seeds or probing runs, especially for the flat 314M curve?
- How sensitive is the exponent b to the smoothing window (3) and the choice of extended probing languages (Chinese, German, Russian)?
- Given b = 0.8269 ± 0.1027 and similar in-sample R² to the linear model, what really rules out near-linear scaling?
- How wide are the confidence intervals when extrapolating Equation (2) to 1B and 7B models in Section 5.1?