acceptodds
Withdrawn from ICLR 2027

Tip-of-the-Tongue Effect: Latent Geometric Structures Resist Catastrophic Forgetting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The phenomenon of catastrophic forgetting, where models trained on successive tasks rapidly lose the ability to perform early-learned tasks, has largely been characterized by behavioral metrics such as accuracy and loss, leaving unanswered the question of whether underlying task knowledge was permanently overwritten or merely suppressed. We investigate whether representational similarity metrics can detect persistent knowledge by tracking the internal geometry of speech and vision models across abrupt task transitions, comparing against single-task baseline models to capture task-specific representational geometry. We find that this task-specific geometry is preserved long after task accuracy and loss decay to chance, and is retained more strongly by larger models despite equivalent chance-level performance. We further find that models with task-specific geometry relearn the original task 8.3%–18.0% faster on average than matched controls without prior exposure to that task. In sum, our findings show that behavioral metrics alone do not capture the true scope of model retention, with implications for how catastrophic forgetting should be evaluated moving forward.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual learning, catastrophic forgetting, representational geometry, model scale, savings effect

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VY2eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether task knowledge persists after catastrophic forgetting. Using RSA and CKA, the authors compare representations of sequentially trained models with single-task controls on CIFAR-100 and speech recognition tasks. They find that task-specific representational structure remains detectable after accuracy drops to chance, with stronger traces in larger models.

Critical strengths

The paper examines catastrophic forgetting beyond accuracy and loss, offering a useful perspective on whether task-related information persists after performance deteriorates. The experiments cover both vision and speech, and the relearning results provide behavioral evidence that complements the representational analysis.

Critical weaknesses
  1. The main concern is that representational similarity does not establish that task-specific knowledge remains accessible. The relearning results are suggestive, but do not show that the retained geometry causes faster relearning.
  2. The evidence for a model-size effect is also less conclusive than the paper suggests. Some speech results have high variance, and the size trend is not consistent under median-based analysis. These findings warrant a more cautious interpretation of the general conclusions.
Questions for authors
  1. How do the authors distinguish task-specific retained information from generic representational similarity? Can probing experiments or targeted interventions demonstrate that the measured traces contain information useful for solving the original task?
  2. Can the authors isolate the effect of model capacity from architectural differences? Given the limited seeds and the sensitivity of relearning results to aggregation, do the reported capacity and relearning trends remain consistent under alternative statistical analyses?
Reviewer_VbWfOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper argues that catastrophic forgetting is often measured solely in terms of performance degradation, but that this does not imply that the information represented in the model is actually forgotten or removed. Quite the contrary, it claims that while performance metrics might show catastrophic loss of performance, the representation of the model still preserves information about the old tasks, and that this information offers faster relearning of the old tasks. The paper has an empirical methodology analyzing model representation similarities with two different metrics and their evolution through time, in different parts of the models, and for different data modalities.

Critical strengths

The paper reads well, the language is clear, and so is its presentation. Its scope, goal, and motivation are conveyed clearly. I believe that a better mechanistic understanding of forgetting is a timely research objective, and thus the broad research question is interesting and important.

Critical weaknesses

I believe that while the spirit of the paper is appreciable, it has major weaknesses that prevent me from recommending acceptance. These weaknesses, elaborated below, are particularly present in terms of (i) novelty, (ii) strength of the evidence and its interpretation, (iii) experimental soundness.

Novelty

The paper overlaps both in methodology and results with multiple prior works that are ignored or downplayed. For example, Ramasesh et al. [1] already perform layerwise CKA analysis, studying task semantics and their consequences on performance. Mirzadeh et al. [2] and Wenger et al [3] study forgetting and the role of model width, and Graldi et al. [4] directly connect scale, representation dynamics, and feature evolution via CKA measures. Most notably, however, are the similarities with Davari [5]: this prominent prior work already shows how internal representation influences performance-forgetting. A more diligent discussion would clarify the paper’s contribution, and — I am afraid — highlight the limited novelty proposed in the work.

Evidence and Interpretation

The main limitation of the results to me is the recurring comparison of similarity metrics across different setups, dimensions, and absolute values. In particular:

  • It is very hard to interpret and judge if a “5% similarity survives” is actually a lot or not. Especially as the results are mostly aggregates across layers, disregarding the dimensionality of the layers, and thus it is hard to compare how the actual representation is evolving.
  • Figure 2 is misleading in depicting accuracy and RSA on the same 0-1 axis as if the two quantities would be meaningfully related. It is misleading as it invites comparing the two absolute values while this is not possible. If one were to compare the relative RSA and relative accuracies from the baselines, I think the results would be even less convincing than they already are, and the surviving similarities even more questionable and hard to interpret.
  • I don't find solid enough results and insights to interpret the surviving similarity as “the representation survives”: it could just be background noise fitted spuriously that hardly provides any meaningful and semantic representation in the model.
  • For the reason above, I think it is very hard to interpret Figure 3: each block might have a different dimension, and therefore it is unclear to me if and how one should compare similarities across layers without confounding the similarity with the actual dimensionality varying. I think one should for sure compare mainly the relative scores (which is only done at times), but even then a more thorough (possibly statistical) analysis of the role of dimensionality would be needed. Recent work on measuring geometric change as a proxy for the usefulness of model representations (Göring et al. [6]) further questions how CKA-style metrics can be interpreted in this context.

Experiments

At least some of my previous comments above could be dismissed with a strong, fair, and interventional experimental plan. This is however missing. Indeed, the only experiment proposed is the “relearning speed”, which, however presents significant limitations

  • Firstly, it is completely arbitrary to set the threshold at 70% (note that in a plot a wrong label with 75% appears); unclear if the results would change (and how) when this threshold is higher. For example: would the advantage also reflect in final saturation performance?
  • Secondly, but most importantly, the experimental setup does not guarantee a fair evaluation of the various baselines: hyperparameters are not tuned separately for each run, but rather kept constant, and therefore it is impossible to claim whether the shown relearning results are due to surviving information, or rather to better/worse tuning. The bare minimum would be to separately tune the runs on a common reasonable LR grid.
  • Even with these further results, however, I am afraid the current experiments on re-learning would not be enough to convincingly show a cause-and-effect relationship between surviving representation and forgetting. Examples of stronger evidence are Ramasesh [1] and Davari [5].

References

[1] Vinay Venkatesh Ramasesh, Ethan Dyer, and Maithra Raghu. Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics. In International Conference on Learning Representations (ICLR), 2021.

[2] Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Huiyi Hu, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar. Wide Neural Networks Forget Less Catastrophically. In Proceedings of the 39th International Conference on Machine Learning, PMLR 162:15699–15717, 2022.

[3] Jonathan Wenger, Felix Dangel, and Agustinus Kristiadi. On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspective. arXiv preprint arXiv:2310.00137v2, 2024.

[4] Jacopo Graldi, Alessandro Breccia, Giulia Lanzillotta, Thomas Hofmann, and Lorenzo Noci. The Importance of Being Lazy: Scaling Limits of Continual Learning. In Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:20326–20367, 2025.

[5] MohammadReza Davari, Nader Asadi, Sudhir Mudur, Rahaf Aljundi, and Eugene Belilovsky. Probing Representation Forgetting in Supervised and Unsupervised Continual Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16712–16721, 2022.

[6] Niclas Alexander Göring, Charles London, Abdurrahman Hadi Erturk, Chris Mingard, Yoonsoo Nam, and Ard A. Louis. Disentangling Feature Learning from Generalization in Neural Networks. In High-dimensional Learning Dynamics (HiLD) Workshop at ICML, 2025.

Questions for authors

Did you consider using only CKA directly, which is more standard in the literature? What exactly are the benefits of RSA over CKA in this case?

Reviewer_W2znOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This work investigated representation forgetting in continual learning by analyzing representational similarity across task stimuli before and after forgetting. The authors found that the representation similarity remains above the control level even after the readout performance becomes the chance level, and this effect is more prominent in a large model.

The topic addressed in the manuscript, how forgetting is manifested at the level of hidden layer representation, is a potentially important question that has only been partially resolved.

Critical strengths

None

Critical weaknesses

The main conclusion is not well supported by their numerical experiments. While the authors claimed that RSA is preserved long after behavioral accuracy goes to chance level, Figure 2A shows that in ResNet-101, RSA quickly approaches the control level after 6K steps of training, while the behavioral readout remains significantly above the chance level up to 12K steps of training. Although RSA remains above the control level, the gain is very small, and thus, its effective utility is unclear. Fig. 5 is plotted for very small step sizes (<2K), which is outside the “tip-of-the-tongue regime” they defined.

The novelty and significance of the presented work are limited. As noted in the related work section, forgetting at the representation level has been studied previously, which found that linear probing from a hidden layer achieves better readout performance than the output readout in continual learning (Davari et al., 2022). CKA-based analysis of representation forgetting was also studied previously (Ramasesh et al., 2020; cited below), making the conceptual and methodological significance unclear.

Ramasesh, Vinay V., Ethan Dyer, and Maithra Raghu. "Anatomy of catastrophic forgetting: Hidden representations and task semantics." arXiv preprint arXiv:2007.07400 (2020).

Questions for authors

Both Task 1 Accuracy and Task 1 RSA at the beginning (training step = 0) in Fig. 2A are different from the chance level. Are the results estimated after a few updates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.