Tip-of-the-Tongue Effect: Latent Geometric Structures Resist Catastrophic Forgetting

The phenomenon of catastrophic forgetting, where models trained on successive tasks rapidly lose the ability to perform early-learned tasks, has largely been characterized by behavioral metrics such as accuracy and loss, leaving unanswered the question of whether underlying task knowledge was permanently overwritten or merely suppressed. We investigate whether representational similarity metrics can detect persistent knowledge by tracking the internal geometry of speech and vision models across abrupt task transitions, comparing against single-task baseline models to capture task-specific representational geometry. We find that this task-specific geometry is preserved long after task accuracy and loss decay to chance, and is retained more strongly by larger models despite equivalent chance-level performance. We further find that models with task-specific geometry relearn the original task 8.3%–18.0% faster on average than matched controls without prior exposure to that task. In sum, our findings show that behavioral metrics alone do not capture the true scope of model retention, with implications for how catastrophic forgetting should be evaluated moving forward.