acceptodds
Under review as a conference paper at ICLR 2027

Matryoshka Continual Learning: Nested Representations for Continual Learning

Abstract

Continual learners are prone to catastrophic forgetting, where learning new tasks overwrites the representations that earlier tasks depend on. To address this, we propose Matryoshka Continual Learning (MCL), which structures a single embedding as a sequence of nested sub-embeddings like Matryoshka dolls. The first few dimensions constitute a compact representation that encodes coarse features and is trained to support the classification task on its own. Each wider prefix contains the preceding one and extends it with progressively finer, constructive detail, yielding a coarse-to-fine hierarchy within a single vector. Embedding is learned through a teacher–student setup, where the teacher has access to privileged knowledge, forcing a student with the same architecture to follow its lead without access to that knowledge. Because MCL changes the representation rather than the learning rule, it plugs into existing CL methods. We evaluate MCL against 9 baselines on Seq-CIFAR-100, Seq-Tiny-ImageNet, and the Robotic Tactile Internet (RTI) datasets under Task-Incremental Learning (Task-IL) and Class-Incremental Learning (Class-IL) settings. In Task-IL, MCL attains 87.29% accuracy on Seq-CIFAR-100 and 77.59% on Seq-Tiny-ImageNet, 0.32 and 1.97 points above the second-best method, respectively. In Class-IL, MCL ranks the second-best method in accuracy on both benchmarks, while achieving the highest plasticity–stability score.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Nested Learning, Class Incremental Learning, Self Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WtwrOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Matryoshka Continual Learning (MCL), an exemplar-free continual learning framework that adapts Matryoshka Representation Learning to mitigate catastrophic forgetting. The method jointly trains multiple nested feature prefixes using shared classifier weights. During sequential learning, a frozen model from the previous task serves as a teacher, and knowledge distillation is applied at each feature width to preserve old-class predictions. For Class-IL, the framework additionally incorporates local cross-entropy, cosine classification, weight alignment, and class-mean prototype fusion.

Critical strengths

The proposed method provides an exemplar-free solution without introducing task-specific network expansion or additional expert modules. Its fixed-capacity architecture and relatively straightforward implementation make it potentially practical for resource-constrained continual learning scenarios.

Critical weaknesses
  1. The proposed MCL framework largely combines existing Matryoshka Representation Learning (MRL) with multi-width LwF, without introducing substantial new algorithmic mechanisms.

  2. Multi-level and selective feature distillation have been extensively explored in continual learning (e.g., PODNet, AFC, and MDPCR). The paper fails to adequately distinguish its approach from these closely related methods.

  3. The assumption that higher feature reach leads to greater stability is not sufficiently justified. Moreover, the paper lacks critical ablation studies and feature drift analyses to validate this claim.

Questions for authors

See above

Reviewer_24bVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses Matryoshka Representation Learning (MRL) for continual learning, Matryoshka Continual Learning (MCL). The main idea is inherited from MRL, which is to define hidden representations in a neural network into nested prefixes that encode features from coarse to fine, but to make it work, several ideas are introduced: MCL trains every prefix on the currents task and distills the previous class predictions from a frozen checkpoint at every width. The distillation weight follows the reach of each prefix, and therefore anchoring narrower prefixes stronger. MCL is compared with 9 buffer-free continual learning methods across Task-IL and Class-IL using 1 model (ResNet-18) on 3 datasets (Seq-CIFAR-100, Seq-Tiny-ImageNet, and RTI) showing mixed results.

Critical strengths
  • MRL is a natural fit for continual learning and its adaptation with distillation is a sensible decision.
  • Good analysis in Figures 4 and 5 of MCL.
Critical weaknesses
  • The main weakness is in the experiments and the lack of variety in the models studied. The experiments only use a ResNet-18 which is an overly simple model but is also one model. Given that this method is a representation method, several models should be studied.
  • The lack of generality to transformer architectures is also a strong limitation.
  • The results are mixed at best, "MCL ranks first or second in 13 of the 24 columns", and only one of those are on RTI. I would conclude that SVN outperforms MCL albeit at an increased computational cost but MCL is also not computational the most efficient either (it terms of training time) compared to baselines other than MCL.
Questions for authors
  • "We call this scheme Self-Distillation (SD)." What is the difference between Self-Distillation and functional regularisation continual learning methods (such as Learning without Forgetting (LwF))?
  • "Continual learning methods typically apply a single mechanism uniformly to the feature vector shared across all coordinates. MCL instead imposes an order on the coordinates, so that the degree of stability varies along the vector." What is the difference between this and parameter regularisation continual learning methods (e.g., EWC or SI)?
  • Line 307 "However, because they freeze subnetworks, they are not scalable." and Line 318 "SNV maintains the highest continual accuracy". Does the latter statement discredit the former?
Reviewer_LW9nOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Matryoshka Continual Learning, a method for exemplar-free class-incremental learning based on Matryoshka representations. The main idea is to learn nested feature representations with different levels of stability and plasticity. By applying different knowledge distillation strengths to different feature prefixes, the method aims to preserve previously learned knowledge in lower-dimensional representations while allowing higher-dimensional representations to adapt to new tasks.

Critical strengths
  1. The paper addresses an important problem in continual learning, namely balancing stability and plasticity without storing exemplars from previous tasks.
  2. The idea of exploiting nested representations to preserve previous knowledge while accommodating new knowledge is intuitive and interesting. Applying Matryoshka representations to continual learning is a reasonable approach to this problem.
Critical weaknesses
  1. My main concern is the limited technical novelty. The proposed method is largely based on two existing ideas: Matryoshka Representation Learning (Kusupati et al. 2022) and knowledge distillation for continual learning, such as LwF. The main contribution appears to be applying different distillation strengths to nested feature representations. While this is a reasonable design, I am not fully convinced that it constitutes a sufficiently substantial methodological contribution.

  2. The individual contributions of the proposed components are not sufficiently established. In particular, it is important to distinguish the benefit of using Matryoshka representations from that of applying different distillation strengths across feature prefixes. A comparison between MRL with uniform knowledge distillation and MRL with the proposed variable-strength distillation would be especially useful. Without such evidence, it is difficult to determine whether the proposed distillation strategy is essential to the performance improvement.

  3. The central motivation is not sufficiently supported by direct evidence. The paper assumes that shorter feature prefixes can preserve stable knowledge while additional dimensions provide greater plasticity for learning new tasks. However, applying stronger distillation to shorter prefixes does not necessarily guarantee such a separation of roles. More direct analysis would be helpful to demonstrate that different parts of the representation actually exhibit the intended stability and plasticity.

  4. The empirical performance is not consistently competitive with the strongest baseline. In Class-IL, which is a particularly important setting for evaluating catastrophic forgetting, MCL underperforms SNV in all three benchmarks.

Questions for authors

Please refer to the weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.