acceptodds
Under review as a conference paper at ICLR 2027

Address Migration for Replay-Free Continual Learning with Sparse Routed Memories

Abstract

Updating a representation can invalidate the coordinates used to retrieve previously learned memories. We study this problem in a continual learner that stores sparse routing prototypes and local experts, but retains no past examples or feature exemplars. Our approach, address migration, fits a linear map from new activations to the previous memory coordinates using paired evaluations of current unlabeled inputs. A wake–sleep schedule separates supervised expert updates from unsupervised projection updates, and a distance-profile regularizer limits geometric distortion during sleep. We characterize exact recovery for linear drift, separate finite-sample error from irreducible approximation error, bound changes in top- codes by the distribution of alignment error relative to coordinate margins, and show that cosine nearest-class-mean classification in code space is a special case of the architecture. On Split CIFAR-100 with a frontend pretrained on all classes without labels, migration with guided sleep achieves final accuracy, compared with for unguided migration and for the frozen configuration (three seeds), recovering about 80% of the accuracy cost of moving the projection and bringing a mobile representation within one point of the frozen configuration. Increasing routing capacity improves retention, whereas routing selectivity alone cannot compensate for weak per-task learning. The results establish coordinate alignment as a mechanism for keeping sparse routed memories addressable while the representation changes.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, catastrophic forgetting, replay-free learning, sparse distributed representations, wake-sleep cycle, address migration, guided sleep, mixture-of-experts routing, self-supervised features

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_kkBfOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how to preserve access to previously learned knowledge when a representation changes in exemplar-free continual learning. It proposes address migration, which learns a linear map from updated activations to previous memory coordinates using current unlabeled inputs, combined with a wake–sleep schedule and geometric regularization. Theoretical analysis characterizes recovery conditions and routing stability. Experiments mainly on Split CIFAR-100 show that guided migration mitigates accuracy loss caused by projection updates, although it does not outperform the frozen-projection configuration.

Critical strengths

The paper addresses a relevant failure mechanism in continual learning: representation changes can invalidate routing to previously learned experts. It proposes an exemplar-free alignment approach supported by theoretical analysis of recovery conditions and routing stability.

Critical weaknesses
  1. The experiments support address migration as a partial repair mechanism for projection drift, but do not yet establish a practical advantage for continual learning. For example, as the SSL frontend is trained beforehand on the entire CIFAR-100 training split without labels; it therefore has access to future classes’ input distribution. Pretraining on all future classes limits the applicability of this work to the continual-learning settings where future data is unavailable.
  2. Frozen projection obtains 0.281+- 0.010 final accuracy, which is still higher than 0.273 +-0.011 guided migration proposed by the paper.
  3. Three seeds provide limited evidence for the reported effect sizes.
  4. The evaluation is limited to Split CIFAR-100 and domain-incremental MNIST, providing insufficient evidence of the method’s generalizability. More comprehensive ablation studies are needed.
  5. The paper’s presentation requires substantial clarification, for example, the abstract introduces numerous architectural terms before explaining their relationships.
Questions for authors

Please address the comments listed in the critical weaknesses.

Reviewer_5yTXOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies coordinate drift in a replay-free learner with a frozen frontend, a trainable projection, sparse top-k codes, and routed experts. It proposes a linear migration map estimated from current-task activations and a geometric regularizer for projection updates. The analysis characterizes recoverability and code stability. Validation mainly uses Split CIFAR-100, with additional MNIST editing experiments. Guided migration improves final accuracy from 23.9% to 27.3% in the main configuration, compared with 28.1% for freezing the projection.

Critical strengths

The analysis explicitly distinguishes fitting an alignment on current inputs from preserving representations of historical classes. The recoverability conditions and coverage counterexample clarify when migration can work and when a low alignment residual is insufficient. This is a useful qualification of the mechanism's applicability.

Critical weaknesses
  1. The practical value of the representation updates is not established. Table 1 reports 28.1 ± 1.0% accuracy for the frozen projection, 23.9 ± 0.1% for sleep with migration, and 27.3 ± 1.1% after adding guidance. The method therefore recovers part of the loss introduced by its own update procedure, without demonstrating an advantage over freezing the projection. The 20-task experiment also remains below the frozen control. Moreover, guidance reduces projection drift itself, and the main configuration lacks a complete migration/guidance factorial ablation. It remains unclear how much of the recovery requires migration and how much follows from making the representation change less. A mechanism study need not achieve SOTA, but the paper should establish why the adaptation it enables is useful.

  2. The experiments do not establish relevance or value relative to existing methods. The main external references are a dense MLP, EWC, and joint training. There are no experimental comparisons with NCM, FeCAM, RanPAC, or the closest drift-compensation approaches, despite their discussion in the paper. The capacity comparison is also problematic: depth 11 permits approximately 182M expert parameters, versus approximately 0.58M for the dense MLP. Activating one expert does not eliminate the storage cost of the others. Comparisons under matched frontend, data-access, and resource conditions are needed to establish the value of the proposed architecture. This is particularly important because the frontend has already seen all CIFAR-100 classes during unlabeled pretraining and only a linear projection moves. The reported evidence does not establish that the findings explain or improve an important limitation of existing continual-learning methods.

  3. Figure 2(c) also contains overlapping annotations and unusual visual artifacts. The 0.042 annotation overlaps the error bar and gap arrow, while the marks near the 0.008 callout form an unusual crossed shape. These clustered visual patterns make it difficult to distinguish uncertainty indicators from gap annotations and raise concerns about whether the figure was adequately checked. Although the annotated gaps agree with Table 1, the central comparison remains unnecessarily difficult to inspect. The authors should provide a clean plot with clearly separated annotations and error bars.

Questions for authors
  1. Clarification: What useful adaptation do the existing results demonstrate that cannot be obtained by freezing the projection? Is there evidence of improved acquisition or transfer that offsets the additional update cost?

  2. Essential comparative validation: Under the same frontend and pretraining access, how does the approach compare with established frozen-feature classifiers and the closest drift-compensation methods? Please report actual persistent memory and training costs so that these comparisons can be interpreted fairly.

  3. Essential ablation: At depth 11 with SSL-1000, what happens when migration and guidance are independently enabled or disabled? This would determine whether the main improvement requires coordinate alignment or primarily follows from suppressing drift.

Reviewer_YTEZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper describes a method that helps update prototypes and experts in a continuous learning environment without the need for examples from previous tasks. Starting with a pre-trained model trained on all the data in a self-supervised manner, the method then divides the dataset into 10 tasks, which it subsequently uses to evaluate the method. Results show that keeping the model frozen (Evaluation A) performs better than using the method, but they also show that using what is called “guided sleep” helps achieve better performance than using Sleep and Migration alone.

Critical strengths

In current CL, it is common to train experts for specific tasks, which are then selected during inference, either through learned routers or some distance metric. Often, selection is crucial for the method to work properly; however, it is not addressed during training. From what I understand of the method, this paper aims to move toward better selection of experts and prototypes, which I consider a very interesting line of research that could have a major impact if it is explained correctly and evaluated properly.

Critical weaknesses

The way the paper is written makes it very difficult to follow the main point. There is no clear definition of the problem or a clear rationale for the problem the paper aims to solve. The method is poorly explained, which makes it difficult to understand the contributions mentioned in the introduction. Unfortunately, this poor writing is not limited to sentences or paragraphs that could be improved; there are words that are misused or that are commonly used in other fields of research and have been carried over without consideration for the vocabulary typically used in this field; this makes reading the document difficult. Unfortunately, it also lacks a thorough evaluation and ablation of the method.

Questions for authors
  • As mentioned in the first paragraph of the introduction, using a frozen pretrained model as a baseline is a good replay-free baseline. It even shows better results than the proposed methods. Can you explain the contribution of the proposed method, and when it can be more useful to use it over the baseline?
  • There are some terms that I didn’t fully understand after reading the document. It may be due to my lack of understanding of some other topics, can you explain them differently:
    • Sparse routed memory
    • Frontend model
    • Code space
  • What does the colour of the blocks represent in Figure 1?
  • In the experimental setup, it is mentioned that 2048 experts were used. Why so many? What does an expert represent in this context?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.