acceptodds
Withdrawn from ICLR 2027

Beyond Pointwise Replay: Pathwise Knowledge Consolidation for Continual Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continual learning (CL) aims to acquire new knowledge without forgetting what was learned before. Replay-based methods mitigate forgetting by revisiting a small memory buffer, but they supervise each replayed sample only at isolated targets, its label or its stored logits. Such pointwise supervision leaves the model underconstrained in function space and permits sharp, localized updates that cause catastrophic forgetting. To address this limitation, we propose Pathwise Knowledge Consolidation (PKC), which turns each replayed sample into a continuum of prediction tasks indexed by a corruption level: the label is geodesically corrupted toward the stored logits, and a lightweight training-only conditioner produces a prediction for each corrupted label. This connects the two previously isolated replay targets by a continuous trajectory of supervision, so the model preserves not only what to predict but also how its predictions evolve. A variational objective couples target fidelity along the path with path smoothness; its Euler-Lagrange characterization balances target fitting against path curvature, and a Hölder-type bound shows that the smoothness energy controls prediction drift across corruption levels. Extensive experiments demonstrate that PKC improves CL accuracy over strong replay methods while substantially reducing forgetting at no additional inference cost. PKC further outperforms recent prompt-based methods.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, catastrophic forgetting, experience replay, class-incremental learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ciupOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PKC extends replay-based continual learning by constructing a path between each sample’s label distribution and stored prediction. A training-only conditioner generates predictions along this path, with label supervision at each node and a consistency loss between neighboring nodes. Experiments with ResNet and pretrained ViT backbones report improved accuracy and reduced forgetting.

Critical strengths
  1. The method reuses backbone features across path nodes and removes the conditioner at test time, keeping inference costs unchanged.

  2. The experiments cover both training from scratch and pretrained models, with sizable reported gains on CIFAR-100.

Critical weaknesses
  1. The loss function of PKC seems overly complicated. Besides the two proposed path losses, it also combines several existing components, including LODE, DER++, teacher distillation, and anchor refresh. This makes it difficult to determine how much of the performance improvement actually comes from PKC itself. More direct ablation studies are needed to isolate its contribution.

  2. The overall objective contains six loss terms, but some hyperparameters, such as α and η in the ResNet setting, are not clearly reported, while β varies across datasets. The hyperparameter settings are not sufficiently transparent, and a more systematic sensitivity analysis would help improve reproducibility.

  3. The importance of the path smoothness term is not fully convincing. Removing it reduces accuracy from 45.16% to 43.93%, whereas removing the fidelity term drops accuracy to 40.36%. This suggests that most of the gains may come from multi-node fidelity supervision rather than the path smoothness mechanism emphasized in the paper.

Questions for authors
  1. The theoretical analysis does not match the training objective: the smoothness penalty used in practice differs from the one analyzed. How do the theoretical results apply to the implemented loss? More importantly, how does stability along the conditioning path translate into less forgetting across tasks?

  2. The ablations do not isolate the contribution of the path component from teacher distillation and memory updates. How much of the improvement remains when the path component is removed but all other settings are kept unchanged? This baseline is needed to identify where the gains come from.

  3. Does PKC use known task boundaries or session-level class information in the Si-Blurry experiments? The algorithm requires task boundaries for teacher updates and class information to distinguish old from new classes, but it is unclear how these are obtained in a blurry stream. Please clarify whether the baselines have access to the same information.

Reviewer_9CXxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on improving the replay-based method of continual learning. It argues that previous work supervised buffered samples only from isolated targets, so the model can be updated sharply, leading to catastrophic forgetting. The authors introduce the concept of corruption and a training-only conditioner to mitigate the problem. The conditioner produces several additive offsets on the backbone features corresponding to different corruption levels. Then, the shared linear head produces one prediction per level. Two novel losses act on these predictions: a cross-entropy requiring every level to predict the true label, and a smoothness penalty between adjacent levels. The authors report lower forgetting than replay-based and prompt-based baselines on several settings.

Critical strengths
  1. The paper proposes a light-weight modular extension to replay-based method. This matters because the method is novel and easy to applied to existing or future replay-based method.
  2. The proposed method have consistent gains. This means it improves the forgetting problem.
Critical weaknesses
  1. The mathematical theory in section 3 only consider how predictions vary across corruption levels, not across tasks. Also, section 3 mainly focuses on smoothness term while the ablation study shows otherwise. This matters cause the claim about why the proposed method works is not supported.
  2. The results on CIFAR-10, Tiny-ImageNet and Si-Blurry do not isolate the contribution of the proposed method. The proposed method builds on a stronger objective (LODE, DER++ and distillation) than the baselines. This leaves the sizes of the gains remain uncertain on these benchmarks.
  3. Another concern is that the experiments are mostly conducted on traditional small-scale benchmarks such as CIFAR and Tiny-ImageNet, the task sequences remain relatively short. Given the recent progress in continual learning with larger pretrained models and longer task sequences, I would like to see whether the proposed method still provides similar benefits in more challenging settings. This would help establish the generality of the contribution.
Questions for authors
  1. The authors may report a baseline that removes fidelity and smoothness terms from the full objective (i.e., keeping LODE, DER++ and distillation) on CIFAR-10, Tiny-ImageNet and Si-Blurry.
  2. The authors may conduct experiments to support the claim in section 3.3: that corrupting toward the stored logits rather than toward an uninformative distribution is essential.
Reviewer_KBu5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper considers the continual learning setting, where methods aim to improve performance on new data distributions while maintaining performance on old ones.

The method starts off from a replay-based method: a memory buffer contains some of the previously encountered data, each entry consisting of (input, true label, logits that the model predicted after training). Given a batch of new data points, a batch of old data is sampled, both are processed through the model f to extract hidden features, and the predictions are made using the output head h. 4 losses are combined (all from previous papers): 1 loss on the new task and 3 replay-related losses (denoted L_ce, L_rep, L_anc, L_tea). The paper then proposes to add two more losses on top.

Given an old data point, the paper makes the observation that only restricting the model’s behaviour on this one input has limited impact on reducing the forgetting. Instead, they seek to regularize the model’s predictions in a neighbourhood around the old data. For a single old data point tuple (input, true label, logits), they create K different probability distributions by interpolating the true label and the probability distribution resulting from the logits. These K different probabilities are fed along with the embedded interpolation coefficient into an auxiliary neural network g, which then outputs a delta to be added to the predicted hidden features f(input) which then results in K different predictions. The authors use these K predictions to compute 2 more losses that are added to the 4 losses already in the objective. The 2 new losses are fidelity (which requires that the corrupted hidden representation to be able to recover the true label), and smoothness (which computes l2 loss between distributions, to try to encourage that neighbouring predictions be close to each other).

The method is shown to outperform competing baselines on image classification tasks.

Critical strengths

The paper’s motivating insight is valuable, namely, that replaying a single data point does not sufficiently regularize the function space. The paper demonstrates promising empirical results.

Critical weaknesses

Presentation: The abstract, introduction and especially the method section are very difficult to parse. In addition, there are problems with notation, e.g. line 153 suggests that f takes t as input, which appears to be wrong. The derivations seek to justify the losses, but it is not clear how these losses prevent forgetting.

The paper adds 6 losses (4 of which are from previous work), without justifying the need for their combination, and does not carefully ablate them. This further limits the scientific insights that can extracted.

Questions for authors

The auxiliary network g appears to be trained on the two new losses which you introduce. What prevents it from collapsing to predicting some degenerate delta (e.g. a constant delta)?

Could you please explain how your theoretical results relate to the goal of reducing forgetting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.