acceptodds
Withdrawn from ICLR 2027

Dual-State Predictive Momentum for Online Continual Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Online continual learning (OCL) empowers models to learn from non-stationary streams in a single pass while retaining prior knowledge. In this context, exponential moving average (EMA)-based strategies have achieved impressive results and underpin many state-of-the-art approaches. However, their first-order temporal smoothing only tracks the average parameter level, leading to a low-fidelity approximation of the evolving low-loss center. To this end, we propose ual-tate redictive omentum (DSPM), which jointly estimates the level and trend states of the student trajectory through two momentum estimators and extrapolates them to construct a one-step predictive teacher. From the perspective of linear mode connectivity, we disclose the tracking bias of EMA and show that DSPM yields an unbiased trajectory error and a tighter upper bound on the average empirical loss. We further introduce dual-granularity complementary distillation, which aligns teacher-student output distributions while preserving stable level-state representations, promoting classifier plasticity and representation stability. Extensive experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-100 validate the effectiveness of DSPM. Specifically, DSPM improves model stability by 2.66%, 1.40%, 3.36%, and 5.79% over the latest state-of-the-art method on these benchmarks, respectively, with an average test accuracy gain of 1.08%. Our code is available at https://anonymous.4open.science/r/DSPM.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Online continual learning, knowledge distillation, model connectivity, momentum estimate

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YDSfOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Dual-State Predictive Momentum for online continual learning. DSPM maintains a teacher model from exponential averaging while utilizing this teacher model to distill the student learner during training for online continual learning.

Critical strengths
  1. The method implementation is simple and is relatively effective.
  2. The paper is well written and easy to undrestand
Critical weaknesses
  1. Lack of novelty. The main idea is to use Holt's linear trend exponential smoothing on model parameters, the idea is not new and not as novel as claimed.
  2. The experimental result showed limited and inconsistent improvement over other benchmarks.
  3. Through ablation studies, we see the main improvement comes from the teacher-student distillation, further weakening the effect of the trend EMA developed by the author.
Questions for authors
  1. This paper Parabolic Continual Learning [1] suggested another type of interpolation between data. Curious on the author's take on the connection between their methods (or EMA of parameter weights) and this paper.
  2. The author needs to address clearly on how is their method novel, and how is this novelty translated towards the empirical gains, otherwise the author's claim is severely weakened.

Yang, H., Hasan, A. & Tarokh, V.. (2025). Parabolic Continual Learning. Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 258:2620-2628 Available from https://proceedings.mlr.press/v258/yang25b.html.

Reviewer_yZXXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Dual-State Predictive Momentum (DSPM) for class-incremental online continual learning. To address the tracking lag of conventional EMA teachers, it maintains level and trend estimates of student parameters and combines them into a one-step predictive teacher.

The method applies complementary distillation: output distributions are distilled from the predictive teacher, while intermediate representations are distilled from the level-state model. Inference averages the student and predictive teacher parameters.

The paper analyzes tracking behavior under an idealized linear-drift model and evaluates DSPM on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-100. Experiments cover multiple memory sizes, component ablations, hyperparameter sensitivity, representation diagnostics, and runtime comparisons.

Critical strengths
  1. Useful complementary teacher design. Using the predictive teacher for output distillation and the level-state model for feature distillation provides a sensible mechanism for combining adaptation with representation preservation.

  2. Competitive empirical results. DSPM achieves the best reported ACC in 8 of 11 settings, including improvements of 1.13–1.60 percentage points over the strongest reported competitors on ImageNet-100.

  3. Substantial experimental validation. Four datasets, multiple memory sizes, five runs, component ablations, sensitivity analyses, and runtime comparisons provide broad support for the method’s practical value.

Critical weaknesses
  1. Limited novelty of the core update. Eqs. (12)–(14) match classical Holt linear exponential smoothing. The paper should acknowledge this connection and position its contribution around the OCL adaptation and complementary distillation design.

  2. Restricted theoretical support. The zero-error result assumes an already-optimal student, constant linear drift, and steady-state tracking, while the stability proof requires correction. The theoretical claims should be qualified accordingly.

Questions for authors
  1. Connection to Holt smoothing: Please acknowledge and clarify the relationship between Eqs. (12)–(14) and Holt linear exponential smoothing. Which aspects of the teacher construction and distillation design constitute the main OCL-specific contribution?

  2. Theoretical justification: Please correct the characteristic polynomial in Eq. (33) and provide a Schur-stability argument that allows complex eigenvalues. What conclusions remain when the student has nonzero optimization error or the optimum does not follow constant drift?

  3. Matched stability–plasticity comparison: Table 3 shows that adding Trend to Level+CKD1+CKD2 increases ACC but also increases AFM on all three datasets. Does the predictive variant improve the achievable ACC–AFM trade-off over an EMA teacher with its decay retuned, while keeping distillation and inference averaging matched?

Reviewer_YrAmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DSPM for online continual learning, constructing a predictive teacher by tracking both the level and trend of student parameters. Under idealized local linear-drift assumptions, the analysis shows reduced tracking bias and a tighter empirical loss bound compared with conventional EMA. The method combines output-level distillation from the predictive teacher with feature-level distillation from the level state to balance plasticity and stability. Experiments on four image classification benchmarks demonstrate competitive accuracy and reduced forgetting compared with existing methods.

Critical strengths
  • Detailed hyperparameter analysis. The paper examines the sensitivity of both accuracy and forgetting to the level/trend smoothing coefficients and distillation weights.

  • Well-motivated complementary distillation. Using the trend-aware teacher for output-level supervision and the level state for feature-level supervision provides an intuitive mechanism for combining adaptation with representation preservation.

Critical weaknesses
  • Limited evaluation at scale. All experiments use ResNet-18 trained from scratch, leaving the effectiveness of DSPM on larger and pre-trained models, vision transformers, and pre-trained models unverified. Moreover, evaluation is restricted to CIFAR-10/100, Tiny-ImageNet, and ImageNet-100. Experiments on larger benchmarks is required to show its scalability.

  • Limited novelty and missing discussion of prior work. The updates in Eqs. (12)–(14) are mathematically equivalent to Holt’s linear trend method applied to model parameters, yet this established method is neither cited nor discussed. While its application to OCL and integration with complementary distillation may be useful, the novelty of the predictive update itself is unclear. The authors should acknowledge this connection and clarify which contributions extend beyond the existing forecasting method.

  • Restrictive theoretical assumptions. The analysis assumes constant local drift and an ideal student that exactly tracks the cumulative-risk optimum (θt=θt∗)(\theta_t=\theta_t^*). Its asymptotic guarantees therefore do not establish an advantage under realistic OCL dynamics involving optimization errors, noisy updates, and abrupt task transitions. Analysis or experiments addressing these conditions would strengthen the theoretical justification.

  • Marginal accuracy gains and unclear statistical significance. In several settings, DSPM provides only small accuracy improvements over the strongest baselines. Although results are averaged over five seeds, no statistical significance tests (e.g., t-test) are reported. The authors should provide appropriate significant tests for the performance differences to show whether these gains are significant.

  • Insufficient evidence of an improved stability-plasticity trade-off. In Table 3, adding the trend component while retaining both distillation losses improves accuracy but increases forgetting across all three datasets. This demonstrates a shift toward plasticity, but does not establish a better attainable trade-off. The authors should compare against a trend-free variant with tuned EMA coefficients and distillation weights, ideally showing higher accuracy at matched forgetting levels.


As an additional comment, and as a researcher who really loves continual learning area, I would encourage the authors to reflect more deeply on the practical problem this work aims to address. With large pre-trained models increasingly central to real-world applications, and even RSI becoming a important research direction, I find it difficult to assess the broader significance of modest improvements in class-incremental classification using ResNet-18 trained from scratch. The paper would benefit from a clearer explanation of which fundamental limitation of practical continual learning scenario the proposed method addresses and why that limitation matters in real-world continual learning scenarios.

Questions for authors
  • Since DSPM’s parameter updates are architecture-agnostic, could the method also be effective for continual learning in LLMs?

Please refer to the weakness for more questions

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.