acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Distillation via Online Drift Projection for Exemplar-Free Class-Incremental Learning

Abstract

Exemplar-free class-incremental learning must preserve old classes without storing raw examples. Prototype-based methods address this constraint by caching compact class statistics, but these statistics become stale as the feature extractor evolves. Representation drift can be heterogeneous: some old classes remain relatively stable, whereas others move substantially. This motivates adapting knowledge distillation to class-wise drift instead of weighting all old classes equally. We propose Online Drift Projection (ODP), a co-trained module that estimates class-wise drift during incremental training. ODP is optimized jointly with the evolving backbone and projects stored old-class prototypes into the current feature space. The induced prototype displacement provides an exemplar-free drift signal for each old class, without requiring access to past samples. We use this signal to construct drift-weighted distillation, which increases the contribution of high-drift classes to the distillation objective. Experiments on CIFAR-100, TinyImageNet, and ImageNet-100 show gains over AdaGauss and DPCR, while results on pretrained CUB-200 are mixed. Additional analyses show agreement between surrogate and oracle displacement, and an association between oracle displacement and class-wise forgetting, supporting online drift projection as a useful signal for adaptive distillation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
deep learning, continual learning, exemplar-free, class-incremental learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mUS4OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper proposes a new method for drift estimation based exemplar free continual learning. The method compensates past prototypes for backbone drift building upon a series of previous works on drift compensation. I think the method most similar to the proposed method is DPCR, which also has two projectors. The proposed method, in addition, proposes non-uniform distillation loss. The results are excellent for training from scratch (and on CUB from pretrained).

Critical strengths

Critical

  • The paper combines a number of existing ideas of drift compensation and adds some new ones, mostly the non-uniform distillation regularization.
  • The results of the paper are a notable improvement over existing methods, outperforming the most similar work DPCR with significant margin.

Non critical:

  • the paper is well written and the presentation is clear (except for importance of mathematical proofs).
Critical weaknesses

Critical

  • For me the main critical weakness is the clear definition of the novelty of the paper. I think the method section also requires citations to the most relevant papers, and the difference with the most relevant papers should be better explained. E.g. the projector comes from LDC, DPCR. The main difference with DPCR seems to be the drift-weighted distillation but in the ablation this does not really explain the large gap with DPCR (??).
  • the purpose of the mathematical proofs is not clear to me (especially theorem 2) And I would almost prefer to move them to appendix all. The authors should motivate better their purpose

[a] Tan, Yuwen, et al. "Semantically-shifted incremental adapter-tuning is a continual vitransformer." CVPR 2024. [b] Jiarui Zhao,; Representation-Steered Incremental Adapter-Tuning for Class-Incremental Learning with Pre-Trained Models , CVPR 2026

Questions for authors

see weaknesses above

  • some recent methods show that drift compensation can also be combined with LoRA update of large pretrained networks. It would be nice to see if the method also works in that case [a][b]. For example replacing SDC in [a] with the proposed method (I understand that this might be too much for rebuttal)

Scoring explanation: I am quite borderline on the paper, I could go up if the rebuttal addresses the raised weaknesses, but also down if I think they do not. I do not mind that the paper does not have results on large pretrained backbones currently required by mainy reviewers, I think both these research lines still have their merit.

Reviewer_t5oCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses heterogeneous representation drift in Exemplar-Free Class-Incremental Learning (EFCIL). To tackle this, the authors propose Online Drift Projection (ODP), a co-trained module that estimates class-wise drift and projects stored old-class prototypes into the current feature space. Then the authors leverage the prototype drift signal to construct drift-weighted distillation, so as to enhance the contribution of highly drifting classes to the distillation objective. ODP is evaluated on CIFAR-100, TinyImageNet, ImageNet-100, showing consistent gains over strong baselines. Extensive analyses further validate the effectiveness of the proposed mechanism, while the authors also transparently report the method's limitations on pretrained CUB-200.

Critical strengths

1.The authors propose Online Drift Projection (ODP), which estimates class-wise representation drift without storing old samples and couples naturally with the inference pipeline. It is an elegant design that adds almost no extra post-processing overhead. 2.The diagnostic analyses in paper are solid. The authors report surrogate-oracle Spearman correlations across four datasets and separately analyze the association between drift and forgetting. This detailed experimental analysis enhances the credibility of the results.

Critical weaknesses

1.In Table 2, ODP trails AdaGauss by 1.9/1.6 on CUB-200 at T=20, and falls behind EFC by an even larger margin. This indicates that the proposed method has insufficient effectiveness on the fine-grained CUB-200 dataset and lacks explanatory evidence. 2.The comparison of DWD in Table 3 contrasts the full model with (Lproj, Lcons) enabled and PTR disabled against a version without DWD, yet the authors acknowledge that attributing the entire gain specifically to class weighting would require a matched uniform KD comparison. Table 3 should include a matched uniform KD control.

Questions for authors

1.How do you explain the performance gap on the CUB-200 dataset at T=20? Have you considered adopting a different drift-to-weight mapping specifically for pretrained fine-grained datasets? 2.Could you provide a matched uniform KD control in Table 3? This would directly test whether the gain of DWD comes from class weighting.

Reviewer_3W7HOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focues on exemplar-free class-incremental learning (EFCIL) under the cold-start setting. This paper pinpoints that the issue of heterogeneous reprsentation drift: some old classes remain relatively stable,whereas others move substantially. To address this problem, online drift projection is proposed to provide drift signals of prototype and a weighted distillation loss is presented based on this. Experiments on several benchmarks demonstrate good performance of proposed ODP.

Critical strengths
  1. This paper is well written and easy to follow.
  2. Adaptive distillation built from the angle of prototype distillation is new.
  3. Expeiments on common benchmarks shows good performance of proposed method.
Critical weaknesses
  1. Major concern: relation and difference between projection alignment in proposed ODP and bidirectional alignment and cycle consistency in BiCyc [R1]. It seems that the projection alignment is the same as those in BiCyc, and the projection alignment is crucial in the proposed method. Further discussion should be provided. If not, the contribution of this paper will be limited to present a weighting distillation on top of BiCyc. Also, comparison between BiCyc should be provided.
  2. Further comparative experiments with more modern techniques should be included. The most recent baseline used in this paper is the DPCR in ICML25. Other recent baseline, e.g., GATF [R2], should be included.
  3. Lack of validation on large-scale dataset, for example, ImageNet-1k. Also, it is also interesting to validate ODP in warm-start setting.

[R1] Hongye Xu, et al., Two-Way Is Better Than One: Bidirectional Alignment with Cycle Consistency for Exemplar-Free Class-Incremental Learning, in ICLR 2026. [R2] Hongye Xu, et al., Geometry-Anchored Transport Framework for Exemplar-Free Class-Incremental Learning, in ECCV 2026.

Questions for authors

Please refer to the weaknesses. My major concerns is the weakness 1. I am willing to raise the score if the weaknesses are satisfactorily addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.