acceptodds
Under review as a conference paper at ICLR 2027

Affine Compensation and Recoverable Feature Learning for Exemplar-Free Class-Incremental Learning

11 views

Abstract

Exemplar-free class-incremental learning must preserve past knowledge while adapting representations to new classes. Class means and covariances offer a compact memory, but keeping them aligned with an evolving feature space is difficult without old images. Our key observation is that matching these low-order moments leaves more freedom than aligning individual features. This motivates a single affine map shared across classes, fitted on current-task feature pairs to update stored statistics in closed form. Our analysis characterizes affine moment matching and bounds a shared map's old-class errors in terms of feature drift and cross-class response differences. Building on this foundation, we propose Recoverable Feature Learning with Affine Compensation (ReFAC), which combines this memory update with flexible representation learning. ReFAC trains a decoder to recover previous features and their within-class covariances, preserving recoverable information while the representation adapts to new classes. Replacing the statistics updates of four representative learners with the shared affine map largely preserves or improves final accuracy while reducing update cost. Extensive experiments on standard benchmarks show that ReFAC consistently achieves competitive or superior performance compared with state-of-the-art methods.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
class incremental learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kv4hOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses exemplar-free class-incremental learning. The key idea is to use a task-shared affine transformation to compensate for feature-space drift. Therefore, it updates the means and covariances without replaying old samples. The paper provides a theoretical analysis of affine matching and derives error bounds. Building on this idea, the authors propose Recoverable Feature Learning with Affine Compensation (ReFAC), which combines projected distillation and within-class covariance matching to preserve recoverable information while allowing the representation to adapt to new classes. Experiments show that the proposed affine compensation can replace the statistics update mechanisms of several existing EFCIL methods while largely preserving or improving their performance.

Critical strengths
  1. The representation in EFCIL is continuously evolving, while only statistics are retained for the historical classes. This mismatch is indeed important.
  2. Shared affine compensation provides a simple and direct way to alleviate this mismatch.
  3. The paper provides an analysis of the error bounds. The theoretical results are intuitive and consistent with expectations.
  4. The experimental evaluation is fairly comprehensive. Replacing the memory update with the proposed affine update in four existing methods demonstrates that the proposed approach is not effective only for a particular framework.
Critical weaknesses

Theorem 1 serves as the starting point of the proposed method, but its theoretical result is arguably too general in terms of applicability. This generality may also limit the effectiveness of the proposed approach for specific machine learning tasks. This concern is also reflected in the experimental results, where the proposed method does not appear to push the performance to a clearly new level. Nevertheless, I think that the current contribution is sufficient to warrant acceptance.

Questions for authors

Here are some suggestions regarding the presentation of the paper:

  1. Since the paper focuses on addressing a problem in incremental learning, its primary audience will likely come from the incremental learning community. Therefore, I suggest that the authors provide a broader review of incremental learning methods in the related work section, rather than focusing primarily on exemplar-free replay methods. For example, the authors may consider discussing different lines of research, such as manifold-based approaches, e.g., [Reducing Class-wise Confusion for Incremental Learning with Disentangled Manifolds], and compression-based approaches, e.g., [Prompt-Based Exemplar Super-Compression and Regeneration for Class-Incremental Learning], among others. In addition, providing more discussion of the advantages and limitations of different technical approaches, rather than simply describing existing works, would make the paper more accessible to newcomers and better highlight the unique position of the proposed work within the field.

  2. The authors provide an analysis of the error bounds in the paper. I suggest briefly discussing the practical implications of these theoretical results. This would make the theoretical analysis more accessible and helpful to readers who are less familiar with the theoretical aspects.

Reviewer_UoJmOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the exemplar-free class-incremental learning (EFCIL) under the cold-start setting. To address the shift of feature distribution drift, this paper presents a new compensation method based on affine mapping. Moreover, the proposed ReFAC utilizes a learnable decoder to improve the feature learning and the mahalanobis distance classifier to achieve the class-incremental learning. Experiments demonstrates the performance of proposed methods.

Critical strengths
  1. This paper is easy to follow.
  2. The performance of proposed affine transformation is demonstrated in the experiments.
Critical weaknesses
  1. The proposed ReFAC, except the task-shared affine compensation, seems a stack of existing methods. The learnable decoder borrows the idea in BiCyc, LfdL_{fd} is a common feature distillation loss in the EFCIL, the LacL_{ac} comes from AdaGauss, and prediction with Mahalanobis distances is core idea of FeCAM. It seems that the only new technique in this paper is the affine transformation and this paper does not provide enough discussion and analysis on it, which weaken the contribution of this paper.
  2. The details of how affine compensation is replaced in the baselines are not given, which can lead to misunderstanding.
  3. Lack of validation on large-scale dataset, for example, the ImageNet-1k.
  4. Lack of the sensitivity study of batch size when applying LprojL_{proj}. Since this loss relies on the batch size, sensitivity study shoule be included.
Questions for authors
  1. Could you please provide more analysis on the proposed affine compensation? For example, why a task-shared affine compensation is better than the learnable drift compensator in LDC? What if the a compensation focus on each class is applied? Can the affine compensation be capable of other types of classifier in CIL including nearest-class-mean classifier and the fully-connected networks? Including more analysis on affine compensation itself can strenthen the contribution of this paper.
  2. Why are the results of DCPR on CUB-200 unavailable? DPCR provides the results on CUB-200 in the paper.

For other questions, please refer to the weaknesses.

Reviewer_ggyuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a key problem in cold-start Exemplar-Free Class-Incremental Learning (EFCIL): how to keep stored class means and full covariance matrices consistent with the new feature space as the feature extractor continues to update. The authors argue that classification only requires matching low-order moments rather than aligning individual features point-by-point, suggesting that a simple mapping might suffice. To this end, they propose the ReFAC method, which builds upon AdaGauss by introducing a training-only MLP decoder, a within-class covariance matching term, and a weak direct feature drift penalty, while retaining the anti-collapse term.

Critical strengths
  1. The method shows simplicity, practicality, and transferability. The closed-form ridge mapping is easy to implement, introduces no extra hyperparameters other than ρ\rho, and can be directly plugged into existing methods.
  2. It replaces the iteratively trained adapters in AdaGauss/BiCyc with a single least-squares solve.
  3. In the experiments, the authors recompute reference statistics offline using old-class images to measure the recovery error of the means and covariances directly.
Critical weaknesses
  1. There is a theoretical gap in the core question. The theory fails to answer the core question of why a shared mapping can generalize to old classes. Theorem 1 is a per-class result, and the standard whitening-coloring construction is unrelated to the "shared mapping" design.
  2. The contradiction between fitting method and motivation is under-addressed. While the paper is motivated by "matching moments rather than aligning individual features," the actual implementation performs least-squares regression on individual feature pairs, and it lacks a residual correction baseline.
  3. The key problem is limited novelty. The closed-form linear drift compensation is closely related to LDC and DPCR. Furthermore, ReFAC mainly adds a covariance term and an L2L_2 drift penalty on top of AdaGauss.
Questions for authors
  1. I'm wondering: have you tried incorporating the residual covariance CrC_r or fitting the mapping directly via moment matching?
  2. What are the specific differences compared to DPCR and LDC? How does the ∥W−I∥\|W - I\| regularization perform compared to standard ridge regularization?
  3. Could you provide the mean ±\pm standard deviation for Tables 1 and 5, as well as a step-by-step ablation study progressing from AdaGauss+Affine to ReFAC?
  4. How much of the improvement in EFC comes from the newly added covariance update? Could you supplement the results with a baseline where the affine mapping only updates the means?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.