acceptodds
Under review as a conference paper at ICLR 2027

Measuring Class Expansion Capacity in Frozen Representations

Abstract

How many new classes can a frozen representation support without updating its feature extractor? We introduce frozen-representation class expansion — an extractor is trained on B base classes, frozen permanently, and then used to recognise further classes from newly stored statistics alone — and define expansion capacity as the largest label space served at a stated accuracy and memory budget. Measured on the arriving classes alone, capacity is much lower than on the pooled label space (65.8±6.8 vs. 88.2±3.1 classes at 50% accuracy), and it grows with the number of base classes even as closed-set accuracy falls. Second-order estimation, not prototype multiplicity, is the dominant lever (+16.4 points against at most +1.1). This motivates a readout combining shared whitening with a low-rank class-specific residual, which holds the accuracy–storage frontier at its own budget in nine of ten representation–dataset settings and on a 300-class stress test, and matches or beats full per-class covariance at 2.6–5.3% of its memory on every pretrained extractor, including at native resolution on Food-101. Whitening instead by the base features' uncentred second moment adds 1.7 to 4.3 points on backbones trained on the base classes, partly through the directions they learned to separate them, a gain whose sign a measured specialisation index predicts in 14 of 15 settings; it does not help pretrained ViTs. The prototype and depth findings are largely specific to backbones trained from scratch on few classes, at 32 pixels and at native resolution alike.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
frozen representations, class-incremental learning, exemplar-free continual learning, representation evaluation, covariance estimation, memory-efficient classifiers, prototype classifiers, expansion capacity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_goGiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how well frozen representations can handle new classes without updating the feature extractor and introduces a new metric called Expansion Capacity to evaluate this ability. The authors also propose a memory-efficient method combining shared whitening and low-rank covariance information, with experiments on several datasets showing competitive performance while using much less memory.

Critical strengths
  1. The paper studies a meaningful problem: how many new classes a frozen feature extractor can support without further training. This is an important question for continual learning, especially when retraining a model is expensive.

  2. The authors conduct extensive experiments on different datasets and backbone architectures. The results show that their method achieves competitive performance while using much less memory. The authors also provide detailed ablation studies to support their findings.

Critical weaknesses
  1. The paper does not seem to follow the standard ICLR template. For example, the title and font style look different from typical ICLR papers. I suggest the authors check the official template and fix the formatting issues.

  2. The paper is quite difficult to read, including the introduction. Even after carefully reading the introduction, I still find it hard to understand the main motivation and the key contributions of this work. The paper contains too much text and technical detail but very few figures to help readers understand the main ideas. I think the authors should improve the writing and add some illustrations, especially a framework figure to explain how the proposed method works. This would make the paper much easier to follow.

Questions for authors

Could the authors explain the main motivation and contributions more clearly? I found the Introduction difficult to follow, and I think a more intuitive explanation of the problem and the proposed method would help.

Could the authors provide more details about the computational cost of the proposed method? In particular, how does its training and inference efficiency compare with existing methods under similar memory budgets?

Reviewer_KYbCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how many new classes a frozen feature extractor can support without retaining previous images or features. It defines expansion capacity through an accuracy threshold and proposes Shared+LRr, which combines shared whitening with low-rank class-specific residual. Experiments show substantial saving of storage and utilizing second-order information gains more than additional prototypes. They also show that conclusions about layer depth and the best prior depend on how the backbone was trained.

Critical strengths
  1. The accuracy-storage result is strong. Shared+LR32 achieves 47.60% accuracy at 7.46 MB, compared with FeCAM's 46.44% at 100.20 MB. Similar savings appear on pretrained extractors and Food-101. The comparisons include checks of baseline settings, making the evidence for an effective accuracy-storage tradeoff more convincing.
  2. The controlled experiments help explain the results. When fixing sample count, pooling more classes improves performance. Fine-tuning and augmentation experiments also show when adding between-class scatter helps and when it does not. These findings help explain the method's strengths and limits.
Critical weaknesses
  1. The CIFAR-100 test set was read during development, and the later validation includes base-class images already used to train the backbone. This weakens the independence of the validation used to support the strongest configuration.
  2. The value of EC as a main contribution is limited. AUC-EC ranks methods almost identically to average incremental accuracy, while the new-class version cannot be calculated for some weaker methods at the chosen threshold. This limits its usefulness as a common comparison measure.
Questions for authors
  1. Given that the backbone was trained on the base-class validation images, would the same prior and regularization strength still be selected using validation images excluded from both backbone training and the estimation of statistics?
  2. Beyond expressing performance in numbers of classes, what comparison or decision does EC enable that is not yet apparent from separate base-class and new-class accuracy curves? An example from the existing experiments would help show its added value.
Reviewer_fvYjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the capacity of frozen representations to accommodate new classes without updating the feature extractor or retaining previously observed examples. The authors introduce Expansion Capacity (EC), a metric that measures the maximum number of classes a frozen representation can support at a specified accuracy threshold and memory budget, distinguishing between performance on the overall label space and newly introduced classes.

Based on empirical analyses showing that second-order feature statistics contribute more to class expansion than increasing the number of prototypes, the authors propose Shared-Prior Low-Rank Memory (Shared+LR). The method combines shared covariance-based whitening with class-specific low-rank residuals to achieve memory-efficient classification. Experiments across CIFAR-100, Tiny-ImageNet, Food-101, and multiple CNN and pretrained vision backbones demonstrate competitive accuracy with substantially lower storage requirements than full per-class covariance methods. The paper further examines the effects of base-class count, representation depth, sample efficiency, and covariance estimation, identifying conditions under which these findings generalise.

Critical strengths

Strong empirical evaluation and mechanistic analysis. The paper provides extensive experiments across multiple architectures, datasets, and representation settings, including pretrained models and a 300-class benchmark. Beyond reporting performance improvements, the authors systematically investigate the roles of second-order statistics, prototype multiplicity, sample efficiency, and representation depth. These controlled analyses help substantiate the central findings and clarify their limitations, making the empirical contribution more convincing.

Substantial memory efficiency with competitive classification performance. The proposed Shared+LR method achieves comparable or better accuracy than full per-class covariance estimation while requiring substantially less storage. For example, on CIFAR-100, the uncentred-prior variant achieves 51.73% accuracy using 7.46 MB, compared with 46.44% using 100.20 MB for FeCAM. The memory-efficiency advantage also extends to pretrained representations and Food-101. This represents a practically meaningful contribution for expanding classification systems under constrained storage budgets and is the strongest argument in favour of acceptance.

Critical weaknesses

Limited methodological novelty. The proposed Expansion Capacity (EC) metric is essentially a threshold-based reparameterisation of the conventional accuracy-versus-class-count curve. The authors acknowledge that AUC-EC is almost perfectly correlated with average incremental accuracy (Spearman ρ = 0.992), raising questions about the additional insights it provides. Similarly, the proposed Shared+LR method combines established techniques, including covariance whitening, low-rank covariance approximation, and Mahalanobis classification. Although the combination is empirically effective, the paper does not sufficiently establish what fundamentally new methodological or theoretical insights it offers beyond existing approaches. This limits the originality of the contribution.

Incomplete theoretical justification and limited generalisability of the mechanistic findings. Proposition 1 appears to overlook the finite-base-set covariance heterogeneity when characterising the estimation error of the pooled covariance, potentially weakening the theoretical explanation for why shared estimation improves performance. Furthermore, several key findings, including the benefits of intermediate representations and between-class covariance information, are highly dependent on the backbone and training configuration. The proposed specialisation index is also evaluated retrospectively using data from classes outside the base set, limiting its practical predictive utility. These issues weaken the generality and explanatory strength of the paper's central conclusions.

Questions for authors
  1. Novelty and added value of Expansion Capacity (EC). Given the very high correlation between AUC-EC and average incremental accuracy (Spearman ρ = 0.992), could the authors clarify what additional scientific insights or practical decisions EC enables beyond conventional accuracy curves? In particular, is there a concrete example where the proposed metric or its base/new-class decomposition leads to a conclusion that would otherwise be overlooked? This clarification would help assess the conceptual novelty of the proposed evaluation framework.

  2. Theoretical justification for shared covariance estimation. Proposition 1 argues that pooling covariance estimates across BB base classes reduces estimation variance at the cost of a class-specific bias term. However, for a finite randomly selected base set, the pooled covariance also depends on the average deviation of those base classes from the population covariance. Could the authors clarify whether the bound is conditional on a fixed base set or accounts for this additional source of variability? If the latter, how is the finite-base-set heterogeneity incorporated into the estimation-error analysis? This is important for evaluating the theoretical explanation of the method's effectiveness.

  3. Generalisability and practical applicability of the specialisation index. The authors report that the proposed specialisation index predicts the sign of the between-class covariance contribution in 14 of 15 settings. However, its computation relies on labelled examples from classes outside the base set, and the evaluated settings are not fully independent. Could the authors clarify whether this index is intended as a retrospective explanatory measure or a prospective criterion for selecting the covariance prior? If the latter, how could it be computed using only information available before class expansion? This distinction would materially affect the practical significance of the mechanistic findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.