acceptodds
Withdrawn from ICLR 2027

The Features We Make Along the Way: Intermediate Representations for Training-Free Continual Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Intermediate representations can improve frozen-backbone classification, but combining depths, selecting a better layer, and changing the classifier can each contribute. We examine these choices across eleven image datasets and four frozen ViTs, comparing layer selection, normalization, and classifier heads on the same features. On five datasets covering digits, traffic signs, and overhead imagery, normalized all-layer features outperform the best validation-selected single block on 18 of 20 backbone-dataset pairs with KLDA and 19 with RanPAC. Equally wide random nonlinear views of the final block do not recover these gains. Direct ridge regression on raw all-layer features performs best on these datasets, reaching 91.82% accuracy, compared with 88.94% for normalized RanPAC and 88.97% for normalized KLDA. Per-block normalization improves the random-feature heads but leaves ridge accuracy essentially unchanged. With hyperparameters selected without future-task data, all-layer inputs add approximately four percentage points of average incremental accuracy for both random-feature heads on the same five datasets, although restricted baseline tuning likely inflates RanPAC's gain. The dataset grouping was exploratory. A prediction stated before evaluating six held-out datasets separated three MNIST-format datasets from three natural-photograph datasets by relative error reduction under both random-feature heads. A second test separated content from format: the gain on digits survives a change to color photographic images, but converting CIFAR-10 to grayscale also produces a gain, so image format contributes as well.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Pretrained Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_GeH2OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper performs empirical evaluations of intermediate model layer output on model classification, and concludes that direct ridge regression on all-layer features perform best.

Critical strengths

None

Critical weaknesses

This paper only empirical explores whether combining all-layer feature representation is better than other alternatives (single block output or part of blocks' outputs). There is no any methodological novelty and contribution. Details of experimental results are described in the main text, more like a common engineering report. In addition, there are few experimental results about continual learning, although the paper title emphasizes training-free continual learning.

Questions for authors

Please clarify technical novelty and contributions if there is any.

Reviewer_2GaROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the use of intermediate representations from frozen pretrained transformers for rehearsal-free continual classification. It compares different choices of depth, per-layer normalization, and classifier heads, including ridge regression, RanPAC, and KLDA. The paper introduces a normalized all-layer representation referred to as a “firing pattern” or FIRE and evaluates it across four frozen vision backbones and eleven image datasets, with additional text and held-out image experiments. The main empirical observation is that combining representations from multiple transformer layers can improve over a final-layer representation, particularly on a subset of digit, traffic-sign, and overhead-imagery datasets. The paper further evaluates whether these gains can be explained by normalization, feature dimensionality, or the presence of one particularly strong intermediate layer. However, the strongest configuration reported in the matched comparison is direct ridge regression on raw all-layer features rather than the proposed normalized FIRE representation. The study also contains a strictly continual protocol, held-out dataset evaluations, significance analyses, and several post-hoc investigations intended to characterize when intermediate representations are useful

Critical strengths

The paper contains a relatively large number of control experiments attempting to separate the effects of feature depth, normalization, classifier choice, and representation width. In particular, the comparison against validation-selected single layers and width-matched random transformations of the final layer is useful for showing that the observed gains cannot be explained solely by increased dimensionality. The authors are also unusually explicit about several limitations of the experimental protocol, including test-informed design decisions, numerical instability, exploratory dataset grouping, and differences between feature-extraction pipelines. This transparency is useful for interpreting the reported results. These strengths, however, do not overcome my concerns regarding novelty, experimental organization, and the strength of the central contribution.

Critical weaknesses
  1. The central contribution does not appear sufficiently novel or substantive relative to prior work, particularly LayUP. The closest prior work, LayUP, already establishes that intermediate-layer representations can improve rehearsal-free continual learning by concatenating representations across multiple depths and applying a ridge-style classifier. The present paper performs a much more extensive decomposition of this idea, but the strongest configuration reported by the paper itself is raw all-layer features with ridge regression. The normalized FIRE construction does not improve the ridge result: on the high-gain datasets, raw all-layer ridge reaches 91.82 while normalized all-layer ridge reaches 91.81. This makes it difficult for me to identify a sufficiently new learning principle. Much of the paper appears to provide additional ablations, controls, and dataset-specific characterization of a phenomenon that was already established by prior work. The empirical effort is substantial, but the amount of experimentation does not by itself make the underlying methodological contribution substantially different.

  2. The experimental story is highly fragmented and contains substantial exploratory and test-informed components. The paper combines several different evaluation protocols: an original grid, matched comparisons, a second feature extraction, extended hyperparameter searches, a strictly continual evaluation, exploratory dataset grouping, held-out dataset predictions, and later boundary tests. Several choices were influenced by observed test performance, including the layer weighting strategy and DTD feature dimensionality, and some earlier ablations selected hyperparameters using test results. The paper also explicitly states that the dataset grouping was chosen after observing the results, and the later boundary experiments show that the proposed interpretation is confounded by image format: converting CIFAR-10 to 28×28 grayscale itself creates substantial gains. Most importantly, the paper acknowledges that the earlier test-informed choices and the change in feature extraction prevent the whole study from being interpreted as a single evaluation planned in advance. I therefore find the overall empirical narrative difficult to interpret. The manuscript contains many individual results, but they do not form a sufficiently clean and unified validation of one central contribution.

Questions for authors
  1. What precisely is the main methodological contribution relative to LayUP? In particular, given that the strongest configuration in the paper is raw all-layer ridge regression, which is very close to the core construction already explored by LayUP, what central result in this paper should be considered genuinely new rather than a more extensive empirical characterization of an existing approach? The answer would materially affect my assessment of the novelty and significance of the paper.

  2. Which result should be regarded as the primary fully confirmatory experiment, with the representation construction, hyperparameters, dimensionality, and evaluation protocol all fixed before observing the corresponding test results? This is important because the paper currently mixes test-informed choices, exploratory grouping, multiple feature extractions, and later confirmatory analyses, making it difficult to determine which conclusions are supported independently of the exploratory development process.

  3. The paper contains many experiments but lacks a sufficiently clear hierarchy of evidence. The repeated transitions between original-grid experiments, matched comparisons, different feature extractions, extended hyperparameter searches, continual evaluation, exploratory grouping, held-out datasets, and boundary analyses make the central argument difficult to follow. A large portion of the manuscript is spent qualifying earlier experiments or explaining protocol differences. Although these qualifications are scientifically responsible, the final presentation reads more like a sequence of experimental corrections and post-hoc analyses than a cleanly designed study supporting one central claim.

  4. The main high-level observation—that intermediate representations from a frozen pretrained model can outperform the final layer in rehearsal-free continual learning—has already been established in closely related prior work. The paper provides substantially more experimentation around this observation, but I did not find that it establishes a comparably substantial new principle or method.

  5. I do not believe the reported empirical effects are necessarily incorrect. However, the study mixes exploratory analysis, test-informed design choices, multiple feature extractions, and later confirmatory evaluations. These issues complicate the interpretation of the results and weaken the strength of some broader claims. The paper is therefore not fundamentally unsound, but I do not think the current experimental organization provides a sufficiently clean foundation for the conclusions emphasized by the manuscript.

Reviewer_hh2DOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how intermediate features from a frozen ViT contribute to training-free class-incremental learning. The authors compare ridge regression, RanPAC, and KLDA using identically cached features across 11 image datasets and four backbones, isolating the effects of depth combination, per-block normalization, and classifier choice. They propose a feature representation called FIRE, which concatenates normalized and weighted block states, and evaluate it through various controlled experiments and held-out dataset evaluations. The main findings are that combining all blocks outperforms the best validation-selected single block on digit, traffic-sign, aerial, and satellite image datasets; direct ridge regression on raw all-layer features achieves the highest accuracy; and normalization benefits only random-feature heads.

Critical strengths

The comparisons among different methods are conducted under controlled experimental conditions.

The authors transparently report the limitations of their study, including design choices informed by test-set results, post-hoc dataset grouping, and unsuccessful a priori predictions.

Critical weaknesses

The proposed method introduces very little novelty compared with existing approaches that utilize features from multiple layers. The use of intermediate features in frozen-backbone continual learning has already been proposed by LayUP. The minor modifications introduced by FIRE consist of per-block L2 normalization followed by layer-wise weighting. However, normalization provides no benefit for the ridge head, which achieves the highest accuracy. For image datasets, uniform weighting amounts to global scaling that can be absorbed into the ridge regularization strength or kernel bandwidth, and thus plays no substantial role. For text datasets, linear weights are selected based on test accuracy. Furthermore, the authors do not investigate whether these test-selected choices transfer to other domains or to backbones of different scales and architectures. Consequently, I do not see a substantive methodological contribution in this paper.

Given the limited methodological novelty, the analytical contribution becomes particularly important. However, the analysis does not provide substantial new insights. Previous work, including LayUP, has already demonstrated that intermediate features can be beneficial for datasets that differ substantially from the natural images used in pretraining, as they provide complementary low-level information. The observation that normalization benefits RBF and random-projection heads is not particularly novel, but rather an expected consequence of their architectures. These heads are sensitive to input scale, so concatenating features without normalization naturally allows blocks with larger activation norms to disproportionately influence their outputs.

Similarly, the finding that linear ridge heads are largely insensitive to these choices is also expected, since linear models can compensate for feature scaling through their coefficients. Therefore, the observed head-dependent effects of normalization largely amount to an empirical confirmation of already understood properties.

The finding that performance improvements persist even when CIFAR-10 is converted to 28 × 28 grayscale images is a potentially novel observation, but the paper does not provide a sufficiently detailed analysis of this phenomenon.

Overall, rather than offering an analysis that changes how these methods are understood or applied, the study primarily serves as an ablation of existing approaches that quantifies largely expected effects.

The experimental evaluation lacks standard benchmarks and settings commonly used in pretrained-model-based class-incremental learning. The paper does not evaluate on standard benchmarks such as ImageNet-R, ImageNet-A, CUB-200, OmniBenchmark, or VTAB. Furthermore, first-session adaptation is removed from both RanPAC and LayUP. As a result, it is difficult to relate the reported findings to existing work or to compare the methods under their originally proposed settings.

Questions for authors

Given that per-block normalization has no effect on the ridge head and uniform weighting for image datasets effectively amounts to global scaling, what is the practical advantage of FIRE over simply concatenating raw intermediate features, as in LayUP?

Could the authors provide results on standard pretrained-model-based class-incremental learning benchmarks, including first-session adaptation, and compare LayUP and RanPAC under their original experimental settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.