The proposed method introduces very little novelty compared with existing approaches that utilize features from multiple layers. The use of intermediate features in frozen-backbone continual learning has already been proposed by LayUP. The minor modifications introduced by FIRE consist of per-block L2 normalization followed by layer-wise weighting. However, normalization provides no benefit for the ridge head, which achieves the highest accuracy. For image datasets, uniform weighting amounts to global scaling that can be absorbed into the ridge regularization strength or kernel bandwidth, and thus plays no substantial role. For text datasets, linear weights are selected based on test accuracy. Furthermore, the authors do not investigate whether these test-selected choices transfer to other domains or to backbones of different scales and architectures. Consequently, I do not see a substantive methodological contribution in this paper.
Given the limited methodological novelty, the analytical contribution becomes particularly important. However, the analysis does not provide substantial new insights. Previous work, including LayUP, has already demonstrated that intermediate features can be beneficial for datasets that differ substantially from the natural images used in pretraining, as they provide complementary low-level information. The observation that normalization benefits RBF and random-projection heads is not particularly novel, but rather an expected consequence of their architectures. These heads are sensitive to input scale, so concatenating features without normalization naturally allows blocks with larger activation norms to disproportionately influence their outputs.
Similarly, the finding that linear ridge heads are largely insensitive to these choices is also expected, since linear models can compensate for feature scaling through their coefficients. Therefore, the observed head-dependent effects of normalization largely amount to an empirical confirmation of already understood properties.
The finding that performance improvements persist even when CIFAR-10 is converted to 28 × 28 grayscale images is a potentially novel observation, but the paper does not provide a sufficiently detailed analysis of this phenomenon.
Overall, rather than offering an analysis that changes how these methods are understood or applied, the study primarily serves as an ablation of existing approaches that quantifies largely expected effects.
The experimental evaluation lacks standard benchmarks and settings commonly used in pretrained-model-based class-incremental learning. The paper does not evaluate on standard benchmarks such as ImageNet-R, ImageNet-A, CUB-200, OmniBenchmark, or VTAB. Furthermore, first-session adaptation is removed from both RanPAC and LayUP. As a result, it is difficult to relate the reported findings to existing work or to compare the methods under their originally proposed settings.