- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes InnerCIL, an analytic class-incremental learning method that constructs a richer frozen representation from the internal neurons of a PTM. The method combines multi-depth CLS features with attention-pooled MLP neuron responses from intermediate layers. A checkpoint-based registration step selects a fixed candidate dictionary, Visual Fisher then selects discriminative local coordinates at each incremental stage, and a closed-form ridge classifier is fit from accumulated statistics without replay data. The paper provides theoretical motivation for accessing intermediate neuron responses and empirical results on CLIP and DINOv2 backbones.
Critical strengths
- The core representation idea is interesting and motivated. In particular, reading individual MLP neuron responses before the output projection gives access to information that may be lost by the normal synthesis operations, and the paper provides an ablation for the synthesis-null component argument.
- The method is straight forward from an optimization perspective: the backbone remains frozen and the classifier is reconstructed analytically from statistics. This avoids representation drift and preserves exact batch-equivalence to the ridge-regression model.
- The ablations are useful and support several parts of the design. In particular, table 3 shows gains from both multi-depth CLS features and the native local dictionary, while table 4 analyzes native versus random features, attention versus uniform pooling, and the synthesis-null component.
Critical weaknesses
- The main scalability issue is the size of the Gram matrix. Although Fisher reduces the deployed dimension from P to K, the method must retain G_t^{raw} over the pre-Fisher candidate representation so that a different Fisher support can be selected at later steps. Therefore, Fisher does not remove the quadratic O(D_P^2) memory bottleneck. This may become problematic for wider backbones, more layers, or larger native dictionaries.
- Fisher is a univariate selection criterion: each coordinate is scored independently using between-class versus within-class variance. Two individually strong coordinates may be highly redundant, while two individually mediocre features may become useful when considered jointly. The Fisher objective differs from the joint multivariate objective when coordinates are correlated. This is also interesting considering prior analytic methods, for example RanPAC, which uses random projections to capture nonlinearity between feature dimensions; could such a component complement the proposed native features?
- I have some concerns about comparison fairness with the baselines. For example, table 3 shows a large improvement by replacing final CLS with multi-depth CLS, from 73.05 to 77.50 last accuracy. Multi-depth representations are not specific to InnerCIL and could be incorporated into several competing analytic methods. It would be useful to compare against stronger versions of the baselines using multi-depth global representation.
- More broadly, a significant part of the contribution can be viewed as constructing a stronger representation by combining multi-layer global and local features. I am not very familiar with the broader literature on multi-level feature fusion, but there appears to be relevant prior work that deserves more discussion and potentially comparison. For example, Chen et al. [1] study hierarchy-aware aggregation of transformer features, while Lin et al. [2] study layer selection and fusion of multi-layer visual representations. Although these works consider different settings, they seem relevant for understanding how much of the gain comes from the proposed native-neuron representation versus the more general benefit of aggregating features across layers.
[1] Yongjie Chen, Hongmin Liu, Haoran Yin, and Bin Fan. Building Vision Transformers with Hierarchy Aware Feature Aggregation. ICCV 2023.
[2] Junyan Lin, Haoran Chen, Yue Fan, Yingqi Fan, Xin Jin, Hui Su, Jinlan Fu, and Xiaoyu Shen. Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practices. CVPR 2025.
Questions for authors
- Have you considered combining the proposed global/local native representation with an additional nonlinear feature expansion such as the random projection used in prior analytic CIL methods? The dimensionality could become large, but this might capture interactions that the current univariate Fisher selection and linear ridge readout cannot represent. I would be interested to know whether native features and random nonlinear features are complementary.
- What happens if Fisher filtering is removed and the classifier is fit using all P registered native coordinates? Is Fisher mainly included to reduce the final ridge-solve dimension, or does it improve accuracy? Since the method already stores the Gram statistics before Fisher selection, an ablation comparing all P coordinates vs. Fisher-K would clarify this.
4 comments
Response to W3 and W4
W3: Fairness against stronger multi-depth CLS baselines
Following your suggestion, we strengthen Direct Ridge and RanPAC with the same L2-normalized CLS concatenation from blocks 9–12 used by InnerCIL. The frozen backbone, class order, and training views are shared. Baseline regularization is selected on a first-task training/validation split; RanPAC uses a fixed 7,168-dimensional random ReLU expansion. InnerCIL retains its default configuration.
| Method and global features | CLIP-CIFAR | CLIP-CUB | DINOv2-CIFAR | DINOv2-CUB | Mean |
|---|---|---|---|---|---|
| Direct Ridge, final CLS | 76.70 | 76.54 | 87.95 | 88.57 | 82.44 |
| Direct Ridge, multi-depth CLS | 81.39 | 78.89 | 89.10 | 90.04 | 84.86 |
| RanPAC, final CLS | 81.13 | 81.90 | 89.59 | 90.27 | 85.72 |
| RanPAC, multi-depth CLS | 81.96 | 80.32 | 90.23 | 90.51 | 85.76 |
| InnerCIL, multi-depth CLS + native neurons | 85.26 | 83.93 | 91.59 | 90.61 | 87.85 |
Final accuracy (%) after ten incremental stages.
Multi-depth CLS improves Direct Ridge throughout and RanPAC in three of four settings. InnerCIL still leads in every setting, exceeding the multi-depth RanPAC baseline by 2.09 pp in mean accuracy. Tables 3 and 10 also isolate the native dictionary's contribution while holding the multi-depth global branch fixed.
The advantage persists after giving competing analytic baselines the same multi-depth global representation, directly identifying the added value of native local responses.
W4: Relation to multi-level feature aggregation and fusion
We have added the two works you recommended to Sec. 2. Their connection to InnerCIL is the use of information from multiple depths; the key differences concern what information is read and how it is learned.
Chen et al.'s HAFA (ICCV 2023) changes token aggregation within a trained ViT through local aggregation in shallow layers and semantic aggregation in deeper layers. InnerCIL leaves the pretrained forward computation unchanged, reuses existing attention and MLP activations, and reconstructs the incremental classifier without gradient-based backbone training.
Lin et al. (CVPR 2025) study layer selection and fusion of block-output patch representations for multimodal language models, with trainable projectors and instruction tuning. InnerCIL reads individual MLP responses before output projection and uses cumulative sufficient statistics instead of training a fusion network.
The empirical distinction is isolated in Table 7: with the multi-depth global branch, selected coordinates, scaling, dimension, and regularization held fixed, restoring the pre-projection null component improves accuracy in all 16 backbone–dataset settings. Thus, the comparison tests information lost by output projection while keeping cross-layer aggregation unchanged.
InnerCIL combines a frozen, replay-free analytic learner with access to pre-projection information that conventional block-output fusion cannot directly recover.
Dear Reviewer vxeg,
Thank you for your time and effort in reviewing our work. As the rebuttal period concludes within two days, we wish to respectfully confirm whether our responses have adequately addressed your initial concerns. Any feedback would be greatly appreciated.
Sincerely,
The Authors
I thank the authors for their detailed rebuttal and additional experiments. I appreciate their efforts to address my concerns, particularly the stronger baseline comparisons and additional ablations. My remaining concern is primarily about the nature of the contribution, which I still view as largely constructing a stronger feature representation through multi-layer global and local feature aggregation, rather than introducing a fundamentally new learning framework. While the authors have clarified the distinctions from related work, I believe this perspective remains relevant when assessing the novelty. Overall, my assessment remains positive, and I will maintain my original score.
Response to W1, W2, Q1, and Q2
Thank you for recognizing the native-neuron representation, analytic reconstruction, and supporting ablations.
W1 + Q2: Raw Gram memory and the effect of removing Fisher
With global width dg, the raw Gram matrix remains quadratic in the candidate dimension DP=dg+P+1: dynamic Fisher reselection requires preserving statistics for every registered candidate. Packed upper-triangular storage uses DP(DP+1)/2 entries (Appendix A.5, Eq. 46).
The two budgets control different costs. Checkpoint registration limits the native dictionary to P atoms, so wider backbones or more layers do not require retaining every available neuron. Fisher then selects K coordinates, reducing the dense solve to O(DK3) and deployed classifier storage to O(DKCt). Here DK=dg+K+1 and Ct is the number of seen classes. Wider backbones and more global blocks also increase dg; both global width and local budget must therefore be controlled.
We vary both budgets on DINOv2/CUB:
The medium budget reduces state by 36.7% with only 0.13 percentage points (pp) less accuracy; the smallest budget reduces state by 72.8% and selection/solve time by 75.5%, retaining 90.30% accuracy. Low-rank or block-diagonal compression of second-order statistics is a further direction discussed in Sec. 5.
Removing Fisher. We fit the classifier using all P=11264 native coordinates, keeping the global branch, RMS rules, training views, and ridge penalty identical. Fisher retains K=8704 coordinates:
Reducing local width by 22.73% lowers selection/solve time by 44.1% on CLIP and 44.9% on DINOv2, while also reducing deployed classifier storage and multiplication cost. Mean final accuracy improves by 0.09/0.07 pp.
The raw candidate Gram remains stored for future reselection; these savings concern the selected solve and deployed classifier.
Candidate budgeting controls raw-state memory; Fisher provides cheaper analytic solves and a modest accuracy improvement at the deployed dimension.
W2 + Q1: Univariate selection, feature correlations, and nonlinear complementarity
Fisher ranks coordinates individually; the full-Gram ridge solve models correlations among the retained coordinates, including global–local and local–local covariances (Appendix A.2, Theorem 4 and Eq. 42). It cannot recover useful coordinates omitted during selection, so we directly test redundancy-aware selection.
At fixed P=11264, K=8704, and λ=0.003, we compare Random-K, maximum-variance selection, Visual Fisher, and greedy Fisher that subtracts 0.25 times the maximum squared correlation with selected features from the normalized Fisher score:
Visual Fisher achieves the highest mean and exceeds the correlation-penalized variant by 0.46 pp on CUB. Correlated responses can form useful difference combinations that cancel shared background variation; penalizing their correlation during selection can discard such information before the joint classifier uses it.
To test nonlinear complementarity, we combine the global/local native representation u(x) with a fixed Gaussian–ReLU map q(x)=ReLU(R⊤u(x)). Input scales use only first-task training data, and R stays fixed. On CUB, all variants retain the same four-layer CLS branch, λ=0.003, and total local budgets P=11264, K=8704.
Native Only assigns the whole local budget to native neurons; RP Only assigns it to random features. The hybrid splits both budgets equally: P=5632+5632, K=4352+4352.
The hybrid gives DINOv2 the highest average and final accuracy, improving over Native Only by 0.05/0.04 pp. This shows a small complementary gain in this DINOv2 setting. On CLIP, however, assigning half the budget to RP displaces useful native responses, and Native Only performs best.
Native Only exceeds RP Only on both backbones and requires no additional projection weights; the hybrid adds 286–308 MiB.
Visual Fisher gives the best mean accuracy in the selection comparison, while the RP hybrid adds a small DINOv2 gain at extra storage cost; native responses remain the efficient default.