acceptodds
Under review as a conference paper at ICLR 2027

InnerCIL: Internal Neuron Representations for Analytic Class-Incremental Learning

Abstract

Class-incremental learning aims to enable a model to sequentially learn new classes while preserving its ability to recognize old ones. Recently, analytic learning methods have achieved advanced performance by updating classifiers via accumulated features, with the pretrained backbone frozen. However, existing approaches typically focus on transforming or resampling the output features during analysis, where transforming and resampling can introduce feature drift and feature noise respectively, limiting the robustness of the obtained classifier. To address this problem, we propose InnerCIL, where the Internal Neuron Representations are exploited to provide informative and stable information for classifier analysis. InnerCIL consists of two key components, Attention-conditioned Native Dictionary Generation (ANDG) and Fisher Refinement Analytic Classification (FRAC). Specifically, instead of merely utilizing output features, ANDG introduces a Neuron-guided attention mechanism to extract a set of local representations within middle layers. These features form a native dictionary where no additional transform is adopted. Then, based on the dictionary containing local and global features of different stages, FRAC filters a subset to accelerate classifier analysis while mitigating the noise samples that can misguide the classifier. As a result, InnerCIL is both effective and efficient. Experiments on different backbones and benchmarks show that our method outperforms existing methods by a large margin.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, class incremental learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_vxegOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes InnerCIL, an analytic class-incremental learning method that constructs a richer frozen representation from the internal neurons of a PTM. The method combines multi-depth CLS features with attention-pooled MLP neuron responses from intermediate layers. A checkpoint-based registration step selects a fixed candidate dictionary, Visual Fisher then selects discriminative local coordinates at each incremental stage, and a closed-form ridge classifier is fit from accumulated statistics without replay data. The paper provides theoretical motivation for accessing intermediate neuron responses and empirical results on CLIP and DINOv2 backbones.

Critical strengths
  1. The core representation idea is interesting and motivated. In particular, reading individual MLP neuron responses before the output projection gives access to information that may be lost by the normal synthesis operations, and the paper provides an ablation for the synthesis-null component argument.
  2. The method is straight forward from an optimization perspective: the backbone remains frozen and the classifier is reconstructed analytically from statistics. This avoids representation drift and preserves exact batch-equivalence to the ridge-regression model.
  3. The ablations are useful and support several parts of the design. In particular, table 3 shows gains from both multi-depth CLS features and the native local dictionary, while table 4 analyzes native versus random features, attention versus uniform pooling, and the synthesis-null component.
Critical weaknesses
  1. The main scalability issue is the size of the Gram matrix. Although Fisher reduces the deployed dimension from P to K, the method must retain G_t^{raw} over the pre-Fisher candidate representation so that a different Fisher support can be selected at later steps. Therefore, Fisher does not remove the quadratic O(D_P^2) memory bottleneck. This may become problematic for wider backbones, more layers, or larger native dictionaries.
  2. Fisher is a univariate selection criterion: each coordinate is scored independently using between-class versus within-class variance. Two individually strong coordinates may be highly redundant, while two individually mediocre features may become useful when considered jointly. The Fisher objective differs from the joint multivariate objective when coordinates are correlated. This is also interesting considering prior analytic methods, for example RanPAC, which uses random projections to capture nonlinearity between feature dimensions; could such a component complement the proposed native features?
  3. I have some concerns about comparison fairness with the baselines. For example, table 3 shows a large improvement by replacing final CLS with multi-depth CLS, from 73.05 to 77.50 last accuracy. Multi-depth representations are not specific to InnerCIL and could be incorporated into several competing analytic methods. It would be useful to compare against stronger versions of the baselines using multi-depth global representation.
  4. More broadly, a significant part of the contribution can be viewed as constructing a stronger representation by combining multi-layer global and local features. I am not very familiar with the broader literature on multi-level feature fusion, but there appears to be relevant prior work that deserves more discussion and potentially comparison. For example, Chen et al. [1] study hierarchy-aware aggregation of transformer features, while Lin et al. [2] study layer selection and fusion of multi-layer visual representations. Although these works consider different settings, they seem relevant for understanding how much of the gain comes from the proposed native-neuron representation versus the more general benefit of aggregating features across layers.

[1] Yongjie Chen, Hongmin Liu, Haoran Yin, and Bin Fan. Building Vision Transformers with Hierarchy Aware Feature Aggregation. ICCV 2023.

[2] Junyan Lin, Haoran Chen, Yue Fan, Yingqi Fan, Xin Jin, Hui Su, Jinlan Fu, and Xiaoyu Shen. Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practices. CVPR 2025.

Questions for authors
  1. Have you considered combining the proposed global/local native representation with an additional nonlinear feature expansion such as the random projection used in prior analytic CIL methods? The dimensionality could become large, but this might capture interactions that the current univariate Fisher selection and linear ridge readout cannot represent. I would be interested to know whether native features and random nonlinear features are complementary.
  2. What happens if Fisher filtering is removed and the classifier is fit using all P registered native coordinates? Is Fisher mainly included to reduce the final ridge-solve dimension, or does it improve accuracy? Since the method already stores the Gram statistics before Fisher selection, an ablation comparing all P coordinates vs. Fisher-K would clarify this.
4 comments
AuthorsOpenReview ↗

Response to W1, W2, Q1, and Q2

Thank you for recognizing the native-neuron representation, analytic reconstruction, and supporting ablations.

W1 + Q2: Raw Gram memory and the effect of removing Fisher

With global width dgd_g, the raw Gram matrix remains quadratic in the candidate dimension DP=dg+P+1D_P=d_g+P+1: dynamic Fisher reselection requires preserving statistics for every registered candidate. Packed upper-triangular storage uses DP(DP+1)/2D_P(D_P+1)/2 entries (Appendix A.5, Eq. 46).

The two budgets control different costs. Checkpoint registration limits the native dictionary to PP atoms, so wider backbones or more layers do not require retaining every available neuron. Fisher then selects KK coordinates, reducing the dense solve to O(DK3)O(D_K^3) and deployed classifier storage to O(DKCt)O(D_KC_t). Here DK=dg+K+1D_K=d_g+K+1 and CtC_t is the number of seen classes. Wider backbones and more global blocks also increase dgd_g; both global width and local budget must therefore be controlled.

We vary both budgets on DINOv2/CUB:

Budget P/KP/KFinal acc. (%)Persistent state (MiB)Select + solve (s)
4096/307290.30119.227.97
8192/614490.48277.3720.58
11264/870490.61438.4132.55

The medium budget reduces state by 36.7% with only 0.13 percentage points (pp) less accuracy; the smallest budget reduces state by 72.8% and selection/solve time by 75.5%, retaining 90.30% accuracy. Low-rank or block-diagonal compression of second-order statistics is a further direction discussed in Sec. 5.

Removing Fisher. We fit the classifier using all P=11264P=11264 native coordinates, keeping the global branch, RMS rules, training views, and ridge penalty identical. Fisher retains K=8704K=8704 coordinates:

BackboneAll-P acc.Fisher-K acc.All-P time (s)Fisher-K time (s)
CLIP82.7982.8844.9025.10
DINOv290.8590.9259.0932.55

Reducing local width by 22.73% lowers selection/solve time by 44.1% on CLIP and 44.9% on DINOv2, while also reducing deployed classifier storage and multiplication cost. Mean final accuracy improves by 0.09/0.07 pp.

The raw candidate Gram remains stored for future reselection; these savings concern the selected solve and deployed classifier.

Candidate budgeting controls raw-state memory; Fisher provides cheaper analytic solves and a modest accuracy improvement at the deployed dimension.


W2 + Q1: Univariate selection, feature correlations, and nonlinear complementarity

Fisher ranks coordinates individually; the full-Gram ridge solve models correlations among the retained coordinates, including global–local and local–local covariances (Appendix A.2, Theorem 4 and Eq. 42). It cannot recover useful coordinates omitted during selection, so we directly test redundancy-aware selection.

At fixed P=11264P=11264, K=8704K=8704, and λ=0.003\lambda=0.003, we compare Random-K, maximum-variance selection, Visual Fisher, and greedy Fisher that subtracts 0.25 times the maximum squared correlation with selected features from the normalized Fisher score:

Selection criterionCLIP-CIFARCLIP-CUBMean
Random-K85.4082.7884.09
Variance-K85.1983.3384.26
Redundancy-penalized Fisher85.2983.4784.38
Visual Fisher85.2683.9384.60

Visual Fisher achieves the highest mean and exceeds the correlation-penalized variant by 0.46 pp on CUB. Correlated responses can form useful difference combinations that cancel shared background variation; penalizing their correlation during selection can discard such information before the joint classifier uses it.

To test nonlinear complementarity, we combine the global/local native representation u(x)u(x) with a fixed Gaussian–ReLU map q(x)=ReLU⁡(R⊤u(x))q(x)=\operatorname{ReLU}(R^\top u(x)). Input scales use only first-task training data, and RR stays fixed. On CUB, all variants retain the same four-layer CLS branch, λ=0.003\lambda=0.003, and total local budgets P=11264P=11264, K=8704K=8704.

Native Only assigns the whole local budget to native neurons; RP Only assigns it to random features. The hybrid splits both budgets equally: P=5632+5632P=5632+5632, K=4352+4352K=4352+4352.

Local representationCLIP avg./final acc.DINOv2 avg./final acc.RP storage (MiB), CLIP/DINOv2
RP Only86.07/80.8193.60/90.49572.0/616.0
Native + RP87.09/81.7793.81/90.65286.0/308.0
Native Only88.93/83.9393.76/90.610.0/0.0

The hybrid gives DINOv2 the highest average and final accuracy, improving over Native Only by 0.05/0.04 pp. This shows a small complementary gain in this DINOv2 setting. On CLIP, however, assigning half the budget to RP displaces useful native responses, and Native Only performs best.

Native Only exceeds RP Only on both backbones and requires no additional projection weights; the hybrid adds 286–308 MiB.

Visual Fisher gives the best mean accuracy in the selection comparison, while the RP hybrid adds a small DINOv2 gain at extra storage cost; native responses remain the efficient default.

AuthorsOpenReview ↗

Response to W3 and W4

W3: Fairness against stronger multi-depth CLS baselines

Following your suggestion, we strengthen Direct Ridge and RanPAC with the same L2-normalized CLS concatenation from blocks 9–12 used by InnerCIL. The frozen backbone, class order, and training views are shared. Baseline regularization is selected on a first-task training/validation split; RanPAC uses a fixed 7,168-dimensional random ReLU expansion. InnerCIL retains its default configuration.

Method and global featuresCLIP-CIFARCLIP-CUBDINOv2-CIFARDINOv2-CUBMean
Direct Ridge, final CLS76.7076.5487.9588.5782.44
Direct Ridge, multi-depth CLS81.3978.8989.1090.0484.86
RanPAC, final CLS81.1381.9089.5990.2785.72
RanPAC, multi-depth CLS81.9680.3290.2390.5185.76
InnerCIL, multi-depth CLS + native neurons85.2683.9391.5990.6187.85

Final accuracy (%) after ten incremental stages.

Multi-depth CLS improves Direct Ridge throughout and RanPAC in three of four settings. InnerCIL still leads in every setting, exceeding the multi-depth RanPAC baseline by 2.09 pp in mean accuracy. Tables 3 and 10 also isolate the native dictionary's contribution while holding the multi-depth global branch fixed.

The advantage persists after giving competing analytic baselines the same multi-depth global representation, directly identifying the added value of native local responses.


W4: Relation to multi-level feature aggregation and fusion

We have added the two works you recommended to Sec. 2. Their connection to InnerCIL is the use of information from multiple depths; the key differences concern what information is read and how it is learned.

Chen et al.'s HAFA (ICCV 2023) changes token aggregation within a trained ViT through local aggregation in shallow layers and semantic aggregation in deeper layers. InnerCIL leaves the pretrained forward computation unchanged, reuses existing attention and MLP activations, and reconstructs the incremental classifier without gradient-based backbone training.

Lin et al. (CVPR 2025) study layer selection and fusion of block-output patch representations for multimodal language models, with trainable projectors and instruction tuning. InnerCIL reads individual MLP responses before output projection and uses cumulative sufficient statistics instead of training a fusion network.

The empirical distinction is isolated in Table 7: with the multi-depth global branch, selected coordinates, scaling, dimension, and regularization held fixed, restoring the pre-projection null component improves accuracy in all 16 backbone–dataset settings. Thus, the comparison tests information lost by output projection while keeping cross-layer aggregation unchanged.

InnerCIL combines a frozen, replay-free analytic learner with access to pre-projection information that conventional block-output fusion cannot directly recover.

Dear Reviewer vxeg,

Thank you for your time and effort in reviewing our work. As the rebuttal period concludes within two days, we wish to respectfully confirm whether our responses have adequately addressed your initial concerns. Any feedback would be greatly appreciated.

Sincerely,

The Authors

I thank the authors for their detailed rebuttal and additional experiments. I appreciate their efforts to address my concerns, particularly the stronger baseline comparisons and additional ablations. My remaining concern is primarily about the nature of the contribution, which I still view as largely constructing a stronger feature representation through multi-layer global and local feature aggregation, rather than introducing a fundamentally new learning framework. While the authors have clarified the distinctions from related work, I believe this perspective remains relevant when assessing the novelty. Overall, my assessment remains positive, and I will maintain my original score.

Reviewer_a47pOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes InnerCIL for exemplar-free class-incremental learning. The authors explore internal neuron responses from frozen pretrained vision transformers instead of only using the final output features. They introduce Attention-conditioned Native Dictionary Generation to extract local neuron representations using CLS-to-patch attention. They also propose Fisher Refinement Analytic Classification to select discriminative features and construct an analytic classifier. The method maintains cumulative statistics without storing previous training samples. The authors evaluate on multiple datasets using CLIP and DINOv2 backbones and report strong results. However, I have some concerns about the theoretical motivation, feature selection strategy, and scalability.

Critical strengths

The authors explore internal neuron responses for analytic class-incremental learning. Using pretrained hidden representations instead of only final output features is an interesting direction. The method also maintains a strict replay-free learning setting.

InnerCIL achieves strong performance across many benchmarks. The authors evaluate with both CLIP and DINOv2 backbones and compare against several recent methods. The results show consistent improvements in average and final accuracy.

The authors provide detailed ablation studies and theoretical analysis. They investigate native neuron features, attention pooling, Fisher selection, and synthesis-null information. The paper also includes computational and storage comparisons.

Critical weaknesses
  1. The authors argue that internal neuron responses preserve useful information that the output projection removes. However, the theoretical accessibility of such information does not guarantee its discriminative value or generalization. Although the authors provide supporting analysis, the connection between synthesis-null information and better recognition remains not fully explained.

  2. The Fisher refinement module selects neurons based on their individual class separation scores. This strategy ignores correlations and redundancy among different neuron features. The model may select highly similar neurons while missing complementary features that have lower individual scores.

  3. The authors use CLS-to-patch attention to aggregate internal neuron responses. However, attention weights do not necessarily reflect the importance of local regions for downstream classification. This may suppress useful fine-grained features when discriminative regions receive low attention.

  4. The authors emphasize analytic closure as an important theoretical property. However, recovering the joint ridge regression solution from cumulative sufficient statistics is already a standard property of analytic incremental learning. The paper mainly extends the feature representation rather than introducing a fundamentally new analytic learning principle.

  5. The authors construct a large feature dictionary and maintain a high-dimensional Gram matrix throughout incremental learning. The storage cost grows quadratically with the candidate feature dimension, which may limit scalability. Although the method shows good efficiency against AnaCP, its computational benefits remain unclear for much larger backbones and feature dictionaries.

Questions for authors

As in weaknesses.

4 comments
AuthorsOpenReview ↗

Response to W1 and W2

Thank you for recognizing the native-neuron representation, replay-free framework, and empirical and theoretical analysis.

W1: From synthesis-null accessibility to discriminative value and generalization

Internal responses contribute when they contain class information unexplained by the global representation. For our ridge classifier (Appendix A.1, Eqs. 14–16), let H,ZH,Z denote global and internal features, YY the one-hot targets, and λ>0\lambda>0 the ridge penalty. With

F=Z^\top R_HY,\quad S=Z^\top R_HZ+\lambda I,$$ adding $Z$ reduces the optimal regularized objective by $$\Delta\mathcal L=\operatorname{tr}(F^\top S^{-1}F)\ge0,\qquad \Delta\mathcal L>0\iff F\ne0.$$ This identifies when internal responses, including their synthesis-null component, improve the analytic objective. We test the benefit on unseen images through a residual ablation. Using training data only, we fit a ridge predictor $B$ from last-four-block CLS features $H$ to internal responses $Z$. We compare the predicted component $HB$, residual $Z-HB$, and full response under matched global features, local budget, and classifier settings: |Classifier input|CLIP-CUB|CLIP-CIFAR|DINOv2-CUB|DINOv2-CIFAR| |---|---:|---:|---:|---:| |CLS: $H$|78.77|81.43|89.97|89.39| |CLS + predicted: $[H,HB]$|78.77|81.43|89.89|89.44| |CLS + residual: $[H,Z-HB]$|80.76|82.18|90.47|89.61| |**CLS + full response: $[H,Z]$**|**83.93**|**85.26**|**90.61**|**91.59**| *Final accuracy (%); $P=11264$, $K=8704$, $\lambda=0.003$.* Predicted responses add almost no benefit, whereas the residual improves every setting; the complete response performs best throughout. This identifies useful information beyond multi-depth CLS rather than an effect of feature width alone. Table 6 measures label-aligned objective reduction from actual synthesis-null responses; Table 7 restores this component while fixing neuron indices, RMS scales, classifier dimension, and regularization. Restoration improves test accuracy in all 16 backbone–dataset settings. **Internal responses contribute both to fitting class targets and to recognizing unseen images, connecting synthesis-null accessibility to measurable classification gains.** --- ## W2: Feature correlations, redundancy, and complementary neurons Visual Fisher ranks coordinates individually; the full-Gram ridge readout models correlations among the retained features. Complementary coordinates omitted by screening remain outside that readout, so the selection criterion trades joint subset optimization for low-cost screening. For a selected set $\mathcal S$, the conditional benefit of adding coordinate $z_j$ depends on the residual after fitting that set: $$u(j\mid\mathcal {S})=\frac{\|z_j^\top R_{H_{\mathcal {S}}}Y\|^2}{z_j^\top R_{H_{\mathcal {S}}}z_j+\lambda}.$$ Here $H_{\mathcal S}$ contains the global and selected local features, and $R$ is the regularized residual operator defined in W1. Updating these conditional scores throughout selection is substantially more expensive than maintaining Fisher's per-class moments. Appendix A.2, Theorem 4 (Eqs. 36–38) gives the additive risk interpretation and a spectral bound relating coordinate selection to the joint objective under feature correlations. High correlation can also reflect shared interference rather than useless duplication. For example, $z_1=y+n$ and $z_2=n+0.5y$ share background variation $n$, but their difference isolates $0.5y$. Joint readout can use such combinations to suppress interference; penalizing correlation during selection can discard useful partners. We compare Random-$K$, maximum-variance selection, greedy Fisher with a correlation penalty, and Visual Fisher under the same $P=11264$, $K=8704$, and $\lambda=0.003$. The greedy variant maximizes normalized Fisher score minus $0.25$ times the maximum squared correlation with an already selected coordinate. |Selection|Complexity|CLIP-CIFAR|CLIP-CUB|Mean| |---|---|---:|---:|---:| |Random-$K$|$O(P)$|**85.40**|82.78|84.09| |Variance-$K$|$O(P)$|85.19|83.33|84.26| |Fisher + correlation penalty|$O(PKD_P)$|85.29|83.47|84.38| |**Visual Fisher + joint readout**|$O(PC_t)$|85.26|**83.93**|**84.60**| *Final accuracy (%); $C_t$ is the seen-class count and $D_P$ the candidate representation width.* Explicit correlation penalization increases selection complexity and lowers CUB accuracy by 0.46 percentage points relative to Visual Fisher, with a lower two-dataset mean. **Fisher gives the best mean accuracy in this comparison at low selection cost, while the full-Gram readout jointly models the retained neurons.**
AuthorsOpenReview ↗

Response to W3, W4, and W5

W3: Does low CLS-to-patch attention suppress discriminative regions?

A patch's class contribution combines attention, neuron responses, and class-specific weights. Very low attention attenuates a patch; averaging 12 heads across the last four blocks reduces dependence on any single attention map (Eq. 4).

Expanding the local class score makes this interaction explicit:

\sum_{b,p}a_{b,p}(x) \sum_{r\in\mathcal S_b}c_{t,(b,r)} \theta_{t,(b,r),c}f_{b,r}(u_{b,p}(x)).$$ Here $\mathcal S_b$ contains the selected neurons of block $b$, $f_{b,r}(u_{b,p})$ is neuron $r$'s response to patch $p$, $a_{b,p}$ is attention, $c_{t,(b,r)}$ is RMS scaling, and $\theta_{t,(b,r),c}$ is the class-specific weight. A small region with selective responses can contribute strongly through the inner sum. Attention provides spatial weighting; the analytic classifier determines which responses support each class. Tables 4 and 11 test attention's downstream effect against uniform pooling under otherwise identical settings: |Dataset|CLIP gain|DINOv2 gain| |---|---:|---:| |CUB-200|+11.82|+1.76| |Aircraft|+6.24|+5.07| |Cars|+2.28|+2.30| |Eight-dataset mean|+3.39|+1.39| *Final-accuracy gain in percentage points over uniform pooling.* Figure 4's target-logit maps locate the strongest contributions on bird heads and breast feathers, aircraft wings and engines, and car headlights. **The class-specific readout converts attention-weighted native responses into strong fine-grained recognition, outperforming uniform pooling on both backbones.** --- ## W4: Analytic closure versus the method's main contribution We agree that joint ridge recovery from cumulative statistics is an established property of Analytic CIL. InnerCIL contributes the construction and analysis of the native-neuron dictionary, together with preserving that property when the selected neurons and their scales change across stages. The deployed feature map is stage-dependent: new classes change the top-$K$ support $J_t$ and cumulative RMS scales. Statistics stored only for currently selected coordinates would not recover historical correlations involving a newly recruited neuron. InnerCIL retains raw Gram and feature–label statistics over a fixed candidate pool and reconstructs the current system with the selection/scaling operator $T_t=E_tR_t$: $$G_t=T_t^\top G_t^{\mathrm{raw}}T_t,\qquad Q_t=T_t^\top Q_t^{\mathrm{raw}},\qquad \Theta_t=(G_t+\lambda I)^{-1}Q_t.$$ Appendix A.2, Eq. 30 and Appendix A.3, Eq. 45 show that this equals the joint ridge solution under the current deployed representation without revisiting old samples. **InnerCIL combines richer native features with stage-adaptive selection while retaining exact analytic updates throughout the incremental sequence.** --- ## W5: Quadratic Gram storage and scalability As you point out, Gram storage remains quadratic in the candidate representation. Let $d_g$ be the global feature dimension, $P$ the candidate local budget, $K$ the deployed local budget, and $C_t$ the class count. With $D_P=d_g+P+1$ and $D_K=d_g+K+1$, the main stored scalar counts are $$\frac{D_P(D_P+1)}2+D_PC_t+(2P+1)C_t+D_KC_t,$$ corresponding to packed symmetric Gram storage, label cross-statistics, class moments, and classifier weights (Appendix A.5, Eq. 46). Registration caps the local candidate pool at $P$; Fisher reduces the deployed width $K$ and hence solve cost, but not the raw Gram dimension. Wider backbones or more global blocks also increase $d_g$, so fixed $P$ alone does not keep total storage constant. Scaling requires budgeting both the global readout and the local candidate pool. The DINOv2 ViT-B/CUB results quantify how candidate and deployment budgets control this trade-off: |Budget $P/K$|Final acc. (%)|State (MiB)|Select + solve (s)| |---|---:|---:|---:| |4096/3072|90.30|**119.22**|**7.97**| |8192/6144|90.48|277.37|20.58| |11264/8704|**90.61**|438.41|32.55| *Selection/solve time is cumulative over ten stages on an eight-thread CPU.* The middle budget reduces state by 36.7% and selection/solve time by 36.8%, with a 0.13-point accuracy difference. The smallest budget retains 90.30% accuracy using 119.22 MiB. **Explicit candidate and deployment budgets provide substantial measured memory and solve-time savings with small accuracy changes.**
Reviewer_a47pOpenReview ↗

Thank you for the response

Thank you for the detailed responses and additional experiments. The rebuttal substantially alleviates my concerns, I keep my positive score.

Dear Reviewer a47p,

We sincerely appreciate your recognition of our work. Your insightful comments have been invaluable in refining our presentation and strengthening the manuscript.

Thank you again for your thoughtful engagement.

Best regards,

The Authors

Reviewer_1ZpGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

InnerCIL studies strict exemplar-free class-incremental learning with a frozen pretrained ViT. It constructs a dictionary by pooling intermediate MLP hidden-neuron responses with CLS-to-patch attention and combining them with multi-depth CLS features. A checkpoint-only rule registers candidate coordinates; cumulative class statistics then rank local coordinates using a Fisher criterion. Support selection and RMS scaling are applied to a retained raw Gram matrix and feature-label cross-statistic, allowing the classifier to recover the batch ridge solution for the stage-specific deployed representation without retaining individual old examples or features. The paper also analyzes synthesis-null hidden responses that are removed by the MLP output projection.

Critical strengths
  1. A useful representation improvement supported by broad, relevant controls. Tables 1-2 report improvements over the complete AnaCP* pipeline of 1.54/2.48 percentage points in mean stage/final accuracy with CLIP and 0.69/0.72 points with DINOv2. The three-seed results in Table 12 retain a positive average advantage. More consequentially, the equal-budget replacements in Tables 4 and 11 support the value of patch-level responses and attention pooling beyond merely concatenating more CLS features. Restoring the selected synthesis-null component improves final accuracy in all 16 backbone-dataset cases in Table 7. Together these results make the representation contribution credible, although the exact source of its advantage could be isolated further.

  2. A principled way to change the deployed feature subset without replay. Equations (8), (30), and (45) explain why retaining statistics over a fixed raw dictionary permits later support changes and rescaling without revisiting old samples. This addresses an important consistency issue that would arise if only statistics over the currently selected coordinates were stored. The batch equivalence is correctly restricted to the same realized training views and deployed representation; the paper also distinguishes strict replay-free AnaCP from its full feature-generation pipeline.

Critical weaknesses
  1. The specific role of pretrained neuron identity and checkpoint registration remains incompletely isolated. The post-CLS random ReLU control changes both the functions and the information they receive. It therefore does not distinguish the value of pretrained neuron functions from the value of accessing contextualized patch states. A fixed random-function dictionary evaluated on those same patch states, using the same attention, capacity, scaling, and analytic readout, would directly test this distinction. Moreover, Lemma 2 and Figure 7 assess preservation of checkpoint write-back variation, whereas the central motivation emphasizes information outside that write-back. They do not establish that the registration rule preserves the best discriminative native coordinates. This limits the mechanistic interpretation and strength of the novelty claim, although it does not negate the measured gains.

  2. Important comparison and implementation choices remain insufficiently specified. Appendix A.4 gives shared views, class orders, and the proposed method's capacities, but does not provide a per-baseline configuration or tuning protocol. In particular, the random-feature widths, regularization choices, and first-session adaptation settings of the closest analytic baselines are unclear. Exact checkpoint identifiers, intermediate CLIP readout details, numerical floors, and the analytic solver/precision are also needed for faithful reproduction. Table 8 varies P and K at a fixed lambda, leaving the regularization sensitivity unresolved. This matters especially for the smaller DINOv2 margins. The reported efficiency advantage is also relative to AnaCP*, whose feature-generation step is excluded by the paper's strict protocol; it does not establish the same advantage over the closest strict replay-free alternatives.

Questions for authors
  1. Can you compare pretrained native responses with fixed random nonlinear functions of the same intermediate patch states, holding attention, P/K, RMS scaling, and ridge fitting fixed? Can you also report downstream accuracy for checkpoint versus random registration at the same P/K, rather than only omitted write-back variation? These controls would clarify how much of the gain specifically requires the proposed native dictionary and registration rule.

  2. Please provide the baseline configurations and model-selection protocol, including feature widths, regularization, first-session adaptation, exact checkpoints, and how B/P/K/lambda were selected. Which seed or aggregation is used in Tables 1-2? Does the average advantage persist under a common, training-only validation protocol and a modest lambda sweep? This request is for verification of the reported comparison, not an allegation of test-set tuning.

  3. What are the extraction time, classifier-construction time, and persistent state for AnaCP without feature generation, RanPAC, and LayUP under the same measurement definitions? Please also specify solver precision and report peak working memory and test-time throughput on representative datasets. These measurements would determine the scope of the efficiency claim, since current totals exclude evaluation and temporary workspace.

5 comments
AuthorsOpenReview ↗

Response to W1, Q1, and Q3

Thank you for your constructive feedback.

W1 + Q1: Pretrained neuron functions and Checkpoint registration

Following your suggestion, we compare native neurons with fixed random nonlinear functions on the same LN2 patch states from the last four blocks. Attention, the multi-depth CLS branch, P=11264P=11264, K=8704K=8704, RMS scaling, and λ=0.003\lambda=0.003 are fixed. The control replaces neuron input directions with row-norm-matched Gaussian directions, retaining the biases and activation functions without altering the backbone's forward pass.

Same-patch readoutCLIP-CIFARCLIP-CUBDINOv2-CIFARDINOv2-CUBMean
Fixed random functions85.0583.8591.4290.4287.69
Pretrained native neurons85.2683.9391.5990.6187.85

Native neurons improve all four settings while reusing activations already computed by the pretrained MLP, with no extra projection weights or projection computation.

Checkpoint registration is computed once from pretrained weights, requiring no training, calibration data, or additional forward passes. Lemma 2 shows that it minimizes the stated upper bound on omitted write-back variation; Figure 7 reports only about 4–6% normalized omission, close to the direct write-back-norm reference.

CLIP dataset, P/K=11264/8704P/K=11264/8704CheckpointRandom
CIFAR-10085.2685.25±0.10
CUB-20083.9383.83±0.07

Checkpoint registration efficiently preserves the pretrained computation; Visual Fisher then selects task-discriminative coordinates from that fixed candidate pool.


Q3: Efficiency against replay-free baselines, peak memory, and throughput

For the replay-free comparison you requested, we evaluate InnerCIL alongside AnaCP without feature generation (FG), frozen RanPAC, and LayUP on the same RTX 4090, eight CPU threads, batch size 64, and DINOv2 checkpoint over ten stages.

Extraction includes image loading, augmentation, and backbone passes for training and test views. First-session adaptation (FSA) is timed separately; construction includes statistic updates, selection/tuning, and solves. Total time also includes stage-wise evaluation.

Dataset/methodFinal acc.Extract (s)FSA (s)Construct (s)Total (s)
CIFAR/InnerCIL91.59171.850.0014.56194.52
CIFAR/AnaCP w/o FG91.03215.4675.582.49309.17
CIFAR/LayUP91.38139.26163.1821.77331.30
CIFAR/RanPAC frozen90.15130.790.005.15144.14
CUB/InnerCIL90.6132.580.0011.7350.67
CUB/AnaCP w/o FG90.5934.5419.831.4058.23
CUB/LayUP90.0226.3630.437.8967.44
CUB/RanPAC frozen90.4029.220.001.0335.29

Final accuracy (%). InnerCIL uses FP32 statistics and FP64 Cholesky solving; LayUP uses FP64, and RanPAC/AnaCP use FP32.

Persistent state includes statistics, adaptation/projection parameters, and classifiers, excluding the frozen backbone and offline caches. GPU peak is maximum allocated pipeline memory; CPU peak is process RSS, including workspace and caches. Throughput includes the full forward pass and readout on GPU-resident batches.

Dataset/methodState (MiB)GPU peak (GiB)CPU peak (GiB)Images/s
CIFAR/InnerCIL415.364.4811.761225.92±14.43
CIFAR/AnaCP w/o FG478.347.886.31798.22±0.06
CIFAR/LayUP173.574.267.051535.91±13.54
CIFAR/RanPAC frozen222.473.373.741698.81±1.18
CUB/InnerCIL438.414.495.211233.83±1.33
CUB/AnaCP w/o FG485.247.882.44801.11±0.23
CUB/LayUP180.604.263.741558.45±1.22
CUB/RanPAC frozen227.941.062.831704.02±1.55

Throughput: mean±standard deviation over three timing windows; InnerCIL uses P/K=11264/8704P/K=11264/8704.

InnerCIL is 37.1%/13.0% faster end to end than AnaCP without FG and 41.3%/24.9% faster than LayUP on CIFAR/CUB, avoiding first-session adaptation. It also uses less GPU memory and delivers higher throughput than AnaCP without FG. Frozen RanPAC is faster and lighter; InnerCIL gains 1.44/0.21 percentage points in accuracy over it while sustaining over 1,225 images/s.

AuthorsOpenReview ↗

Response to W2 and Q2

W2 + Q2: Baseline configurations, model selection, and regularization sensitivity

Checkpoints and readout. CLIP uses OpenAI ViT-B/16 (openai/clip-vit-base-patch16); DINOv2 uses Meta dinov2_vitb14 (facebook/dinov2-base), without register tokens. Both have hidden/MLP widths 768/3072. For blocks B={8,9,10,11}\mathcal B=\{8,9,10,11\} (0-indexed), we apply the final LayerNorm to each CLS state; CLIP additionally uses its fixed 768×512768\times512 visual projection. Each block is L2-normalized, then concatenated and scaled by 1/21/2 (Eq. 3). Global widths are 2048/3072 for CLIP/DINOv2; with K=8704K=8704 and a bias, deployed widths are 10753/11777.

Floors are εh=10−12\varepsilon_h=10^{-12}, εLN=10−5\varepsilon_{\mathrm{LN}}=10^{-5} for CLIP/10−610^{-6} for DINOv2, and εF=εRMS=10−8\varepsilon_F=\varepsilon_{\mathrm{RMS}}=10^{-8}. Extraction and raw-statistic accumulation use FP32; the ridge system is converted to FP64 for Cholesky solving.

Defaults and baselines. We chose B\mathcal B, PP, and KK empirically through detailed testing of many block/budget combinations. We then fixed the last four blocks, P/K=11264/8704P/K=11264/8704, and λ=0.003\lambda=0.003 across all 16 backbone–dataset settings and all seeds, without first-session adaptation (FSA) or feature generation (FG). Appendix A.4, Table 8/Figure 8 reports capacity sensitivity; the regularization sweep below evaluates the shared λ\lambda. Tables 1–2 use seed 1993; Table 12 reports mean±SD over seeds 1993, 2024, and 3407.

MethodFeaturesFSA/FGRegularization/solve
RanPACFinal CLS; Gaussian+ReLU width 10000; frozen control 7168Original first-session adapter protocol where enabled; frozen control; no FGPer-task 80/20 training split, validation MSE; λ∈{10−8,…,108}\lambda\in\{10^{-8},\ldots,10^8\}; FP32 linear solve
LoRanPACGaussian+ReLU width 100000; continual truncated SVD, 25% truncation; rank cap 10000 (20000 on ObjectNet/OmniBenchmark)Original first-session adapter protocol where enabled; no FGOriginal λ=0\lambda=0; FP32 SVD/QR; no per-task ridge search
LayUPLast 6 CLS blocks, 4608 dimensions; second-order prototype expansionAdaptFormer, bottleneck 64; 20 epochs, AdamW, lr 0.003, wd 0.0005; no FGFirst-task validation accuracy; 15 log-spaced values 10−410^{-4}–10310^3, plus 10−810^{-8}; FP64
AnaCP*Frozen+adapted CLS: 1536 DINOv2/512 CLIP; three 5000-d CP heads + 5000-d GELU-ELMRank-8 adapter, 10 epochs, Adam lr 0.0001 on DINOv2; 100 Gaussian pseudo-features/class, shared covarianceFixed λCP=λELM=100\lambda_{\rm CP}=\lambda_{\rm ELM}=100; FP32
AnaCP without FGSame dual branch and three 5000-d CP headsRetains FSA; removes sampling and ELM; analytic/NCM headFixed λCP=100\lambda_{\rm CP}=100; FP32

In their adapter configurations, RanPAC/LoRanPAC use bottleneck 64, trained on the first session for 20 epochs with SGD (lr 0.01, momentum 0.9, weight decay 0.0005, batch 48), then frozen. RanPAC's original CLIP configuration uses a frozen encoder without FSA.

Sensitivity. At fixed P/KP/K, we sweep λ\lambda over both backbones and all eight datasets:

λ\lambda0.00030.0010.0030.010.030.1
CLIP82.6582.8382.8882.8482.7182.39
DINOv290.7690.8990.9290.9090.8190.58

Mean accuracy varies by only 0.23/0.16 percentage points on CLIP/DINOv2 over λ∈[0.0003,0.03]\lambda\in[0.0003,0.03], placing the shared default in a broad, stable region.

Common training-only validation. On all eight DINOv2 datasets, we use a stratified 80/20 split of first-task training images, keeping all three views of each image together. We select λ\lambda from {10−4,3×10−4,10−3,3×10−3,10−2,3×10−2,10−1}\{10^{-4},3\times10^{-4},10^{-3},3\times10^{-3},10^{-2},3\times10^{-2},10^{-1}\}, including each baseline's standard grid where applicable, then hold it fixed throughout the ten-task sequence.

MethodPaper defaultCommon validation
RanPAC92.19/88.2292.31/88.35
LayUP93.15/89.3493.15/89.34
AnaCP without FG92.88/89.4892.94/89.55
AnaCP*93.52/90.2093.56/90.24
InnerCIL94.21/90.9294.18/90.89

InnerCIL retains the highest average and final accuracy under common training-only selection, while its fixed configuration remains stable across a two-order-of-magnitude regularization range.

Dear Reviewer 1ZpG,

Thank you for your time and effort in reviewing our work. As the rebuttal period concludes within two days, we wish to respectfully confirm whether our responses have adequately addressed your initial concerns. Any feedback would be greatly appreciated.

Sincerely,

The Authors

Reviewer_VAkXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes InnerCIL, an analytic class-incremental learning method that exploits internal neuron representations from a frozen ViT. The method consists of two components: Attention-conditioned Native Dictionary Generation (ANDG), which extracts attention-weighted MLP neuron responses from intermediate layers, and Fisher Refinement Analytic Classification (FRAC), which selects discriminative features for ridge regression. Experiments on CLIP and DINOv2 backbones show improvements over existing methods across multiple CIL benchmarks.

Critical strengths

Exploring internal MLP neuron responses rather than relying solely on final-layer representations is an interesting direction for analytic CIL.

The paper provides theoretical analysis of information preserved in the MLP synthesis-null space, supported by corresponding experiments.

The experiments cover two pretrained backbones and multiple benchmarks. The progressive and component-replacement ablations provide useful evidence for the effectiveness of the proposed representations.

Critical weaknesses

The motivation is not sufficiently supported. The paper argues that existing feature transformations and sampling strategies introduce feature drift or noise, leading to confused decision boundaries. However, these problems are not directly measured or demonstrated. It is therefore unclear whether InnerCIL actually addresses these claimed limitations.

The novelty of the representation design needs further justification. The main performance gains come from multi-depth CLS features and intermediate MLP neuron responses. Although the paper compares against random projections, uniform pooling, and visible-only neurons, it lacks direct comparisons with simpler alternatives such as pooled intermediate patch features or multi-layer MLP outputs. It remains unclear whether neuron-level extraction is necessary.

The checkpoint-only registration criterion is not well aligned with its intended purpose. The proposed score is derived to preserve MLP write-back variation, rather than downstream class-discriminative information. Figure 7 validates the former, but does not establish that this selection criterion improves classification accuracy over other registration strategies. Moreover, the use of write-back magnitude appears insufficiently connected to the paper's motivation of exploiting synthesis-null information.

The contribution of FRAC is relatively limited. Tables 3 and 10 show that replacing Random-K with Visual Fisher improves average final accuracy by only 0.54 and 0.23 percentage points on CLIP and DINOv2, respectively. Since Fisher-based feature selection is a standard technique, both its novelty and its practical necessity are questionable. The experiments also do not directly demonstrate the claimed noise filtering or improved robustness.

The theoretical analysis does not fully establish the claimed classification benefits. Theorem 1 establishes the accessibility of synthesis-null variations under hidden-response interventions. While Tables 6 and 7 provide supporting empirical evidence, the theorem itself does not guarantee that such variations are discriminative for naturally occurring images or improve generalization.

Questions for authors

Can the authors provide direct evidence that feature transformations or sampling in existing analytic CIL methods introduce harmful feature drift or noise? How does InnerCIL specifically mitigate these effects?

How does InnerCIL compare with simpler intermediate-feature baselines, such as attention-pooled patch tokens, pooled MLP outputs, or concatenated multi-layer features, under matched feature dimensions and classifier settings?

Why is preserving checkpoint write-back variation a suitable criterion for selecting class-discriminative neurons? Can the authors report downstream classification results for different registration strategies, rather than only the omitted variation shown in Figure 7?

Given the small improvements from Visual Fisher over Random-K, how essential is FRAC? How does it compare with other feature selection criteria under the same feature budget? Can the authors provide evidence that it actually removes noisy or unreliable features?

Theorem 1 considers hidden-response perturbations in the synthesis-null space. To what extent do these perturbations correspond to variations realized by natural images? Can the authors further clarify how the theoretical result relates to the observed classification improvements?

How sensitive is the method to the selection of intermediate blocks? The experiments use the last four ViT blocks. Would earlier or more widely distributed layers provide complementary information, and how does this choice affect performance?

3 comments
AuthorsOpenReview ↗

Response to W1, W2, W3, Q1, Q2, and Q3

Thank you for your constructive feedback.

W1 + Q1: Direct evidence of “harmful feature drift or noise”

To address your concern directly, we track 1,000 fixed task-1 test images across tasks 1–3 on CLIP/CIFAR-100 (B0 Inc10), comparing AnaCP's contrastive projection (CP), a sequential 512→128→512512\to128\to512 CLIP-Adapter trained for 20 epochs/task, and InnerCIL's raw encoding.

Cosine drift is mean 1−fˉ1(x)⊤fˉ3(x)1-\bar f_1(x)^\top\bar f_3(x) for L2-normalized features. Accuracy and squared error against one-hot targets use the fixed task-1 classifier.

RepresentationTask-3 cosine drift ↓Fixed-head accuracy, tasks 1→3 (%)Fixed-head squared error, tasks 1→3 ↓
Seq-Adapter0.310197.50→89.000.04820→0.19450
AnaCP CP0.163997.40→97.300.05113→0.08795
InnerCIL raw encoding0.000098.10→98.100.04102→0.04102

Seq-Adapter loses 8.50 percentage points (pp); AnaCP CP retains similar accuracy but raises squared score error by 72.0%.

For resampling, we freeze AnaCP's CP/ELM after task 3 and fit the same classifier with 100 real or Gaussian pseudo-features per class. We measure class-conditional sliced-Wasserstein distance over 64 directions and outliers beyond the real samples' 95th-percentile nearest-neighbor distance.

Fitting featuresSliced-W distance ↓Outlier rate (%) ↓Real-test squared error ↓30-class accuracy (%)
Real, 100/class0.003923±0.0000435.76±0.980.13667±0.0013291.93±0.06
Gaussian, 100/class0.004758±0.00004241.53±4.360.15701±0.0013691.91±0.11

Mean±standard deviation over three repetitions.

Gaussian resampling increases outliers and test score error. InnerCIL accumulates real-sample statistics with a fixed raw encoding; Fisher selection and RMS scaling are applied consistently to all historical statistics (Eq. 30).

InnerCIL avoids both update-induced raw-feature drift and distributional errors introduced by pseudo-feature generation.

W2 + Q2: “Attention-pooled patch tokens,” “pooled MLP outputs,” and “concatenated multi-layer features”

Following your suggestion, we compare local readouts using blocks 9–12, the same CLS-to-patch attention, four-layer CLS global branch, RMS scaling, and λ=0.003\lambda=0.003. Each has 3,072 local coordinates: 768 per block.

The patch baseline pools pre-MLP LN2 states; the MLP-output baseline pools projected MLP responses. InnerCIL pools hidden nonlinear activations before that projection, registering 768 neurons per block. Multi-depth CLS concatenates four normalized CLS vectors without a local branch.

Backbone/datasetMulti-depth CLSPooled patchesPooled MLP outputsNative neurons, 3072-D
CLIP/CIFAR81.4384.4784.0485.02
CLIP/CUB78.7782.6982.5983.41
DINOv2/CIFAR89.3990.8190.6991.34
DINOv2/CUB89.9789.8090.0190.45
Mean84.8986.9486.8387.56

Final accuracy (%), using ten incremental stages.

Native neurons outperform both pooled alternatives in all four settings, gaining 0.62–0.73 pp on average. They retain nonlinear neuron selectivity absent from pooled patches and responses removed by MLP output projection.

W3 + Q3: Why preserve “checkpoint write-back variation,” and what are the downstream results?

Checkpoint registration defines a fixed candidate pool from pretrained weights, without training or extra forward passes. Fisher then selects task-discriminative coordinates. The score combines activation sensitivity and write-back strength:

\qquad R_b=\sqrt d\|\gamma_b\|_\infty+\|\beta_b\|.$$ Here $w_{b,r}$ and $v_{b,r}$ are input and output weights; $\gamma_b,\beta_b$ are the LayerNorm parameters. Lemma 2 derives the omitted-write-back variation bound minimized by this score. Its retained synthesis-null capacity is documented in Table 6. We compare four registration strategies on CLIP with Visual Fisher, RMS normalization, and ridge fitting fixed. Random results use three seeds. |Dataset|$P/K$|Checkpoint|Input norm|Output norm|Random| |---|---|---:|---:|---:|---:| |CIFAR|4096/3072|83.07|83.14|**83.26**|83.16±0.10| |CIFAR|8192/6144|84.46|84.32|84.40|**84.47±0.09**| |CIFAR|11264/8704|**85.26**|85.25|85.21|85.25±0.10| |CUB|4096/3072|**80.41**|80.38|80.32|80.40±0.11| |CUB|8192/6144|82.59|82.53|82.24|**83.32±0.13**| |CUB|11264/8704|**83.93**|83.50|83.46|83.83±0.07| *Final accuracy (%); random entries are mean±standard deviation.* Capacity has the largest effect; no strategy wins at every budget. **At the default budget, Checkpoint leads on both datasets**, exceeding individual weight norms by 0.43–0.47 pp on CUB while providing a deterministic write-back approximation bound.
AuthorsOpenReview ↗

Response to W4, W5, Q4, Q5, and Q6

W4 + Q4: FRAC beyond a “small improvement”: alternatives and “noisy or unreliable features”

Tables 3 and 10 compare Fisher with Random-KK at the same dimension. We additionally compare against retaining all candidates to measure the efficiency benefit.

BackboneAll-PP accuracyFisher-KK accuracyAll-PP time (s)Fisher-KK time (s)
CLIP82.7982.8844.9025.10
DINOv290.8590.9259.0932.55

Accuracy: eight-dataset mean final accuracy (%). Time: selection and solve over ten CUB stages on an eight-thread CPU, excluding extraction. P=11264P=11264, K=8704K=8704.

Fisher reduces local width by 22.73% and selection/solve time by 44.1%–44.9%, while improving accuracy. We also compare alternative criteria at the same budget. Correlation-penalized Fisher subtracts 0.25 times the maximum squared correlation with the selected set from each normalized Fisher score.

CriterionCLIP-CIFARCLIP-CUBMean
Random-KK85.4082.7884.09
Variance-KK85.1983.3384.26
Correlation-penalized Fisher85.2983.4784.38
Visual Fisher85.2683.9384.60

Final accuracy (%).

Visual Fisher achieves the highest mean accuracy. Large global variance need not indicate class separation; correlation penalization adds selection cost and can discard complementary responses.

To test filtering directly, we select 8,704 neurons using training statistics and measure the between-class/within-class variance ratio on an independent test set for selected and discarded neurons.

CLIP datasetSelected ratioDiscarded ratioSelected/discarded
CIFAR0.42190.14452.92×
CUB0.39510.12263.22×

Discarded coordinates have weaker class separation relative to within-class fluctuations; selected coordinates have test discriminative ratios 2.92–3.22 times as high.

FRAC combines substantially cheaper solves, the best mean accuracy among the tested CLIP criteria, and direct evidence of filtering weakly discriminative features.

W5 + Q5: Do synthesis-null perturbations correspond to “variations realized by natural images”?

Theorem 1 identifies hidden-response directions removed by the MLP output projection. We decompose real-image pooled responses into visible and null components. Appendix A.1 (Eqs. 14–16) shows that the null component strictly reduces the ridge objective when it correlates with label residuals unexplained by existing features.

Backbone, eight datasets eachNull share of response energyNull share of objective reduction
CLIP54.12%–55.36%58.24%–71.15%
DINOv251.84%–52.80%55.63%–68.38%

CLIP extends the DINOv2 analysis in Table 6. Table 7 tests classification with selected coordinates, RMS scales, and classifier dimension fixed: restoring the natural-image null component improves final accuracy in all 16 settings.

Using a linear predictor BB, we separate internal responses ZZ into the component predicted from multi-depth CLS HH and the residual Z−HBZ-HB:

Classifier inputCLIP-CUBCLIP-CIFARDINOv2-CUBDINOv2-CIFAR
HH78.7781.4389.9789.39
[H,HB][H,HB]78.7781.4389.8989.44
[H,Z−HB][H,Z-HB]80.7682.1890.4789.61
[H,Z][H,Z]83.9385.2690.6191.59

Final accuracy (%).

The predicted component adds almost no benefit, while the residual improves all four settings and the full response performs best.

Natural images activate label-relevant internal information lost by projection, and recovering that information produces consistent test-accuracy gains.

Q6: Sensitivity to “earlier or more widely distributed layers”

We change only the blocks supplying internal neuron responses, keeping the last-four-block CLS branch, P=11264P=11264, K=8704K=8704, λ=0.003\lambda=0.003, and training views fixed.

Internal blocks, 1-indexedCLIP-CUBDINOv2-CUB
Middle: 5, 6, 7, 876.4289.75
Distributed: 3, 6, 9, 1281.5789.68
Final: 9, 10, 11, 1283.9390.61

Final accuracy (%).

CLIP is more sensitive: the final blocks exceed middle and distributed blocks by 7.51 and 2.36 pp. DINOv2 varies less, but also benefits from the final blocks.

Later patch states have undergone more cross-region interaction, combining focused semantic attention with mature local-part responses while exposing hidden activations before output projection.

The final four blocks provide the strongest discriminative readout among the tested placements at a fixed budget.

Dear Reviewer VAkX,

Thank you for your time and effort in reviewing our work. As the rebuttal period concludes within two days, we wish to respectfully confirm whether our responses have adequately addressed your initial concerns. Any feedback would be greatly appreciated.

Sincerely,

The Authors

Reviewer_WzJmOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes InnerCIL, a framework that extracts discriminative information from internal neurons of a frozen Vision Transformer (ViT) to enhance feature representations for analytic class-incremental learning (ACIL). The main contributions are as follows: 1.Attention-conditioned Native Dictionary Generation (ANDG): The method extracts neuron responses from intermediate MLP layers of a ViT and aggregates them using CLS-to-patch attention to construct a native feature dictionary. 2.Fisher-Ranked Analytic Consolidation (FRAC): The method uses the Fisher criterion to select discriminative features and combines accumulated sufficient statistics with ridge regression to perform class-incremental learning without storing previous training samples. 3.Theoretical Analysis: The paper presents a theoretical analysis of Synthesis-null Accessibility to explain why internal neurons may preserve information that is lost during the MLP output projection. Experimental Results: The authors evaluate InnerCIL using two pretrained backbones, CLIP ViT-B/16 and DINOv2 ViT-B/14, across multiple classification datasets. Compared with AnaCP*, InnerCIL improves the average accuracy (Aavg) from 86.49% to 88.03% and the final accuracy (Alast) from 80.40% to 82.88% with CLIP. With DINOv2, Aavg increases from 93.52% to 94.21%, while Alast improves from 90.20% to 90.92%. Overall, the paper demonstrates the effectiveness of internal neuron features for enhancing feature representations. However, the paper does not sufficiently explain how these features address the specific challenges of class-incremental learning or why they improve classification performance.

Critical strengths
  1. A relatively novel research perspective. Unlike existing analytic class-incremental learning methods that primarily rely on final output features or random feature mappings, the paper directly explores hidden MLP neuron responses within a pretrained ViT. This provides an alternative perspective on utilizing internal representations of frozen models.
  2. A relatively simple and practical method. The proposed framework does not require modifications to the main architecture of the pretrained ViT or backpropagation-based training of the backbone. It can extract additional features using existing pretrained weights and attention mechanisms.
  3. Experimental evaluation across multiple benchmarks. The authors evaluate the method using two different pretrained backbones and multiple class-incremental learning datasets. They also provide several comparisons and ablation studies involving the native dictionary, attention-based aggregation, feature selection, and null-space information, offering empirical support for the effectiveness of the method.
  4. An attempt to provide a theoretical explanation. The synthesis-null accessibility analysis connects information loss during the MLP output projection with the accessibility of hidden-layer information. Although the relationship between this analysis and actual classification improvements needs further investigation, it provides a potentially useful theoretical perspective.
Critical weaknesses
  1. Insufficient connection to the core challenges of class-incremental learning. The method mainly improves classification performance through enhanced feature representations. However, the paper does not provide sufficient evidence that the proposed method reduces catastrophic forgetting, mitigates interference between old and new classes, or improves decision boundary stability.
  2. Insufficient explanation of the discriminative mechanism of internal neurons. The paper does not clearly identify the discriminative information contained in internal neuron representations or explain their specific contributions to classification decisions. It also lacks a convincing explanation of why these representations can complement the final CLS features.
  3. A gap between theoretical analysis and practical performance gains. Although the synthesis-null accessibility analysis suggests that hidden layers may preserve information unavailable in the MLP output projection, it does not directly establish the discriminative value of this information. Further evidence is needed to explain how it improves incremental learning performance.
  4. Insufficient analysis of the conditions under which the method is effective and its limitations. The paper does not systematically explain the differences in performance gains across pretrained backbones, datasets, and feature dimensions. It also does not clearly identify the conditions under which internal neuron representations are effective or ineffective.
  5. Limited novelty and independent contributions of some technical components. Fisher-based feature selection is built on a classical discriminant criterion and provides only a small additional improvement over random feature selection. This limits its significance as a major technical contribution.
Questions for authors

no

6 comments
AuthorsOpenReview ↗

Response to W1 and W2

Thank you for your constructive feedback and for recognizing the value of our work. We address your concerns below.


W1: Catastrophic Forgetting

(1) Measuring Forgetting. We evaluate the average forgetting rate across eight DINOv2 ViT-B/14 benchmarks:

MethodForgetting (%) ↓
RanPAC4.67
Multi-depth CLS4.55
Complete AnaCP*4.25
InnerCIL3.73

InnerCIL achieves the lowest forgetting rate, reducing forgetting by 0.94, 0.82, and 0.52 percentage points relative to RanPAC, multi-depth CLS, and complete AnaCP*, respectively.

(2) Why InnerCIL Reduces Forgetting. (i) Existing methods mainly use global CLS features or MLP-transformed CLS. Under pretraining–downstream distribution shifts, these features may lack details that distinguish similar classes, increasing ambiguity and interference as new classes arrive. Updating the MLP can also cause feature drift. (ii) InnerCIL reads richer internal responses from the frozen backbone, preserving local discriminative cues and keeping the raw encoding fixed. Stage-dependent selection and scaling are applied consistently to accumulated statistics. This improves class separation while consistently representing past data, supporting the lower forgetting observed above.


W2: Discriminative Mechanism of Internal Neurons and Why They Complement Multi-depth CLS

(1) Theoretical Source of Discriminative Information

Let hh denote the multi-depth CLS representation, zz the attention-pooled internal neuron responses, and YY the class label. By the chain rule of mutual information:

\tag{5}$$ If $I(Y;z\mid h)>0$, $[h,z]$ contains strictly more class information than $h$. For least squares, let $H,Z$ be the corresponding training feature matrices and $Y$ the one-hot label matrix. The orthogonal residual $R = (I - HH^\dagger)Z$ **cannot be linearly predicted from $H$**. Its exact reduction in fitting error is: $$\min_W\|HW-Y\|^2-\min_{W,V} |HW+ZV-Y|^2=|RR^\dagger Y|^2 \ge 0. \tag{6}$$ For ridge regression (**Appendix A.1, Eqs. 14–16**), define the residual operator $P_\lambda = I - H(H^\top H + \lambda I)^{-1}H^\top$, $F = Z^\top P_\lambda Y$, and $S = Z^\top P_\lambda Z + \lambda I \succ 0$. Eliminating global weights via the Schur complement gives the exact reduction in the minimum regularized objective: $$\Delta\mathcal{L}=\operatorname{tr}\left(F^\top S^{-1}F\right)\ge 0, \qquad \Delta\mathcal{L}>0\iff F = Z^\top P_\lambda Y \ne 0. \tag{7}$$ Equations (6)–(7) give exact conditions for reducing the training objective. In particular, **the ridge objective strictly decreases when $Z^\top P_\lambda Y\ne0$**, meaning that the internal responses align with class-target residuals left by the global branch. The following ablation evaluates the corresponding benefit for classification on unseen images. **(2) Residual Ablation** We fitted a ridge predictor $B = (H^\top H + \lambda I)^{-1}H^\top Z$ **using training data only**, splitting responses into the **CLS-predictable component $HB$** and **unexplained residual $Z-HB$**. With training splits, budgets ($P=11264, K=8704$), global-branch scaling, and ridge penalty ($\lambda=0.003$) fixed, we compare four readouts by final accuracy $A_{\mathrm{last}}$ (%): |Classifier Input Representation|CLIP-CUB (%)|CLIP-CIFAR (%)|DINOv2-CUB (%)|DINOv2-CIFAR (%)|4-Benchmark Mean (%)| |---|---:|---:|---:|---:|---:| |① Multi-depth CLS only: $H$|78.77|81.43|89.97|89.39|84.89| |② CLS + Linearly Predicted Response: $[H\, HB]$|78.77|81.43|89.89|89.44|84.88| |③ CLS + Internal Residual: $[H\, Z-HB]$|80.76|82.18|90.47|89.61|85.76| |④ **CLS + True Internal Response: $[H\, Z]$ (InnerCIL)**|**83.93**|**85.26**|**90.61**|**91.59**|**87.85**| **Analysis:** - **(i) Increasing feature width with CLS-predictable responses gives virtually no gain**: Comparing ① and ②, appending the 8,704-dimensional linearly predicted response $HB$ leaves accuracy virtually unchanged (84.89% vs. 84.88%). - **(ii) The unexplained internal residual $Z - HB$ carries discriminative cues**: Comparing ① and ③, appending only the residual $Z - HB$ consistently improves test accuracy across all four benchmarks (**+1.99 pp** on CLIP-CUB, **+0.75 pp** on CLIP-CIFAR, **+0.50 pp** on DINOv2-CUB, and **+0.22 pp** on DINOv2-CIFAR), showing that responses unexplained by CLS carry useful information for test classification. - **(iii) Joint optimization over $[H, Z]$ achieves the best performance**: Comparing ① and ④, complete InnerCIL ($[H,Z]$) improves accuracy by **+5.16 pp, +3.83 pp, +0.64 pp, and +2.20 pp** (mean **+2.96 pp**). The full and residual representations span the same combined linear space, but their parameterizations induce different ridge penalties. Joint fitting on the original native coordinates gives the strongest test performance here.
AuthorsOpenReview ↗

Response to W3 and W5

W3: Discriminative Value of Hidden Responses Beyond MLP Outputs

Following your suggestion, we directly compare pre-projection hidden responses with pooled MLP outputs. We match the local dimension at 3,072 coordinates (768 per block), so the comparison isolates the readout location at equal feature width.

Backbone / DatasetPooled MLP Outputs (%)Hidden Neuron Responses (%)
CLIP / CIFAR-10084.0485.02
CLIP / CUB-20082.5983.41
DINOv2 / CIFAR-10090.6991.34
DINOv2 / CUB-20090.0190.45

Hidden responses outperform MLP outputs in all four settings. Reading the activations before projection preserves discriminative information that is lost when the MLP compresses them back into the residual stream.

We further connect this comparison to Theorem 1 through the synthesis-null analysis in Appendix A.1, Equations 14–16 and Tables 6–7:

  • (1) Benefit for incremental classification. Each block's output projection Vb∈Rd×4dV_b\in\mathbb R^{d\times4d} maps hidden activations to the dd-dimensional residual stream, removing ker⁡(Vb)\ker(V_b) components (dimension at least 3d3d). Neuron readout retains these directions. When aligned with class-target residuals, they supply information not directly retained by the projected features and improve the analytic objective under Equation (7).
  • (2) Synthesis-null information on natural images. Table 6 decomposes pre-registration responses as zbvis=Vb†Vbzbattz_b^{\mathrm{vis}}=V_b^\dagger V_bz_b^{\mathrm{att}} and zbnull=(I−Vb†Vb)zbattz_b^{\mathrm{null}}=(I-V_b^\dagger V_b)z_b^{\mathrm{att}}. Across eight DINOv2 datasets:
QuantityRange Across Eight DINOv2 Datasets
Synthesis-null share of response energy51.84%–52.80%
Synthesis-null share of regularized objective reduction55.63%–68.38%

Natural images activate these null directions, which contribute to fitting class targets.

  • (3) Controlled restoration on all 16 benchmarks. Table 7 fixes selected neuron indices JtJ_t, RMS scales, classifier dimension, λ=0.003\lambda=0.003, and the global branch, comparing visible-only (Vis), null-only (Null), and complete (All) local responses:
Pretrained Backbone (8 Benchmarks Each)CLS + Visible Response Vis (AlastA_{\mathrm{last}}, %)CLS + Complete Response All (AlastA_{\mathrm{last}}, %)Null-Only Gain over Global (ΔAavg/ΔAlast\Delta A_{\mathrm{avg}} / \Delta A_{\mathrm{last}}, pp)Gain from Restoring Null (All−Vis\mathrm{All}-\mathrm{Vis}, ΔAavg/ΔAlast\Delta A_{\mathrm{avg}} / \Delta A_{\mathrm{last}}, pp)Benchmarks Improved
CLIP ViT-B/1681.4982.88+3.37 / +5.39+0.81 / +1.398 / 8 (100%)
DINOv2 ViT-B/1490.1590.92+1.01 / +1.80+0.42 / +0.778 / 8 (100%)

Analysis. Null-only responses improve mean final accuracy over the global branch by 5.39 percentage points on CLIP and 1.80 on DINOv2. Restoring the null component to the visible responses improves test accuracy in all 16 settings. Together with the matched-width MLP-output comparison, these results connect the information identified by Theorem 1 to practical gains in incremental classification.


W5: Independent Contribution and Technical Significance of Fisher Screening (FRAC)

FRAC contributes both to consistent feature selection across incremental stages and to computational efficiency:

(1) Methodological contribution. The preferred feature subset changes as new classes arrive. FRAC enables stage-adaptive Fisher screening while preserving the historical statistics needed for analytic classification. It retains statistics over a fixed raw candidate dictionary, then applies the current selection and scaling to reconstruct the deployed system. This recovers the joint ridge solution for the same training views and current representation, even when the selected neurons change. The contribution is the integration of efficient discriminative screening with statistically consistent, replay-free updates.

(2) Accuracy and Efficiency. We compare Fisher screening with unscreened (All-P) and random (Random-K) selection:

SelectionFeature Dim.CLIP ACC (%) ↑DINOv2 ACC (%) ↑CLIP Time (s) ↓DINOv2 Time (s) ↓
All-P11,26482.7990.8544.9059.09
Random-K8,70482.3490.6924.2331.36
Fisher-K (Ours)8,70482.8890.9225.1032.55

Accuracy is the eight-dataset mean final accuracy; time covers selection and solving.

Compared with Random-K, Fisher screening improves mean final accuracy by 0.54 and 0.23 percentage points on CLIP and DINOv2, respectively, with approximately one second of additional scoring time. Compared with All-P, it reduces feature dimensionality by 22.73% and computational time by 44.1% and 44.9%, respectively, while improving accuracy.

These results demonstrate that FRAC effectively balances discriminative performance and computational efficiency, making it a meaningful component of our continual learning framework.

AuthorsOpenReview ↗

Response to W4

W4: Explaining Performance Variations Across Backbones, Datasets, and Feature Budgets

(1) Differences Across Pretrained Backbones: CLIP vs. DINOv2

Pretrained BackboneAnaCP* (Aavg/AlastA_{\mathrm{avg}} / A_{\mathrm{last}}, %)InnerCIL (Aavg/AlastA_{\mathrm{avg}} / A_{\mathrm{last}}, %)Absolute Gain (ΔAavg/ΔAlast\Delta A_{\mathrm{avg}} / \Delta A_{\mathrm{last}}, pp)
CLIP ViT-B/1686.49 / 80.4088.03 / 82.88+1.54 / +2.48
DINOv2 ViT-B/1493.52 / 90.2094.21 / 90.92+0.69 / +0.72

Analysis. (i) We attribute CLIP's larger final-accuracy gain of 2.48 percentage points to the additional visual detail available beyond its global image–text embedding. CLIP emphasizes image-level alignment and projects its 768-dimensional visual representation to 512 dimensions; local part geometry and texture can remain useful inside the MLP even when they are not retained in the final embedding. (ii) DINOv2 starts from a stronger baseline of 90.20% final accuracy. Its pretraining combines image-level DINO and patch-level iBOT distillation, and its downstream CLS representation retains the full 768 dimensions. This provides a stronger starting point for visual discrimination and helps explain both its higher absolute accuracy and the smaller additional gain. InnerCIL nevertheless recovers complementary internal information, reducing the remaining 9.8% error rate by 0.72 percentage points.

(2) Differences Across Datasets: When Are Internal Neurons Most Effective?

Representative DatasetVisual Task CharacteristicsCLIP AlastA_{\mathrm{last}} (%)Gain over AnaCP* (pp)DINOv2 AlastA_{\mathrm{last}} (%)Gain over AnaCP* (pp; vs. Replay-free AnaCP)
FGVC-AircraftFine-grained structural/part variants70.63+6.9985.48+1.71 (+2.28)
Stanford CarsFine-grained model/grille/headlight cues90.52+2.8391.88+1.23 (+2.34)
ObjectNet-200Severe viewpoint/background/pose shifts66.05+3.18——
UCF-101Localized human–object action regions89.15+1.9890.45+0.66 (+1.94)
Food-101Localized ingredient and texture patterns92.70+0.4893.40+1.19 (+1.14)
CUB-200Fine-grained birds (global + local cues)83.93+0.3190.61+0.51 (+0.02)
CIFAR-100Low-resolution (32×3232\times32 upsampled) objects85.26+3.7091.59−0.62 (+0.56)

Conditions of Effectiveness and Limitations.

  • (i) Where InnerCIL excels: Gains are largest on tasks requiring fine-grained part discrimination or robustness to viewpoint/background clutter (e.g., Aircraft +6.99 pp / +1.71 pp, Cars +2.83 pp / +1.23 pp, ObjectNet +3.18 pp), where global CLS tokens blur localized evidence.
  • (ii) Where gains are smaller: When source images have very low native spatial resolution (e.g., CIFAR-100 at 32×3232\times32) and the backbone's global representation is already strong (DINOv2-CIFAR >91%), marginal gains from patch-level neurons can diminish—though under the matched strict replay-free protocol (vs. AnaCP without synthetic feature generation), InnerCIL still leads by +0.56 pp (91.59% vs. 91.03%). Internal neuron readout is most valuable when the frozen backbone computes task-relevant local details that are not fully retained in its final global embedding.

(3) Impact of Feature Dimension

DINOv2-CUB Dimension (P/KP / K)Final Accuracy AlastA_{\mathrm{last}} (%) ↑\uparrowPersistent State (MiB) ↓\downarrowSelection & Solve Time (s) ↓\downarrow
4096/30724096 / 307290.30119.227.97
8192/61448192 / 614490.48277.3720.58
11264/870411264 / 8704 (Default)90.61438.4132.55

(Note: Selection and solve times are measured on an 8-thread CPU solver to isolate O(DK3)O(D_K^3) matrix-factorization scaling.)

Analysis. Expanding (P/K)(P/K) steadily improves accuracy from 90.30% to 90.48% and 90.61% (with Appendix A.4 Table 8 further documenting a smooth plateau across 33 (P/K)(P/K) pairs), while solve cost scales with DKD_K, motivating FRAC's dimensionality reduction.

Comments on the paper

1 comment
Area_Chair_M77bOpenReview ↗

Comment by AC

Dear Reviewers

As we are now halfway through the discussion period, I encourage all reviewers to engage in the responses and discuss should there be more questions.

---AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.