- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the gap between representation classifiability and actual continual classification accuracy in class incremental learning with pretrained models. Using trajectory linear probes and closed-form ridge readouts, the authors argue that the main issue is the inconsistency between historical classification statistics and the evolving feature geometry. To alleviate this issue, SASKIA combines a global scale-and-shift module frozen after first-task training, an instance-level hypernetwork driven by token means, task-level EMA, and an analytical ridge classifier in a random feature space.
Critical strengths
- Reasonable motivation. Separately examining final performance and representation readability can expose classifier issues that cannot be identified by looking only at end-task accuracy. It has practical engineering significance when historical original images cannot be replayed.
- The experiments are comprehensive. Four datasets, two ViT checkpoints, offline-to-online / always-offline / always-online, and the B50 setting, covering fine-grained and domain shift scenarios.
- Extensive ablation experiments. Table A9 begins decomposing components starting from classifier replacement on frozen representations. Table A8 contains both EMA endpoints and the projection dimensions. These results provide a basis for understanding the compositional method.
- Provided an anonymous code implementation. The anonymous code provided the core SASKIA training and ridge code, the authors also acknowledged that the number of parameters is not equivalent to memory and analyzed the limitations.
Critical weaknesses
- The summarize of existing methods is overly broad, and the positioning of core components and directly related research is insufficient. Sec. 2/3.2 generally regards task-specific adaptation as unable to maintain readout compatibility, but EASE's old-class prototype completion was originally aimed at the incompatibility of the expanded subspace. Finding 4 also changes both NCM/ridge and RP, so it cannot be attributed solely to shared/specialized adaptation. Global SSF and instance-conditioned affine modules have direct precedents in SSF (NeurIPS 2022) and FiLM (AAAI 2018), respectively, yet these are not explicitly matched in the paper's references. LDC (ECCV 2024) has already discussed preserving discriminative information, but old prototypes drift, and there are also 2026 studies on bidirectional alignment and analytical drift compensation. This paper should precisely delimit its novelty and distinguish it from these directions, rather than packaging the traditional drift/classifier mismatch problem as a new mechanism that sufficiently rules out existing explanations.
- RanPAC's numerical anomalies were not ruled out under a unified implementation. A.3.3 states that RanPAC drops by 50–70 points at certain stages and then recovers, inferring ill-conditioned solving, and attributes SASKIA's greater stability to fixed RP and EMA. However, fixed RP is also an existing design of RanPAC. The current SASKIA code uses a float64 CPU solve, while the paper claims low-precision compression and Cholesky, and does not provide identical numerical configurations for the baselines. Recovery after failure is also not the only possible signal of numerical error.
- Some ablation conclusions, parameter statistics, and figures directly contradict each other. Fig. 5b/Sec. 5 claims that mean-pooled is best at every width, but at concatenated 58.10 is higher than pooled 57.40. A.5 states that replacing the RanPAC adapter with SSF “improves both the probe and the gap,” yet Table A9 shows that the probe on CUB is 90.76→90.45 and on IN-A is 60.57→60.34. Table A6 obtains about 0.07M SSF for four equal-width sites, but the released source code includes modulation of QKV, MLP intermediate dimensions, etc. The registered SSF is 205,824 parameters, which together with H is about 0.798M, and cannot support the precise claims of 0.66M and a 17-fold reduction.
- The memory and analytical solve implementation reported in the appendix is materially inconsistent with the anonymized code. A.4.1–A.4.2 and Table A6 claim that is a 100 MB float16 upper triangular matrix, that only the seed is stored for random projection, that Cholesky is used, and that one eigendecomposition is reused during the regularization search. Lines 173–220 of the code actually allocate a full CPU float64 Gram/Correlation, call a generic solve for each candidate, and line 45 of
vit_saskia.pyalso keeps the full projection matrix resident. When M=10000, the Gram term alone is 800 MB, and_accumulate_rqtemporarily stores double-precision features for all current tasks. This contradiction affects the storage conclusions of Tables A6–A7, the end-to-end cost in Fig. 5a, and the stability explanation in A.3.3. - The readout gap currently cannot be interpreted as independently quantifying the loss caused by stale feature statistics. Sec. 3.2 Finding 2/3 attributes the adapted encoder's ridge gap, after subtracting the frozen-encoder residual, to moving geometry, but ridge and the linear probe differ in loss, regularization, and hypothesis class, and the residual can vary with representation. A.5 itself also acknowledges, in the frozen shadow case, the representation dependence of a nonzero gap. More seriously, Table A9 rows 1–3 keep the same probe, yet row 3 adds a ReLU RP. The deployed classifier is already nonlinear with respect to the representation in which the probe resides, so the claim in A.3.4 that the probe “upper-bounds” its capability does not hold. One should match ridge, regularization, and projection across methods on the same final deployed features, and compare historically accumulated statistics with re-extracted full-data statistics. Otherwise Eq. (2) is merely a descriptive identity.
- Numerous defensive statements. Although the authors honestly disclosed the AI assistance in the writing process, the large number of defensive statements in it should be revised. Almost every part that indicates the role/contribution of the proposed method is accompanied by a constraining/qualifying statement. These defensive statements run from the abstract to the experimental section. For example, the abstract mentions "Representations are therefore not the bottleneck, the decision layer is," and then immediately follows with "This does not imply that adaptation is unnecessary, xxxx.". The abstract only needs to concisely state the findings and contributions, it should not make a simple point so complicated. The paper uses more than 20 instances of "rather than," most of which serve only to guard against criticism from some aspect. For example, the Intorduction section mentions "rather than guaranteeing strict feature invariance". Such a statement contributes nothing to understanding the paper.
Questions for authors
- The authors should further clarify the boundary between the proposed method and existing work, i.e., what is the direct contribution of the proposed method?
- Did RanPAC directly use the official open-source code, or did the authors reproduce it themselves? The authors should check whether the issue arises from code implementation.
- Some figures/tables conflict with the paper’s statements. Is this a writing issue or a data issue? The authors should check the data in the figures/tables and the statements in the relevant sections.
- The anonymous code implementation does not match the paper’s statements. Is this a code implementation issue, or did the additional experiments use another version of the code?
6 comments
Official Comment by Authors (Part 2)
W4 & Q4: Description of memory and solver description in appendix.
The anonymized repository was an outdated version, which we inadvertently overlooked. We apologize for this mismatch and have corrected the repository link. The paper's results and all reported experiments were produced using the updated codebase. Therefore, the memory requirements and analytical solver implementation described in the paper are accurate. However, we also understand that the ridge solver implementation may not have been sufficiently clear, and have expanded its explanation in the revised paper.
W5: The readout gap and the loss from stale statistics
Thank you. Eq. 2 is a diagnostic by design: each method is evaluated as it was proposed, and the gap measures how much of what its features linearly support its own classifier does not deliver. Which part of that gap is due to stale statistics is a separate, causal question, and the control you describe is the right test for it. We ran it, stored statistics against statistics re-extracted from all data, and it supports the claim directly. After every task, we refit the ridge classifier from scratch, on all data seen so far, using the current model. Both this refitted classifier and the probe need all seen data at once, so they are not available in continual learning; we compute them only as measurements, and SASKIA never uses them. Everything else is the same as for the continual classifier: the projection, the training examples, the ridge objective and how the regularization is chosen. The only difference is that the continual classifier's statistics were collected earlier, with older versions of the model. So the accuracy difference between the refitted and the continual classifier is exactly the cost of stale statistics. The gap then splits into two parts:
probe − continual = (probe − refitted)-> classifier type + (refitted − continual) ->stale statistics
| Benchmark and setting | Continual | Refitted | Lost to stale statistics | Probe | Probe − refitted |
|---|---|---|---|---|---|
| CIFAR, no merging | 82.32 | 90.54 | 8.22 | 90.89 | 0.35 |
| CIFAR, SASKIA | 90.2 | 90.31 | 0.11 | 92.33 | 2.02 |
| CUB, no merging | 89.10 | 90.84 | 1.74 | 91.05 | 0.21 |
| CUB, SASKIA | 89.8 | 90.11 | 0.31 | 91.39 | 1.28 |
| IN-R, no merging | 76.17 | 80.42 | 4.25 | 80.77 | 0.35 |
| IN-R, SASKIA | 79.0 | 80.66 | 1.66 | 81.24 | 0.58 |
| IN-A, no merging | 56.10 | 60.97 | 4.87 | 61.32 | 0.35 |
| IN-A, SASKIA | 58.8 | 60.09 | 1.29 | 62.36 | 2.27 |
-
Without merging, old statistics cost 8.2 / 1.7 / 4.3 / 4.9 points (CIFAR / CUB / IN-R / IN-A), which is 89–96% of the whole gap. With SASKIA's merging, this drops to 0.1 / 0.3 / 1.7 / 1.3.
-
Without merging, the features are only a little worse (probe 0.3–1.4 points below SASKIA's), but the classifier the model actually uses is 0.7–7.9 points worse. So the damage is in the stored statistics, not in the features.
-
SASKIA's readout gap in Table 3 separates into classifier-type and stale-statistics costs: CIFAR (2.0 + 0.1), CUB (1.3 + 0.3), IN-R (0.6 + 1.7), and IN-A (2.3 + 1.3) points. The classifier-type term measures the cost of using an incrementally accumulable, data-free ridge classifier instead of a probe refitted on all seen data; it is distinct from staleness, which arises from accumulating statistics under evolving representations.
So when the model keeps changing, nearly all of the readout gap is stale statistics, and SASKIA's merging is what removes it.
"Upper bound." Rows 1–3 of Table A9 use the same frozen model, so the probe is the same by construction, while the classifier adds a nonlinear projection. The probe is therefore not an upper bound, and we drop that wording in A.3.4.
Across methods. Table 3 compares methods that each keep their own classifier, so the gaps there describe each method; on their own they do not show that one method's statistics are better aligned. Methods such as EASE grow their feature space with every task, so a classifier accumulated over a fixed space is not defined for them in the same way. This is why each method is evaluated with the classifier it was designed for.
W6: Too many defensive statements
Our aim with these qualifiers was to avoid giving a false impression of what the results show, but we agree that, taken together, they make the text sound defensive and the main points harder to follow. We now revised the text in many places to improve readability.
We believe our replies and the revised paper address all the concerns you raised. If so, we would be grateful if you would consider updating your score.
Dear Authors,
Thank you for your constructive responses and additional experiments. The comparison between accumulated and refitted statistics substantially addresses my concern about distinguishing stale-statistics effects from differences between classifiers. I also appreciate the corrections to the numerical statements and parameter counts, and the clarification that the probe is not an upper bound.
However, some concerns remain. The new experiment supports the explanation within SASKIA and its no-merging variant, but does not fully justify the broader conclusions about shared versus task-specific adaptation. The clarification regarding RanPAC identifies the implementation and limits the failures to B50, but does not establish their cause or validate the proposed explanation for the stability difference.
Overall, the rebuttal resolves several concerns and strengthens the paper, but the remaining issues leave me unconvinced that it warrants acceptance. I therefore maintain my rating of 2.
Best regards,
Reviewer cekJ
Official Comment by Authors (Part 1)
Summary. Thank you for the review, checking the code, and recognizing the motivation, experimental breadth, and ablations. We conducted the requested comparison between accumulated statistics and statistics refitted on all seen data. Without merging, stale statistics account for 89–96% of the readout gap, while SASKIA's merging removes 61–99% of this loss (W5). Regarding implementation discrepancies (W4), the initially shared anonymized repository was outdated. We apologize and have corrected the link. We also corrected the parameter counts while confirming that the method remains under 1% (W3), and strengthened the positioning against SSF, FiLM, EASE, and LDC (W1). We address each point below.
W1 & Q1: Positioning against existing work and our direct contribution
Contribution. Stored statistics becoming outdated as features drift is well established. SDC (CVPR 2020), ADC (CVPR 2024), and LDC (ECCV 2024) estimate feature drift and correct stored prototypes. EASE addresses related mismatch through prototype completion when new subspaces are added, while MOS and TUNA correct classifiers using generated or stored class statistics. Section 2 now acknowledges these contributions. Our contribution is instead a measurement separating feature quality from accumulated-classifier validity throughout continual post-training. Together with the refitted-classifier control (W5), it isolates performance loss due to stale statistics. Without merging, stale statistics account for 89–96% of the readout gap, while feature quality remains largely preserved.
Relation to drift compensation. Existing methods allow features to evolve and subsequently repair accumulated statistics. SASKIA takes a complementary approach by limiting feature drift at its source: adaptive components are shared and gradually merged, preserving the validity of previously accumulated statistics without a separate classifier repair step. The approaches are complementary: drift compensation could also be applied on top of SASKIA, while our measurement quantifies residual drift effects.
SSF and FiLM. These are established building blocks. Section 4 now explicitly cites SSF (Lian et al., NeurIPS 2022) for the global module and FiLM-style conditioning (Perez et al., AAAI 2018) for the instance-level module.
Our contributions (Q1). We summarize our contributions in three parts: (1) a measurement framework decomposing the gap between feature-supported and continual-classifier performance into classifier-type limitations and stale statistics (W5); (2) the empirical finding that features remain largely intact during continual post-training, while accumulated classifier statistics become outdated and account for most resulting performance loss; and (3) the design principle of sharing adaptive components and gradually merging them to preserve classifier validity, instantiated by SASKIA. We do not claim SSF, FiLM, random projection, or ridge regression as novel.
W2 & Q2: RanPAC's failures in B50 and which implementation we used
RanPAC's collapses occur only in the large-first-task setting (B50, Figure A3). In all main experiments, RanPAC runs stably: its accuracy curves in Figures 4, A1, and A2 decline smoothly across all four benchmarks, with no task-level collapse. Thus, comparisons in Tables 3, A2, and A3 involve no solver failure. We use the PILOT implementation (Sun et al., 2023) with its released RanPAC configuration (adapters tuned in the first session, M = 10,000), as stated in A.1. RanPAC uses more memory to ensure numerical stability: it stores G as a full 32-bit matrix and uses direct solve, requiring no safeguards. Our safeguard is only needed when storing statistics in 16-bit to halve memory, a stricter setting RanPAC does not use. We likewise use no safeguards with fp32.
W3 & Q3: Text contradicting figures and tables
We corrected all three discrepancies. Two were writing errors and one a counting error; none changes the main conclusions.
Figure 5b. Mean-pooling performs best at widths 128, 256, and 512; concatenation is best at 768 (58.10 vs. 57.40). The best overall setting is mean-pooling at width 256 (58.77). The text now reflects this.
Table A9. Rows 4 → 5: Shared SSF slightly lowers probe accuracy but narrows the gap and improves final accuracy. We now clarify this interpretation. A weaker probe alongside a better continual classifier supports our analysis: gains come from a classifier that remains valid, not improved features.
Parameter counts. Following the original SSF, the implementation modulates the qkv projection, MLP hidden layer, patch embedding, and final norm. The correct counts are 205,824 for SSF and 592,128 for the hypernetwork, totaling 797,952. This is 0.93% of ViT-B/16's 85.8M parameters, so "under 1%" remains correct. The ratio to RanPAC's adapters is approximately 5.8×. We corrected Table A6, the Params column in Table A9, and the text.