acceptodds
Under review as a conference paper at ICLR 2027

Trajectory-Probing Continual Post-Training: Identifying and Closing the Readout Gap

Abstract

Continual learning with foundation models is dominated by methods that adapt pre-trained representations per task via prompts, adapters, or low-rank updates, under the assumption that feature adaptation is the primary driver of performance. We revisit this assumption and find that the picture is inverted: as tasks accumulate, a linear joint probe fit on the continually adapted backbone climbs toward joint-training while the incrementally trained classifier accuracy falls. We call this discrepancy the readout gap - the share of linearly decodable information that the incremental classifier fails to recover. The gap survives a closed-form readout that is invariant to task order, so it is not reducible to classifier bias. Representations are therefore not the bottleneck; the decision layer is. This does not imply that adaptation is unnecessary, but that task-specific adaptation is incompatible with an accumulated readout: it perturbs the feature geometry the classifier depends on, whereas shared, task-agnostic calibration improves features without destabilizing it. Guided by this, we introduce Scale-and-Shift Knowledge Injection (SASKIA), which calibrates representations at two granularities that are both shared across tasks: a global scale-and-shift module aligning the backbone to the target domain, and an instance-level hypernetwork producing per-sample corrections, refined online by EMA-based model merging rather than task-specific growth. New classes are absorbed by closed-form ridge regression in this progressively refined space, requiring no rehearsal or explicit task boundaries. SASKIA attains the smallest readout gap among all compared methods and matches or outperforms them across benchmarks, using under 1% of the backbone's parameters.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, lifelong learning, incremental learning, evaluation metrics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cekJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the gap between representation classifiability and actual continual classification accuracy in class incremental learning with pretrained models. Using trajectory linear probes and closed-form ridge readouts, the authors argue that the main issue is the inconsistency between historical classification statistics and the evolving feature geometry. To alleviate this issue, SASKIA combines a global scale-and-shift module frozen after first-task training, an instance-level hypernetwork driven by token means, task-level EMA, and an analytical ridge classifier in a random feature space.

Critical strengths
  1. Reasonable motivation. Separately examining final performance and representation readability can expose classifier issues that cannot be identified by looking only at end-task accuracy. It has practical engineering significance when historical original images cannot be replayed.
  2. The experiments are comprehensive. Four datasets, two ViT checkpoints, offline-to-online / always-offline / always-online, and the B50 setting, covering fine-grained and domain shift scenarios.
  3. Extensive ablation experiments. Table A9 begins decomposing components starting from classifier replacement on frozen representations. Table A8 contains both EMA endpoints and the projection dimensions. These results provide a basis for understanding the compositional method.
  4. Provided an anonymous code implementation. The anonymous code provided the core SASKIA training and ridge code, the authors also acknowledged that the number of parameters is not equivalent to memory and analyzed the limitations.
Critical weaknesses
  1. The summarize of existing methods is overly broad, and the positioning of core components and directly related research is insufficient. Sec. 2/3.2 generally regards task-specific adaptation as unable to maintain readout compatibility, but EASE's old-class prototype completion was originally aimed at the incompatibility of the expanded subspace. Finding 4 also changes both NCM/ridge and RP, so it cannot be attributed solely to shared/specialized adaptation. Global SSF and instance-conditioned affine modules have direct precedents in SSF (NeurIPS 2022) and FiLM (AAAI 2018), respectively, yet these are not explicitly matched in the paper's references. LDC (ECCV 2024) has already discussed preserving discriminative information, but old prototypes drift, and there are also 2026 studies on bidirectional alignment and analytical drift compensation. This paper should precisely delimit its novelty and distinguish it from these directions, rather than packaging the traditional drift/classifier mismatch problem as a new mechanism that sufficiently rules out existing explanations.
  2. RanPAC's numerical anomalies were not ruled out under a unified implementation. A.3.3 states that RanPAC drops by 50–70 points at certain stages and then recovers, inferring ill-conditioned solving, and attributes SASKIA's greater stability to fixed RP and EMA. However, fixed RP is also an existing design of RanPAC. The current SASKIA code uses a float64 CPU solve, while the paper claims low-precision compression and Cholesky, and does not provide identical numerical configurations for the baselines. Recovery after failure is also not the only possible signal of numerical error.
  3. Some ablation conclusions, parameter statistics, and figures directly contradict each other. Fig. 5b/Sec. 5 claims that mean-pooled is best at every width, but at r=768r=768 concatenated 58.10 is higher than pooled 57.40. A.5 states that replacing the RanPAC adapter with SSF “improves both the probe and the gap,” yet Table A9 shows that the probe on CUB is 90.76→90.45 and on IN-A is 60.57→60.34. Table A6 obtains about 0.07M SSF for four equal-width sites, but the released source code includes modulation of QKV, MLP intermediate dimensions, etc. The registered SSF is 205,824 parameters, which together with H is about 0.798M, and cannot support the precise claims of 0.66M and a 17-fold reduction.
  4. The memory and analytical solve implementation reported in the appendix is materially inconsistent with the anonymized code. A.4.1–A.4.2 and Table A6 claim that GG is a 100 MB float16 upper triangular matrix, that only the seed is stored for random projection, that Cholesky is used, and that one eigendecomposition is reused during the regularization search. Lines 173–220 of the code actually allocate a full CPU float64 Gram/Correlation, call a generic solve for each candidate, and line 45 of vit_saskia.py also keeps the full projection matrix resident. When M=10000, the Gram term alone is 800 MB, and _accumulate_rq temporarily stores double-precision features for all current tasks. This contradiction affects the storage conclusions of Tables A6–A7, the end-to-end cost in Fig. 5a, and the stability explanation in A.3.3.
  5. The readout gap currently cannot be interpreted as independently quantifying the loss caused by stale feature statistics. Sec. 3.2 Finding 2/3 attributes the adapted encoder's ridge gap, after subtracting the frozen-encoder residual, to moving geometry, but ridge and the linear probe differ in loss, regularization, and hypothesis class, and the residual can vary with representation. A.5 itself also acknowledges, in the frozen shadow case, the representation dependence of a nonzero gap. More seriously, Table A9 rows 1–3 keep the same probe, yet row 3 adds a ReLU RP. The deployed classifier is already nonlinear with respect to the representation in which the probe resides, so the claim in A.3.4 that the probe “upper-bounds” its capability does not hold. One should match ridge, regularization, and projection across methods on the same final deployed features, and compare historically accumulated statistics with re-extracted full-data statistics. Otherwise Eq. (2) is merely a descriptive identity.
  6. Numerous defensive statements. Although the authors honestly disclosed the AI assistance in the writing process, the large number of defensive statements in it should be revised. Almost every part that indicates the role/contribution of the proposed method is accompanied by a constraining/qualifying statement. These defensive statements run from the abstract to the experimental section. For example, the abstract mentions "Representations are therefore not the bottleneck, the decision layer is," and then immediately follows with "This does not imply that adaptation is unnecessary, xxxx.". The abstract only needs to concisely state the findings and contributions, it should not make a simple point so complicated. The paper uses more than 20 instances of "rather than," most of which serve only to guard against criticism from some aspect. For example, the Intorduction section mentions "rather than guaranteeing strict feature invariance". Such a statement contributes nothing to understanding the paper.
Questions for authors
  1. The authors should further clarify the boundary between the proposed method and existing work, i.e., what is the direct contribution of the proposed method?
  2. Did RanPAC directly use the official open-source code, or did the authors reproduce it themselves? The authors should check whether the issue arises from code implementation.
  3. Some figures/tables conflict with the paper’s statements. Is this a writing issue or a data issue? The authors should check the data in the figures/tables and the statements in the relevant sections.
  4. The anonymous code implementation does not match the paper’s statements. Is this a code implementation issue, or did the additional experiments use another version of the code?
6 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1)

Summary. Thank you for the review, checking the code, and recognizing the motivation, experimental breadth, and ablations. We conducted the requested comparison between accumulated statistics and statistics refitted on all seen data. Without merging, stale statistics account for 89–96% of the readout gap, while SASKIA's merging removes 61–99% of this loss (W5). Regarding implementation discrepancies (W4), the initially shared anonymized repository was outdated. We apologize and have corrected the link. We also corrected the parameter counts while confirming that the method remains under 1% (W3), and strengthened the positioning against SSF, FiLM, EASE, and LDC (W1). We address each point below.

W1 & Q1: Positioning against existing work and our direct contribution

  • Contribution. Stored statistics becoming outdated as features drift is well established. SDC (CVPR 2020), ADC (CVPR 2024), and LDC (ECCV 2024) estimate feature drift and correct stored prototypes. EASE addresses related mismatch through prototype completion when new subspaces are added, while MOS and TUNA correct classifiers using generated or stored class statistics. Section 2 now acknowledges these contributions. Our contribution is instead a measurement separating feature quality from accumulated-classifier validity throughout continual post-training. Together with the refitted-classifier control (W5), it isolates performance loss due to stale statistics. Without merging, stale statistics account for 89–96% of the readout gap, while feature quality remains largely preserved.

  • Relation to drift compensation. Existing methods allow features to evolve and subsequently repair accumulated statistics. SASKIA takes a complementary approach by limiting feature drift at its source: adaptive components are shared and gradually merged, preserving the validity of previously accumulated statistics without a separate classifier repair step. The approaches are complementary: drift compensation could also be applied on top of SASKIA, while our measurement quantifies residual drift effects.

  • SSF and FiLM. These are established building blocks. Section 4 now explicitly cites SSF (Lian et al., NeurIPS 2022) for the global module and FiLM-style conditioning (Perez et al., AAAI 2018) for the instance-level module.

  • Our contributions (Q1). We summarize our contributions in three parts: (1) a measurement framework decomposing the gap between feature-supported and continual-classifier performance into classifier-type limitations and stale statistics (W5); (2) the empirical finding that features remain largely intact during continual post-training, while accumulated classifier statistics become outdated and account for most resulting performance loss; and (3) the design principle of sharing adaptive components and gradually merging them to preserve classifier validity, instantiated by SASKIA. We do not claim SSF, FiLM, random projection, or ridge regression as novel.

W2 & Q2: RanPAC's failures in B50 and which implementation we used

RanPAC's collapses occur only in the large-first-task setting (B50, Figure A3). In all main experiments, RanPAC runs stably: its accuracy curves in Figures 4, A1, and A2 decline smoothly across all four benchmarks, with no task-level collapse. Thus, comparisons in Tables 3, A2, and A3 involve no solver failure. We use the PILOT implementation (Sun et al., 2023) with its released RanPAC configuration (adapters tuned in the first session, M = 10,000), as stated in A.1. RanPAC uses more memory to ensure numerical stability: it stores G as a full 32-bit matrix and uses direct solve, requiring no safeguards. Our safeguard is only needed when storing statistics in 16-bit to halve memory, a stricter setting RanPAC does not use. We likewise use no safeguards with fp32.

W3 & Q3: Text contradicting figures and tables

We corrected all three discrepancies. Two were writing errors and one a counting error; none changes the main conclusions.

  • Figure 5b. Mean-pooling performs best at widths 128, 256, and 512; concatenation is best at 768 (58.10 vs. 57.40). The best overall setting is mean-pooling at width 256 (58.77). The text now reflects this.

  • Table A9. Rows 4 → 5: Shared SSF slightly lowers probe accuracy but narrows the gap and improves final accuracy. We now clarify this interpretation. A weaker probe alongside a better continual classifier supports our analysis: gains come from a classifier that remains valid, not improved features.

  • Parameter counts. Following the original SSF, the implementation modulates the qkv projection, MLP hidden layer, patch embedding, and final norm. The correct counts are 205,824 for SSF and 592,128 for the hypernetwork, totaling 797,952. This is 0.93% of ViT-B/16's 85.8M parameters, so "under 1%" remains correct. The ratio to RanPAC's adapters is approximately 5.8×. We corrected Table A6, the Params column in Table A9, and the text.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2)

W4 & Q4: Description of memory and solver description in appendix.

The anonymized repository was an outdated version, which we inadvertently overlooked. We apologize for this mismatch and have corrected the repository link. The paper's results and all reported experiments were produced using the updated codebase. Therefore, the memory requirements and analytical solver implementation described in the paper are accurate. However, we also understand that the ridge solver implementation may not have been sufficiently clear, and have expanded its explanation in the revised paper.

W5: The readout gap and the loss from stale statistics

Thank you. Eq. 2 is a diagnostic by design: each method is evaluated as it was proposed, and the gap measures how much of what its features linearly support its own classifier does not deliver. Which part of that gap is due to stale statistics is a separate, causal question, and the control you describe is the right test for it. We ran it, stored statistics against statistics re-extracted from all data, and it supports the claim directly. After every task, we refit the ridge classifier from scratch, on all data seen so far, using the current model. Both this refitted classifier and the probe need all seen data at once, so they are not available in continual learning; we compute them only as measurements, and SASKIA never uses them. Everything else is the same as for the continual classifier: the projection, the training examples, the ridge objective and how the regularization is chosen. The only difference is that the continual classifier's statistics were collected earlier, with older versions of the model. So the accuracy difference between the refitted and the continual classifier is exactly the cost of stale statistics. The gap then splits into two parts:

probe − continual = (probe − refitted)-> classifier type + (refitted − continual) ->stale statistics

Benchmark and settingContinualRefittedLost to stale statisticsProbeProbe − refitted
CIFAR, no merging82.3290.548.2290.890.35
CIFAR, SASKIA90.290.310.1192.332.02
CUB, no merging89.1090.841.7491.050.21
CUB, SASKIA89.890.110.3191.391.28
IN-R, no merging76.1780.424.2580.770.35
IN-R, SASKIA79.080.661.6681.240.58
IN-A, no merging56.1060.974.8761.320.35
IN-A, SASKIA58.860.091.2962.362.27
  • Without merging, old statistics cost 8.2 / 1.7 / 4.3 / 4.9 points (CIFAR / CUB / IN-R / IN-A), which is 89–96% of the whole gap. With SASKIA's merging, this drops to 0.1 / 0.3 / 1.7 / 1.3.

  • Without merging, the features are only a little worse (probe 0.3–1.4 points below SASKIA's), but the classifier the model actually uses is 0.7–7.9 points worse. So the damage is in the stored statistics, not in the features.

  • SASKIA's readout gap in Table 3 separates into classifier-type and stale-statistics costs: CIFAR (2.0 + 0.1), CUB (1.3 + 0.3), IN-R (0.6 + 1.7), and IN-A (2.3 + 1.3) points. The classifier-type term measures the cost of using an incrementally accumulable, data-free ridge classifier instead of a probe refitted on all seen data; it is distinct from staleness, which arises from accumulating statistics under evolving representations.

So when the model keeps changing, nearly all of the readout gap is stale statistics, and SASKIA's merging is what removes it.

"Upper bound." Rows 1–3 of Table A9 use the same frozen model, so the probe is the same by construction, while the classifier adds a nonlinear projection. The probe is therefore not an upper bound, and we drop that wording in A.3.4.

Across methods. Table 3 compares methods that each keep their own classifier, so the gaps there describe each method; on their own they do not show that one method's statistics are better aligned. Methods such as EASE grow their feature space with every task, so a classifier accumulated over a fixed space is not defined for them in the same way. This is why each method is evaluated with the classifier it was designed for.

W6: Too many defensive statements

Our aim with these qualifiers was to avoid giving a false impression of what the results show, but we agree that, taken together, they make the text sound defensive and the main points harder to follow. We now revised the text in many places to improve readability.

We believe our replies and the revised paper address all the concerns you raised. If so, we would be grateful if you would consider updating your score.

Dear Authors,

Thank you for your constructive responses and additional experiments. The comparison between accumulated and refitted statistics substantially addresses my concern about distinguishing stale-statistics effects from differences between classifiers. I also appreciate the corrections to the numerical statements and parameter counts, and the clarification that the probe is not an upper bound.

However, some concerns remain. The new experiment supports the explanation within SASKIA and its no-merging variant, but does not fully justify the broader conclusions about shared versus task-specific adaptation. The clarification regarding RanPAC identifies the implementation and limits the failures to B50, but does not establish their cause or validate the proposed explanation for the stability difference.

Overall, the rebuttal resolves several concerns and strengthens the paper, but the remaining issues leave me unconvinced that it warrants acceptance. I therefore maintain my rating of 2.

Best regards,
Reviewer cekJ

Reviewer_ERD3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the gap between what a continually adapted visual representation can support and what its incremental classifier actually achieves. At each stage, the authors fit a joint linear probe on the current encoder and compare it with sequentially maintained gradient and ridge readouts. They call the accuracy difference the readout gap and argue that much of it comes from classifier statistics becoming stale as features evolve. The proposed SASKIA method combines first-stage shared scale-and-shift adaptation, an instance-conditioned hypernetwork updated through EMA merging, a fixed random projection, and an accumulated closed-form ridge classifier.

Critical strengths
  1. The diagnosis goes beyond final accuracy. Figure 1 compares CUB increments of 5, 10, and 20 classes. The final linear-probe accuracies remain similar, while the performance of accumulated ridge and gradient readouts deteriorates with more updates; the reported gap grows from roughly 12 to 18 points. Section 3.2 also compares linear, NCM, LDA, and ridge classifiers on fixed features (Table 1), then examines ridge under frozen and adapting encoders. Taken together, these results show that readout estimation and the temporal compatibility of classifier statistics with features deserve attention independently of final representation quality.

  2. The evaluation distinguishes different training budgets and includes useful component controls. In Table 3, SASKIA reports final accuracies of approximately 89.8% on CUB and 58.8% on ImageNet-A under the offline-to-online protocol, along with readout-gap measurements. Table 4 evaluates methods under their always-offline configurations, and Appendix Table A9 separates frozen ridge, random projection, shared scale-and-shift, the hypernetwork, and EMA merging. These experiments make the performance claims easier to evaluate than a single leaderboard comparison would.

Critical weaknesses
  1. The readout-gap measurement does not uniquely identify stale statistics as the cause of the remaining error. Section 3.1 defines Gt=AtLP−AtCLG_t=A_t^{LP}-A_t^{CL}, but the two readouts can differ in estimation objective, regularization, feature projection, and access to labels. In particular, the intermediate-stage probe is fitted using all classes, including classes not yet encountered by the incremental learner, and is evaluated over the full label space. The paper acknowledges that the early trajectories, before the parity point, are not directly comparable. Ridge statistics are order-invariant when the encoder is fixed, which removes one source of order dependence; it does not by itself equate ridge with the separately fitted linear probe or rule out conditioning and estimator effects. The frozen-ridge reference in Section 3.2 helps, but attributing all excess gap to geometric drift remains an inference. The decisive control would hold the final encoder ΦT\Phi_T, training examples, projection, and ridge objective fixed, and compare the historically accumulated G,CG,C against statistics recomputed from the same previously seen examples under the final encoder. The difference would directly quantify the loss due to stale statistics. Without this experiment, the paper's strongest causal claim is not established.

  2. The implementation contradicts parts of the stated stability and task-free setting. The abstract says that the method requires no explicit task boundaries. Appendix A.2, Algorithm 1, takes a sequence of tasks {Dt}\{D_t\} as input and performs EMA merging, ridge-statistic updates, and an active-to-shadow reset at the end of each task. These operations explicitly depend on stage boundaries. If the intended claim is only that task identity is not required at inference, it should be stated accordingly. Section 4 also argues that the projected space is identical across tasks because the random matrix RR is fixed. The actual feature map is ψt(x)=ReLU(R⊤z~t(x))\psi_t(x)=\mathrm{ReLU}(R^\top\tilde z_t(x)), and z~t\tilde z_t changes when the hypernetwork changes. EMA can smooth updates, but it neither preserves historical features exactly nor guarantees that accumulated ridge statistics remain valid. The gains from EMA in Appendix Table A9 could also reflect ordinary weight-averaging regularization. Direct measurements of feature drift and comparisons with simple averaging or feature-alignment controls are required. Furthermore, Table 4 shows 59.21% for SASKIA and 59.78% for TUNA on ImageNet-A in the always-offline setting, so claims of leading under every budget should be qualified.

Questions for authors
  1. Using the same final encoder, projection, training examples, and ridge penalty, what accuracy difference remains between historically accumulated ridge statistics and statistics recomputed from all seen training data? How much of the gap is thereby isolated as feature-statistic staleness?
  2. Without task boundaries, what triggers the EMA merge, ridge update, and active-hypernetwork reset in Algorithm 1? Can the method be evaluated under genuinely boundary-free streaming rather than merely task-agnostic inference?
  3. How strongly do measured feature displacement ∥ψt+1(x)−ψt(x)∥\|\psi_{t+1}(x)-\psi_t(x)\| and historical-statistic drift predict the readout gap? How do ordinary EMA, a frozen encoder, and explicit feature alignment perform under the same update budget?
5 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1)

Summary. Thank you for the detailed review, and for noting that the diagnosis goes beyond final accuracy and that the component controls make the claims easier to evaluate. We ran the decisive control you describe. When the encoder is updated without merging, stale statistics are 89–96% of the readout gap, and SASKIA's merging removes 61–99% of that loss (W1 & Q1). We also measured drift directly and compared EMA with simple averaging, feature alignment and a frozen encoder (W2b, W2c).

W1 & Q1: Stored vs recomputed statistics

After every task, we refit the ridge classifier from scratch, on all data seen so far, using the current model. Both this refitted classifier and the probe need all seen data at once, so they are not available in continual learning; we compute them only as measurements, and SASKIA never uses them. Everything else is the same as for the continual classifier. So the accuracy difference between the refitted and the continual classifier is exactly the cost of stale statistics. The gap then splits into two parts:

probe − continual = (probe − refitted)-> classifier type + (refitted − continual) ->stale statistics

The first part ("classifier type") holds the estimator, regularization and conditioning effects you list, so they are no longer mixed with stale statistics.

Benchmark and settingContinualRefittedLost to stale statisticsProbeProbe − refitted
CIFAR, no merging82.3290.548.2290.890.35
CIFAR, SASKIA90.290.310.1192.332.02
CUB, no merging89.1090.841.7491.050.21
CUB, SASKIA89.890.110.3191.391.28
IN-R, no merging76.1780.424.2580.770.35
IN-R, SASKIA79.080.661.6681.240.58
IN-A, no merging56.1060.974.8761.320.35
IN-A, SASKIA58.860.091.2962.362.27
  • Effect of stale statistics. Without merging, stale statistics account for 8.2 / 1.7 / 4.3 / 4.9 points (CIFAR / CUB / IN-R / IN-A), explaining 89–96% of the overall gap. With SASKIA's merging, this drops to 0.1 / 0.3 / 1.7 / 1.3 points. Importantly, the features themselves are only marginally worse (probe accuracy drops by 0.3–1.4 points), whereas the classifier actually used loses 0.7–7.9 points. This indicates that the main damage comes from stale classifier statistics rather than feature degradation.

  • Decomposing SASKIA's gap. SASKIA's readout gap in Table 3 separates into classifier-type and stale-statistics costs: CIFAR (2.0 + 0.1), CUB (1.3 + 0.3), IN-R (0.6 + 1.7), and IN-A (2.3 + 1.3) points. The classifier-type term measures the cost of using an incrementally accumulable, data-free ridge classifier instead of a probe refitted on all seen data; it is distinct from staleness, which arises from accumulating statistics under evolving representations.

This is the experiment you identified as decisive, and it supports the causal claim: when the model keeps changing, nearly all of the readout gap is stale statistics, and SASKIA's merging is what removes it (your Q1).

Label access. The table is for the last task, where the probe and the classifier have seen exactly the same classes and data; our main claims rest on this point.

W2a & Q2: Task boundaries, and running on a stream without them

We have clarified the wording to avoid ambiguity. We follow the standard class-incremental setting, where task boundaries are known during training but task identity is unavailable at test time. SASKIA uses these boundaries to trigger the merge and reset operations, as other compared methods use them for their per-task updates. In contrast, the classifier does not depend on task boundaries: its statistics are accumulated sample by sample, allowing it to be solved at any point during training. We therefore do not claim that SASKIA operates without task boundaries. A boundary-free stream falls outside the class-incremental protocol considered in our work and would not allow a direct comparison with existing methods.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2)

W2b & Q3: A fixed projection does not keep features fixed; does drift predict the gap?

You are right, and lines 369–370 overstated this; we now rewrite it. The random projection never changes, but the features it produces do change whenever the hypernetwork is updated. EMA does not stop this; it only slows it down. We now measure how much the features move:

α (IN-A)Feature changeChange in stored statistics
0 (no merging)0.290.28
0.50.230.17
0.90.110.08
0.95 (SASKIA)0.070.06
0.990.020.02
1 (frozen)0.000.00

How we measure. Feature change: We take the first task's test images, compute their features right after the first task, and compute them again after the last task. Feature change is the average distance between the two, relative to the average size of the features, so 0.29 means the features moved by 29% of their size. Change in stored statistics: the classifier's statistics are added up task by task, each with the model of that time; we recompute the same statistics with the final model and take the relative difference, so 0.28 means the stored statistics are 28% off.

Stronger averaging (higher α) means less movement. At SASKIA's setting both move about four times less than without merging (0.07 vs 0.29), and the accuracy lost to old statistics drops from 4.9 to 1.3 points (W1).

W2c & Q3: Is the EMA gain just weight averaging? EMA vs frozen vs feature alignment

EMA is weight averaging and may also act as a regularizer; our claim does not depend on that either way. What we claim is that merging keeps the stored statistics in line with the current features, and the data show this is where the accuracy comes from:

  • In Table A8 (IN-A), between α = 0.9 and 0.99 the probe changes by only 0.21 points, but final accuracy changes by 1.24 points. So α changes how well the stored classifier fits, not how good the features are.

  • Compared with no merging, SASKIA's probe is 1.0 point higher, but its final accuracy is 2.7 points higher, and the accuracy lost to old statistics falls from 4.9 to 1.3 points (W1).

Below, we share the controls you asked for, with the same training budget:

Rule (IN-A)What it doesFinal accuracyFeature change
EMA, α = 0.95 (SASKIA)Blends 5% of the newly trained model into the deployed one after each task58.8 ± 1.20.07
Uniform averageAverages all task models equally57.67 ± 1.250.17
Polyak averageAverages the training steps within a task, with no anchor to the deployed model56.97 ± 1.230.20
Feature alignment, weight 0.1 / 1 / 10No merging, plus a training penalty that keeps new features close to the old ones55.81 / 56.37 / 56.590.29 / 0.27 / 0.26
No mergingTakes each newly trained model as is56.100.29
Frozen after first taskNever updates the deployed model again56.810.00

What this shows:

  • Averaging alone is not enough. If EMA helped only as a regularizer, any averaging would do about as well. Uniform and Polyak averaging keep the features less stable (0.17–0.20 vs 0.07) and score 1.1–1.8 points lower.

  • Penalizing feature change directly does not work. Feature alignment, at every strength we tried, moves the features about as much as no merging and scores no better.

  • Freezing avoids stale statistics but cannot adapt. A frozen model's statistics never go out of date, but it cannot adjust to the classes that arrive later, and it scores lower than SASKIA.

So what matters is keeping the features and the stored classifier in line, and EMA does this best among the rules that keep updating the model.

W2d Table 4: SASKIA trails TUNA on ImageNet-A in the offline setting

Agreed, and Section 5 already says so: "SASKIA leads on CIFAR, CUB and IN-R, trailing TUNA by 0.6% on IN-A alone." The sentence "leads on every benchmark" is about the offline-to-online setting of Table 3. We now add the same note to the caption of Table 4. Two facts give context for that one result. TUNA stores per-class feature means and covariances to correct its classifier (Appendix A.4.1), and SASKIA does not. And when each task is seen only once, TUNA loses 5.45 points on IN-A while SASKIA loses 0.44, so SASKIA leads TUNA by 4.4 points in the setting we consider most realistic.

We believe we have answered all your concerns and would be grateful if you would consider raising your score. We are happy to discuss any remaining questions.

Dear Reviewer,

We would like to briefly follow up on our rebuttal and the changes made to the manuscript. We believe and hope that our responses have adequately addressed the concerns raised in your review and strengthened the paper accordingly. We would be grateful if you could reconsider your initial assessment.

Thank you once again for your time and thoughtful consideration!

Reviewer_oAEsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the gap between jointly probed representation quality and continual classification accuracy. It proposes SASKIA, combining initial scale-and-shift adaptation, an EMA-updated hypernetwork, and an accumulated ridge classifier.

Critical strengths
  • Results are consistently strong across four benchmarks, two pretrained model configurations, and five seeds. The additional offline comparisons help address concerns about insufficient training of competing methods.
  • The component and EMA ablations are useful. They demonstrate improvements beyond the underlying analytic classifier and initial adaptation.
Critical weaknesses
  • Lines 9–11 train on the current task, merge the hypernetwork, and then accumulate statistics over that same task using the updated model. This appears to require another pass or cached task data/features. Merging and resetting also explicitly use task boundaries. These details affect both protocol fairness and memory accounting.

  • The readout gap does not isolate stale-statistic error. It also reflects differences between probe and ridge estimation, regularization, and feature mappings. Table A9 even keeps the probe unchanged when adding a nonlinear random projection. Consequently, a smaller gap does not establish that historical statistics remain better aligned. A matched ridge classifier refitted on current features is needed to separate these effects.

Questions for authors
  • How are Algorithm 1’s statistics computed without revisiting or buffering the current task, and how are updates triggered without task boundaries?
  • How much error remains when ridge statistics are recomputed using the current encoder, with the same projection and regularization?
  • Do the gains persist when RanPAC receives the same numerical safeguards, particularly in the runs with reported solver failures?
3 comments

Summary. Thanks for the review, and noting the consistent results with useful ablations. We ran the matched refit. When the encoder is updated w/o merging, stale statistics account for 89–96% of the readout gap, and SASKIA's merging removes 61–99% of that loss (W2). We answer each point below.

W1 & Q1: Second pass, task boundaries and memory

Second pass. After training on each task, the merged model makes one additional forward pass over that task's images, without gradients, to accumulate the classifier statistics. This is standard practice for classifiers built from class statistics: RanPAC collects its ridge statistics this way, while SimpleCIL, EASE, MOS, and TUNA use a similar post-training pass to compute prototypes or class statistics. Thus, all compared methods follow the same protocol. We now make this step explicit in Algorithm 1.

Task boundaries. We follow the standard CIL setting, where task boundaries are known during training but unavailable at test time. SASKIA uses task boundaries to trigger the merge and reset operations, as other methods use them for their per-task updates. The classifier, however, accumulates statistics sample by sample, independently of task boundaries. We clarify this distinction in the abstract and Section 4.

Memory. No images are stored. Across tasks we keep only the classifier statistics. The current task's features are held only until the regularization strength is chosen, as in RanPAC.

W2 & Q2: Separating stale statistics.

After every task, we refit the ridge classifier from scratch, on all data seen so far, using the current model. Both this refitted classifier and the probe need all seen data at once, so they are not available in continual learning; we compute them only as measurements, and SASKIA never uses them. Everything else is the same as for the continual classifier. The only difference is that the continual classifier's statistics were collected earlier, with older versions of the model. So the accuracy difference between the refitted and the continual classifier is exactly the cost of stale statistics. The gap then splits into two parts:

probe − continual = (probe − refitted)-> classifier type + (refitted − continual) ->stale statistics

The first part ("classifier type") holds the estimator, regularization and conditioning effects you list, so they are no longer mixed with stale statistics.

Benchmark and settingContinualRefittedLost to stale statisticsProbeProbe − Refitted
CIFAR, no merging82.3290.548.2290.890.35
CIFAR, SASKIA90.290.310.1192.332.02
CUB, no merging89.1090.841.7491.050.21
CUB, SASKIA89.890.110.3191.391.28
IN-R, no merging76.1780.424.2580.770.35
IN-R, SASKIA79.080.661.6681.240.58
IN-A, no merging56.1060.974.8761.320.35
IN-A, SASKIA58.860.091.2962.362.27
  • Effect of stale statistics. Without merging, old statistics cost 8.2 / 1.7 / 4.3 / 4.9 points (CIFAR / CUB / IN-R / IN-A), which is 89–96% of the whole gap. With SASKIA's merging, this drops to 0.1 / 0.3 / 1.7 / 1.3. Features are only a little worse (probe 0.3–1.4 points below SASKIA's), but the classifier the model actually uses is 0.7–7.9 points worse. So the damage is in the stored statistics, not in the features.
  • Decomposing SASKIA's gap. Table 3 separates into classifier-type and stale-statistics costs: CIFAR (2.0 + 0.1), CUB (1.3 + 0.3), IN-R (0.6 + 1.7), and IN-A (2.3 + 1.3). The classifier-type term measures the cost of using incrementally accumulable, data-free ridge classifier instead of a probe refitted on all seen data; it is distinct from staleness, which arises from accumulating statistics under evolving representations.

So a smaller gap does mean better-aligned statistics: when the model keeps changing, nearly all of the gap is stale statistics, and SASKIA's merging is what removes it.

Table A9. Rows 1–3 use the same frozen model, so the probe is the same by construction, while the classifier adds a nonlinear projection. So the probe is not an upper bound there, and we drop that wording.

Q3: RanPAC and numerical safeguards.

The solver failures occur only in the large-first-task setting (Figure A3). In all main experiments RanPAC runs stably with no collapse at any task.

We want to clarify that RanPAC is more generous in terms of memory to keep the solution safe; therefore needs no numerical safeguards: it keeps G as a full 32bit matrix and uses direct solve. Our safeguard only enters when we store the statistics in 16bit to halve memory, a stricter setting RanPAC does not use. When we use fp32, we do not have any safeguards as well.

We would be grateful if you would consider raising your score. If anything remains unclear, we are glad to clarify during the discussion period.

Dear Reviewer,

We wanted to kindly follow up on our rebuttal and the revised manuscript. We would be grateful to hear whether our revisions have addressed your concerns or if there are any remaining questions we could clarify before the discussion period ends. We sincerely believe and hope that our responses and revisions have sufficiently addressed your concerns. In light of the positive reassessment by another reviewer, we would also greatly appreciate your reconsideration of your current score and any possible upward adjustment, if you feel that our effort in clarifications and additional results warrant it.

Thank you once again for your time and consideration!

We would like to kindly make one final request for your reconsideration of the current score, given that today is the final day of the discussion period. We have made considerable efforts to address your concerns through our rebuttal and manuscript revisions, and we believe the revised paper is substantially stronger in its contributions, experimental grounding and results. Given the borderline nature of the current assessment, we believe these changes may warrant a more favorable evaluation and we would be very grateful if you could reconsider your assessment in light of these clarifications and revisions. Even a slight adjustment to your score would make a meaningful difference to the paper's final outcome. Thank you again for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers, ACs and PCs

We thank the Area Chair, the Program Chairs and all three reviewers for their time and careful reading. The reviews were detailed, and several of them engaged with our code as well as the paper. We addressed every point raised, and we believe the paper is substantially stronger and cleaner as a result. The revised PDF marks all changes in blue. The main concerns and how we addressed them are summarized below:

1. A direct test of the central claim. All three reviewers asked the same question: is the readout gap caused by stale statistics, or by differences between the probe and the ridge classifier? We ran the control they proposed. After each task we refit the ridge classifier from scratch on all seen data under the current encoder, with everything else unchanged, which splits the gap exactly into an estimator term and a stale-statistics term (new Eq. 3): without merging, stale statistics account for 89–96% of the gap, while the features stay nearly as good. SASKIA's merging reduces this loss to 0.1–1.7 points (new Table 5, Appendix A.3.5).

2. Precise protocol and task boundaries. The paper now states the task boundaries of the class-incremental setting, that the classifier itself does not need them, and that no task identity is used at test time. Algorithm 1 clearly indicates that now.

3. Positioning. Section 2 now credits SSF, FiLM, the drift-compensation line (SDC, ADC, LDC and recent work on transition operators, bidirectional alignment and analytic drift compensation), and the classifier corrections in EASE, MOS and TUNA. It states our contribution precisely: a measurement of where the gap comes from, and a design that keeps every adaptive part shared and slowly merged.

4. Clarity of the text. We corrected every statement that disagreed with a figure or table, dropped the "upper bound" wording for the probe, rewrote the abstract and introduction without defensive qualifiers, and added appendix sections on measuring stale statistics and on encoder drift.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.