acceptodds
Withdrawn from ICLR 2027

Orthogonal Selection of Gradient-Free Task Posteriors for Class-Incremental Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In class-incremental learning (CIL), a prediction is correct only when both within-task prediction (WP) and task-id prediction (TP) succeed. Methods built on a frozen pre-trained model realize TP through prompt matching, expert routing, or task-identity heads and learn WP within the selected modules. Yet, their modules and classifier heads are trained with a cross-entropy loss restricted to the classes of the current task. This paper shows that this objective is invariant to every logit shift that is constant within each task, and its gradient has no component in the task-level subspace of the logits, which affects not WP but TP. The same subspace is exactly the set of additive corrections that leave WP unchanged for every logit vector. Building on these two results, we propose a method that supplies TP through orthogonal selection: a task posterior, computed in closed form from class-wise first- and second-order moments of frozen features with shared whitening and rank-4 class-specific corrections, is added to the logits without any gradient path. Across 15 methods, five datasets, 14 task splits, and five seeds (1,050 runs under a common training protocol), TP fails in 92.8–97.4% of the errors of every existing method, and the variation of final accuracy across methods is almost entirely a variation of TP. The proposed method raises TP by 16.70 points and final accuracy by 15.74 points while leaving WP unchanged, reaches 78.25% average final accuracy, and its gain increases monotonically with the number of tasks on all five datasets.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
class-incremental learning, continual learning, task-id prediction, within-task prediction, pre-trained models, prompt-based continual learning, exemplar-free, Gaussian task posterior

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ahEtOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a logit adjustment strategy to help select the task ID for exemplar-free class-incremental learning. Experiments on multiple datasets supports the efficacy of the proposed strategy. While theoretical analysis is provided trying to justify the proposed strategy, possibly serious issues exist in the theoretical analysis.

Critical strengths

The main contribution lies in the logit adjustment strategy. Specifically, class-wise Gaussian distributions are used to estimate the probability of any test image belong to each task, and such task-level probablity is used to adjust the logit of each task's classifier head for task ID determination.

Critical weaknesses
  1. Crucial justification of the proposed method is missing. Particularly, why are the mixture of class-wise Gaussian distributions used to estimate the probability of the test input belonging to each task, and why is such estimate resonable? Why is the adjustment of logit helpful for task ID selection? For a test image which is visually similar to two classes from two different tasks, how can such adjustment help select the correct task?
  2. The theoretical analysis is questionable. First, some key variables and concepts are introduced without clear definition and elaboration. For example, what does the C-dim vector l_j look like, and how does it represent the classes of task j? Isn't the span of l_1, l_2, ..., l_t the whole R^C space? The descriptions of the span (Lines 151-155) are unclear, with new notations and conclusions introduced but without explanation. Second, proofs of propositions and lemma lack enough elaborations and might be partly erroneous. For example, for the proof of Proposition 1, why does a vector v take a constant value vu for all logit elements? where is v from, and why is it a constant? For the proof of Proposition 2, what is g(x) and where is it from? why are elements of all g(x) identical? If g(x) is funciton of the test input, it is morel likely a vector containing various element values. For Lemma 2, pi(x) appears and is not used afterward. For Lemma 3, what is Eq (4) used for?
  3. The empirical evaluations are not sufficient and one key setting is questionable. Considering that the ViT backbone is pre-trained on the natural image dataset ImageNet-21k, continual learning tasks should include image dataset from other domains (e.g., medical image dataset) to confirm the generalization of the proposed method. For the hyper-parameter setting, usign 5 epoches for task learning is not a widely adopted protocol.
Questions for authors
  1. Why is the adjustment (Eq. 5) reasonable and effective?
  2. Please provide clear and detailed proofs, with every notation explicitly elaborated.
  3. Please provide additional empirical evaluation to suppor the generalizaiton of the method.
2 comments
AuthorsOpenReview ↗

Response to Reviewer ahEt (1/2)

W1. Justification of the method

The three questions are answered in order, and a direct measurement of the case raised by the reviewer, visually similar classes that belong to different tasks, has been added.

(i) Why a mixture of class-wise Gaussians. A task is a union of distinct classes and is multimodal in feature space. One Gaussian per task gives a TP gain of +4.61 points, whereas the class mixture gives +7.49. The effect of each component, as average Final over the 14 settings, is:

ControlClass means only (no whitening)+ shared whitening+ rank-4 class correction (proposed)
62.5169.7777.3378.25

(ii) Why adjusting the logits helps task selection. By Lemma 2, the predicted task is arg⁡max⁡j[mj(x)+βγj(x)]\arg\max _ j[m _ j(x)+\beta\gamma _ j(x)]. The offsets among the block maxima mjm _ j are not determined by training (Proposition 1), so the task posterior decides this comparison.

(iii) Similar classes from two tasks. The head logits of two such classes are never trained against each other, so their difference is arbitrary, whereas the Gaussian likelihoods are computed in a whitened space shared by all classes and are directly comparable. Classes were ranked by the similarity to their nearest class in another task, and the test samples of the top 10% were compared with the rest.

SettingTP gain, top-10% most similarTP gain, restCross-task errors corrected
CIFAR-100 10T+14.38+5.6042.4%
ImageNet-R 20T+11.68+6.9526.9%
ImageNet-A 20T+11.45+12.4128.4%
CUB-200 20T+32.65+26.7272.4%
VTAB 10T+57.19+42.2281.1%

The case raised by the reviewer is precisely where the posterior is most effective: on samples with a similar class in another task the TP gain reaches 57 points, and 27–81% of the predictions that fall into a wrong task are corrected. This analysis has been added to the manuscript.

W2. Theoretical analysis

All propositions and proofs hold, and the points raised are resolved by making the notation explicit.

  • 1j\mathbf{1} _ j is the vector whose entries are 1 on the classes of task jj and 0 elsewhere. For C=4C=4, $\mathcal{Y} _ 1=\lbrace 1,2\rbrace $, $\mathcal{Y} _ 2=\lbrace 3,4\rbrace $: 11=(1,1,0,0)\mathbf{1} _ 1=(1,1,0,0), 12=(0,0,1,1)\mathbf{1} _ 2=(0,0,1,1), and dim⁡VT=t=2<C\dim\mathcal{V} _ {\mathrm T}=t=2\lt C. The span equals RC\mathbb{R}^C only when every task consists of a single class.
  • Lines 151–155: ΠT\Pi _ {\mathrm T} replaces every logit by the mean logit of its task, and ΠW=I−ΠT\Pi _ {\mathrm W}=I-\Pi _ {\mathrm T}.
  • The constant in the proof of Proposition 1: v∈VT\mathbf{v}\in\mathcal{V} _ {\mathrm T} means v=∑jνj1j\mathbf{v}=\sum _ j\nu _ j\mathbf{1} _ j, so v\mathbf{v} equals νt\nu _ t on the classes of task tt. It is constant within a task, not over all logits, and this is the definition of VT\mathcal{V} _ {\mathrm T}.
  • g(x)\mathbf{g}(x) in Proposition 2 is defined in Definition 2 by gc(x)=γτ(c)(x)g _ c(x)=\gamma _ {\tau(c)}(x). It varies with the input and across tasks and is identical only within a task.
  • π(x)\boldsymbol\pi(x) in Lemma 2 is instantiated by p(j∣x)p(j\mid x) in Eq. 5 and is used in the analysis of the selection strength.
  • Eq. 4 in Lemma 3 is the quantity LcL _ c that Algorithm 1 computes at inference and is the basis of the inference cost.

The propositions were also verified numerically (float64, 10410^4 random logit vectors).

QuantityImageNet-R 20T split (C=200C=200)C=4C=4, 2 tasks
rank⁡VT\operatorname{rank}\mathcal{V} _ {\mathrm T}20 (=t=t)2 (=t=t)
Projection of the masked-CE gradient onto VT\mathcal{V} _ {\mathrm T} (max)1.2×10−161.2\times10^{-16}4.8×10−164.8\times10^{-16}
Change of the loss after adding v∈VT\mathbf{v}\in\mathcal{V} _ {\mathrm T} (max)7.1×10−157.1\times10^{-15}5.3×10−155.3\times10^{-15}
Changes of WP after adding v∈VT\mathbf{v}\in\mathcal{V} _ {\mathrm T}00
Counterexample of Proposition 2(ii) constructed for v∉VT\mathbf{v}\notin\mathcal{V} _ {\mathrm T}100%100%

In actual training, the relative deviation between the recursion of Corollary 1 and the measured task sums is at most 3.7×10−43.7\times10^{-4}, and Δ\DeltaWP is 0.00 in all 56 method–setting pairs to which the posterior was added. A notation table, the two-task example above, and proofs expanded line by line have been added. The theory is exact and agrees with the experiments to machine precision; a reconsideration of the soundness assessment is respectfully requested.

AuthorsOpenReview ↗

Response to Reviewer ahEt (2/2)

W3. Other domains and the 5-epoch protocol

Both points are covered by the experiments, and the results show that the proposed method is most effective on domains far from the pre-training data.

VTAB combines five datasets from different domains, and CUB-200 is a fine-grained benchmark. The gain in final accuracy over the control, ordered by domain, is:

ImageNet-RCIFAR-100ImageNet-ACUB-200VTAB
+6.77+6.81+11.63+26.49+32.60

When the posterior is added to other classifiers, the gain is again largest on VTAB (APER-SSF +6.64, SimpleCIL +8.39, EASE +8.54). Since the posterior uses only statistics of frozen features, its role grows as the domain gap makes the training of the classifier harder.

Regarding the 5-epoch protocol, the 13 baselines were re-run under the settings of their own papers (up to 50 epochs), 910 runs reported in the supplement, and the ranking of Table 1 is preserved under those settings. The posterior involves no training, so the number of epochs affects only the base classifier.

Q1–Q3

Q1 is answered in W1, Q2 in W2, and Q3 in W3.

Reviewer_61YnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies class-incremental learning with a frozen pretrained backbone and decomposes accuracy into within-task prediction (WP) and task prediction (TP). It proves that masked cross-entropy is invariant to logit shifts that are constant within each task, that its gradient has no component in that task-level subspace, and that the subspace is exactly the set of additive corrections that leave WP unchanged for every logit vector. It then proposes an exemplar-free, gradient-free task posterior computed in closed form from class-wise first- and second-order moments of frozen features, with shared whitening and rank-4 class-specific corrections, added to the logits at fixed weight. Validation covers 15 methods, five datasets, 14 task splits, and five seeds: 1,050 runs under a common protocol plus 910 baseline runs under original settings. The audit reports TP in 92.8 to 97.4 percent of every method's errors, a +16.70 TP and +15.74 final-accuracy gain over the matched control with WP unchanged, first place in 12 of 14 settings, and gains that grow with task count. A supplement reports the design analysis, including a Gaussian-classifier-only variant that recovers most of the gain. Code is promised through an external anonymous link; no code is in the package.

Critical strengths

The audit is the most complete measurement of the WP/TP decomposition in this literature, and it reframes the field's failure mode. Across 1,960 runs, 15 methods, and 14 settings, task prediction accounts for 92.8 to 97.4 percent of every method's errors, and final-accuracy ranking coincides exactly with task-prediction ranking (Spearman 1.00) while within-task accuracy explains little (0.43). This is consequential because it changes what the community should measure, and the scale makes the claim credible.

The theory is exact and verified, and the paper's candor is exemplary. Proposition 2 characterizes the WP-preserving subspace precisely, including a converse, and its prediction, delta-WP = 0.00 for every finite beta, holds in the experiments and breaks exactly where the theory says it must. The paper also publishes the design analysis that prices its own method, which is the kind of disclosure reviewers should reward.

Critical weaknesses

The paper's own ablation prices the named contribution at about half a point. On ImageNet-R, the Gaussian-classifier-only variant recovers +6.30 of CLASP's +6.77 gain, so the orthogonal-selection apparatus contributes roughly 0.5 points, and the paper does not discuss this row's implication. It sells the method (first in 12 of 14, +15.74) rather than the audit that is its real contribution. This is consequential because the central claim is method superiority, and the paper's own design analysis contradicts it.

The comparison omits the competitor that beats the method and touches reported test data. RanPAC is cited, and even listed in the supplement, but never run; its published numbers sit above CLASP's on the headline datasets. FeCAM and SLCA are also missing. The common protocol was selected by a 72-configuration search on the ImageNet-A test split, and the fusion weight is the argmax of an ImageNet-R sweep, a headline dataset; the promised hyperparameter audit never appears. This is consequential because the first-place claim holds only within a pool that excludes the strongest competitor, under tuning that touches the reported data.

Questions for authors

Clarification requests (these could change my recommendation):

1- Where were the four method hyperparameters selected, and on what data? The audit promised on page 10 is not in the supplement.

2- Did the ImageNet-A test split used for the 72-configuration protocol search count among the reported runs?

3- What happens when the task posterior is added to APER-SSF, SimpleCIL, and EASE, the analogue of the table that adds it to the weak prompt baselines? If it lifts them, the paper undersells itself; if not, the first-place ranking rests on competitors being denied the component.

4- Can you run RanPAC, FeCAM, and SLCA under your protocol, or reconcile their published numbers with the reported 72.13 on ImageNet-R and 85.83 on CIFAR-100?

5- Given the Gaussian-classifier-only row, what does the trained prompt classifier add beyond the WP guarantee, and what is that variant's WP on datasets where the control is weak?

6- How do you explain SimpleCIL's gap to published values on three datasets and PEARL's 63.85-to-43.45 inversion between the common and original protocols?

7- Does Corollary 1's zero-sum recursion survive Adam in practice, and can you report the task-sum drift for Adam-trained baselines?

8- Can you report paired per-setting tests, and the standard error of the thinnest first place (ImageNet-R, 20 tasks)?

Suggestions for future work (not needed for my assessment):

  • Sweep the whitening shrinkage and the RDA interpolation on a second dataset.
  • Add a long-horizon split (20 or 50 tasks) where the TP deficit is largest.
  • Report GPU-hours and peak memory for the 1,960 runs.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 61Yn (1/4)

CLASP-G denotes the training-free configuration: Gaussian within-task prediction plus the proposed task posterior. All new numbers are means over five seeds; base classifiers in the plug-in tables were re-run to store per-sample scores.

W1. The Gaussian-classifier-only row recovers most of the gain

Thank you for this precise observation. Extending that row to all 14 settings shows that it is not a counterexample to the proposed method but its simplest configuration.

The prediction of the Gaussian classifier, arg⁡max⁡cLc\arg\max _ c L _ c, decomposes as Lc=(Lc−LSEc′∈Yτ(c)Lc′)+LSEc′∈Yτ(c)Lc′L _ c=\bigl(L _ c-\mathrm{LSE} _ {c'\in\mathcal{Y} _ {\tau(c)}}L _ {c'}\bigr)+\mathrm{LSE} _ {c'\in\mathcal{Y} _ {\tau(c)}}L _ {c'}, and the second term equals the task posterior of Eq. 5 up to a constant. The Gaussian-only row is therefore "Gaussian WP + the proposed posterior", and the difference between the two rows is a difference in which classifier supplies WP. Adding the posterior to five base classifiers gives the following (Final, %, 14 settings × 5 seeds).

Base classifierBase+ proposed posteriorΔ\DeltaFinalΔ\DeltaWP
Prompt, masked CE (control of Table 1)62.5178.25+15.740.00
SimpleCIL72.9277.86+4.940.00
EASE72.9078.12+5.220.00
APER-SSF75.1378.34+3.210.00
Gaussian WP (no training)62.4878.97+16.490.00

Base classifiers that are 12.6 points apart converge to within 1.1 points once the posterior is added, and WP is unchanged in every case. This confirms the conclusion of the audit by an intervention that replaces only the task-level term: the audit and the method are the measurement and the construction of the same statement. The last row further shows that the proposed class-wise low-rank Gaussian alone, without any training, exceeds every baseline of Table 1. The table and this discussion have been added to the design analysis. The contribution is thus not 0.5 points but a task-level term worth +3.2 to +16.5 points that can be added to any classifier with WP preserved.

W2. Missing competitors, test-split selection, missing audit

Both points have been examined and the conclusions hold; details are in Q1, Q2, and Q4.

  • RanPAC has been added under both protocols (14 settings × 5 seeds). The training-free configuration CLASP-G matches or exceeds RanPAC in 7 of the 14 settings, with zero seed variance in all 14 (Q4).
  • Re-selecting the hyperparameters on a holdout of the training split, without any test data, gives a 14-setting average of 78.32–78.35, not lower than the reported 78.25 (Q1). Excluding ImageNet-A 10T, the setting used in the protocol search, leaves the ranking unchanged (Q2).
  • The hyperparameter audit that was missing from the supplement has been added.

Q1. Where the four hyperparameters were selected

ρ=0.05\rho=0.05, α=0.1\alpha=0.1, and r=4r=4 were fixed before the protocol search and were not changed in any of the 1,050 runs; β\beta was chosen among four values on ImageNet-R, and that entire sweep is reported in the design analysis. The missing audit has been added to the supplement with the following content.

In every setting, 10% of the training images of each class were held out, the classifier was retrained on the rest, and all 480 combinations of ρ×α×r×β\rho\times\alpha\times r\times\beta were ranked by holdout accuracy alone.

Selection (no test data)Average test Final, 14 settings
Reported setting (0.05, 0.1, 4, 0.3)78.25
Selected on the holdout of ImageNet-R 10T only → (0.05, 0.1, 8, 0.3)78.35
Leave-one-dataset-out78.32
Selected per setting78.56

The suggested sweep on a second dataset was also run. On CUB-200 20T and ImageNet-A 20T, Final varies by at most 0.4 and 0.8 points over ρ∈[0.001,0.5]\rho\in[0.001,0.5] and α∈[0,1]\alpha\in[0,1], and in all six settings examined every β\beta larger than 0.3 performs equally or better. These results show that the reported numbers are not fitted to the test split; the reported setting is conservative, and selection without any test data yields the same or higher accuracy.

AuthorsOpenReview ↗

Response to Reviewer 61Yn (2/4)

Q2. Whether the ImageNet-A test split is among the reported runs

It is. However, that search was not used in a way that raises the reported numbers.

The best of the 72 configurations was Adam with learning rate 3×10−33\times10^{-3} (62.40), whereas the adopted protocol, SGD with learning rate 0.01 and batch size 64, ranks 23rd (60.19). It was adopted because it lies inside the range of the SGD-based baselines and coincides with the original settings of APER-SSF and PEARL (Appendix C). On other settings as well, the leading configurations of the search are 2.4–5.6 points more favorable to the proposed method than the adopted protocol (CIFAR-100 5T, ImageNet-R 10T, ImageNet-A 40T). The common protocol is therefore a choice aligned with the baselines, and it is the less favorable one for the proposed method.

With ImageNet-A 10T excluded, the 13-setting average is 79.61 for the proposed method against 76.43 for the runner-up APER-SSF, and the proposed method ranks first in 11 of the 13 settings. The use of that split does not affect the conclusions.

Q3. Adding the task posterior to APER-SSF, SimpleCIL, and EASE

The requested experiment supports the claim of the paper most directly. The three methods were run under the common protocol with per-sample scores stored, and the proposed posterior (r=4r=4, fixed β=0.3\beta=0.3) was added without retraining (Final, averaged over the splits of each dataset).

CIFAR-100ImageNet-RImageNet-ACUB-200VTABAll 14
APER-SSF83.8769.9957.9482.5084.4375.13
+ posterior85.6771.3360.9686.9191.0778.34
SimpleCIL80.4864.4858.5481.9882.2472.92
+ posterior84.7970.1461.1586.8690.6477.86
EASE83.6864.8256.5180.4682.0972.90
+ posterior85.8770.2761.1386.8690.6378.12

Final increases in 41 of the 42 method–setting pairs, Δ\DeltaWP is 0.00 in all 42, and the gain grows with the number of tasks on every dataset. In the remaining pair (APER-SSF, ImageNet-R 10T), the TP of the base is already at the level reached after adding the posterior. As the reviewer anticipated, the manuscript undersold the method: the posterior is not limited to prompt-based classifiers and lifts the strongest prototype-based methods by 3.2–5.2 points with WP unchanged. This table has been added to the plug-in results.

AuthorsOpenReview ↗

Response to Reviewer 61Yn (3/4)

Q4. RanPAC, FeCAM, and SLCA

RanPAC has been added under both the common protocol and its original settings (14 settings × 5 seeds). Final accuracy under the common protocol:

SettingRanPACCLASP (Table 1)CLASP-G (no training)
CIFAR-100 5T90.41±0.1185.97±0.1985.96±0.00
CIFAR-100 10T89.05±0.3385.90±0.0985.96±0.00
CIFAR-100 20T87.54±0.0685.63±0.2185.96±0.00
ImageNet-R 10T72.99±0.2872.60±0.1271.65±0.00
ImageNet-R 20T72.62±0.2472.14±0.4871.65±0.00
ImageNet-R 40T72.63±0.1271.64±0.5371.65±0.00
ImageNet-A 10T61.30±0.4960.56±0.6561.40±0.00
ImageNet-A 20T61.40±0.4160.28±0.9861.40±0.00
ImageNet-A 40T47.89±26.4460.23±0.4961.40±0.00
CUB-200 10T87.86±0.1687.02±0.1887.57±0.00
CUB-200 20T87.35±0.4787.10±0.1387.57±0.00
CUB-200 40T87.54±0.4587.05±0.1987.57±0.00
VTAB 5T91.89±0.2990.57±0.5692.92±0.00
VTAB 10T76.17±35.1088.81±1.0692.92±0.00
Average77.6278.2578.97

The table shows three things.

First, the proposed posterior reaches the level of RanPAC without training. CLASP-G matches or exceeds RanPAC in 7 of the 14 settings, and in the remaining settings of ImageNet-R and CUB-200 the difference is at most 1.4 points; RanPAC is clearly ahead on CIFAR-100. Second, stability. The accuracy of CLASP-G does not depend on the class order or the task split, so its seed variance is zero, and the standard deviation of CLASP is at most 1.06 in every setting. RanPAC re-solves a ridge system over 10410^4-dimensional random features and re-selects its coefficient at every task; 4 of its 140 runs end below 20% (ImageNet-A 40T and VTAB 10T above), which is why it trails in the 14-setting average. Third, the result of RanPAC agrees with the analysis of the paper. Its classifier is not trained with the masked cross-entropy but computed from second-order statistics of all classes, so Proposition 1 does not apply to it, and 92–95% of its errors are still TP failures. RanPAC is thus independent evidence for the claim that TP can be supplied outside the training objective, in closed form, from second-order statistics; the proposed method does so without backbone adaptation, random projection, or ridge selection, and without replacing the existing classifier (Q3).

Relation to published numbers: all methods in the paper use ViT-B/16 pre-trained on ImageNet-21k and fine-tuned on ImageNet-1k (timm augreg2_in21k_ft_in1k); the description of the backbone in the manuscript now states the fine-tuning explicitly. With this checkpoint and the original settings of RanPAC, RanPAC attains 79.52 on ImageNet-R 10T and 89.47 on CIFAR-100 10T, and the full original-settings table is in the supplement. FeCAM classifies frozen features with class-wise Gaussians and thus corresponds to the WP part of CLASP-G above; the 14-setting runs of FeCAM and SLCA will be included in the revision.

The RanPAC row has been added to Table 1 and reported per setting in a new table, and the ranking statement now specifies that it refers to the 13 baselines of Table 1.

Q5. What the trained prompt classifier adds; WP of the Gaussian-only variant

The proposed method does not depend on a particular classifier, and this is a property of the method. The two sources of WP compare as follows (WP, averaged over the splits of each dataset).

Source of WPCIFAR-100ImageNet-RImageNet-ACUB-200VTABAll 14
Trained prompt head96.4189.8686.4697.5496.6893.15
Gaussian96.6789.5985.6098.3098.5993.40

The trained head is higher on ImageNet-R and ImageNet-A, and the Gaussian is higher on CUB-200 and VTAB, where the control is weakest in final accuracy; all differences are below 2 points. WP is already high for every classifier, and final accuracy is decided by the task-level term (table in W1). The control of Table 1 serves as the base of a controlled test, in which only the task-level term changes on a classifier trained with the masked cross-entropy to which Proposition 1 applies; the same posterior raises all five bases to 77.9–79.0.

AuthorsOpenReview ↗

Response to Reviewer 61Yn (4/4)

Q6. SimpleCIL versus published values; PEARL under the two protocols

Neither point affects the fairness of the comparison.

SimpleCIL involves no training, so its accuracy is determined by the backbone and the split. Every run in the paper uses ViT-B/16 pre-trained on ImageNet-21k and fine-tuned on ImageNet-1k (verified in all 1,050 logs and by file hash), whereas the main table of the APER paper is based on ViT-B/16-IN21K, hence the different values. All 15 methods load the same file, so the comparison among methods is made under identical conditions, and the description of the backbone has been corrected.

The difference for PEARL lies entirely in TP. Under its original settings, WP is 93.80, at the level of the common protocol (93.54), while TP falls from 64.81 to 44.92 with a forgetting of 42.30. This pattern is consistent with a task-level component biased toward recent tasks by the longer original schedule, and it is exactly the failure type that the audit describes: WP is retained while TP collapses. Table 1 reports PEARL under the common protocol, which is 20 points more favorable to it.

Q7. Corollary 1 under Adam

The recursion of Corollary 1 is a statement about SGD and does not hold under Adam; this scope is stated in the last paragraph of Section 3. The claims of the paper rest not on Corollary 1 but on Proposition 1(i)–(ii), which is independent of the optimizer: the gradient of the masked cross-entropy has no component in VT\mathcal{V} _ {\mathrm T} under any optimizer, and in a numerical check its projection is at most 1.2×10−161.2\times10^{-16}. When the task sums move under Adam, the movement originates from coordinate-wise normalization and not from the objective, and therefore carries no information about the task. The posterior has no gradient path and does not depend on the optimizer; it yields the same gain on five bases with different training procedures (table in W1). The task-sum trajectories of the control trained with Adam will be added to the task-sum figure in the supplement.

Q8. Paired tests and the standard error of the thinnest first place

The requested tests were carried out and support the conclusions. In seed-paired tests with the 14 settings as blocks, the proposed method is significantly higher than each of the 13 baselines (p≤0.0084p\le0.0084), and against the strongest baseline chosen separately in every setting the mean difference is +2.61 (SE 0.57, p=0.0005p=0.0005). Per-setting results (difference = proposed − strongest baseline of that setting, five seeds):

SettingStrongest baselineDiff.SEpp
CIFAR-100 5TAPT−0.540.240.090
CIFAR-100 10TAPT+1.090.120.0008
CIFAR-100 20TAPER-SSF+3.090.380.0012
ImageNet-R 10TAPT−0.800.250.031
ImageNet-R 20TAPT+0.510.430.300
ImageNet-R 40TAPT+4.020.270.0001
ImageNet-A 10TAPT+1.630.390.014
ImageNet-A 20TEASE+1.690.540.036
ImageNet-A 40TAPER-Adapter+1.690.220.0015
CUB-200 10TAPER-SSF+4.440.14below 0.0001
CUB-200 20TAPER-SSF+4.550.18below 0.0001
CUB-200 40TAPER-VPT+4.650.330.0001
VTAB 5TAPER-SSF+6.080.440.0002
VTAB 10TAPER-SSF+4.410.630.0023

The proposed method is significantly higher in 11 settings. The thinnest case, ImageNet-R 20T, has a standard error of 0.43 and is a tie at five seeds; the text has been adjusted accordingly. The gain is larger and more significant in settings with more tasks, which is the trend claimed in the paper.

Suggestions for future work

All three suggestions have been adopted. The sweep of ρ\rho and α\alpha on a second dataset is reported in Q1 and added to the supplement. The computational cost, as the sum of wall-clock time over the runs of the 14 methods of Table 1 under both protocols, is 1,515.5 hours (710.5 under the common protocol and 805.0 under the original settings), and a per-method table has been added. Regarding long horizons, the gain is largest in the three 40-task settings (+9.79, +16.60, and +35.16 points).

Reviewer_fm9yOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies class-incremental learning (CIL) with pretrained models (e.g., pretrained ViTs). In this setting, a model is trained on a sequence of tasks, where prediction correctness can be decomposed into two components: (1) whether the predicted class belongs to the true task (TP), and (2) whether the prediction restricted to the classes of the true task is correct (WP). Since pretrained models are used, the encoder is frozen, while only lightweight modules, such as prompts, adapters, or experts, are learned, typically with a selection rule or a module pool.

Under this setting, the authors find that (1) existing methods using cross-entropy loss restrict optimization to the classes of the current task, while the paper shows that the objective is invariant to logit shifts and that its gradient has no component in the task-level subspace of the logits; and (2) this affects TP but leaves WP unchanged. Based on these findings, the authors propose a method to improve TP through orthogonal selection and introduce a task posterior based on class-wise low-rank Gaussian distributions, without requiring exemplars, gradients, or additional training.

The paper contains substantial mathematical formulations and presents extensive experiments to demonstrate the effectiveness of the proposed method.

Critical strengths

(1) This paper contains extensive mathematical formulations that, in some sense, show the theoretical contribution of this problem. (2) The author conducts 1,050 experiments, which is an extensive set of experiments.

Critical weaknesses

Limitations of my evaluation

I have already informed the AC and SACs that I have difficulty fully evaluating this paper. I have not actively worked in this research area for several years, and my background is not primarily theoretical. The paper contains extensive mathematical formulations that I am unable to confidently assess for correctness.

Although I understand the general motivation and direction of the paper, I have difficulty following its technical details. Therefore, my confidence in evaluating the paper's contributions is limited. I would encourage the AC and SACs to place less weight on my evaluation and rely more heavily on reviewers with relevant expertise.

Weaknesses

(1) Clarity and organization

I found the paper difficult to follow, particularly in its presentation of motivation and technical details. I believe the paper could benefit from better organization and more accessible explanations, especially in the introduction, to help readers who are less familiar with this specific class-incremental learning setting.

For example, in line 055 of the introduction, masked cross-entropy is introduced without sufficient background or motivation. Readers who are not familiar with this particular setting may struggle to understand why masked cross-entropy is needed and how it relates to the problem being addressed.

Similar issues arise in other parts of the paper; e.g., moment accumulation and whitening are directly in their proposed method without much explanation of the intuition or why it would be beneficial. Additional intuition or background explanations could help readers better understand the motivation behind the theoretical analysis and the proposed method.

Questions for authors

This paper contains lots of mathematical formulations with the proposed method, but I do have difficulty intuitively understanding the idea:

(1) Could you intuitively explain why orthogonal selection for TP? (2) This paper just gives the method of using moment accumulation as well as whitening without explanation. Beyond the procedure itself, what's the intuition behind such techniques for this class-incremental learning task?

1 comment
AuthorsOpenReview ↗

Response to Reviewer fm9y

W1. Clarity and organization

The presentation has been revised accordingly. The introduction now explains why the masked cross-entropy is standard: without data from earlier tasks, a softmax over all classes only lowers the logits of old classes and biases the prediction toward the most recent task, so rehearsal-free methods restrict the loss to the classes of the current task. Each step of Section 4.2 now carries one sentence of intuition, as given in Q2 below.

Q1. Intuition of orthogonal selection

Classifiers trained separately on each task are like exams graded on different scales. The ranking within one exam (WP) is reliable, but scores from different exams have never been compared. The proposed method estimates, independently of training, to which exam an input belongs and adds this estimate as a bonus shared by all classes of that exam. Such a bonus cannot change the ranking within an exam (Proposition 2), and it enters only the component that the training gradient never reaches (Proposition 1). It is orthogonal in these two senses. In practice, adding it to five different classifiers leaves WP unchanged and raises final accuracy by 3.2–16.5 points.

Q2. Intuition of moment accumulation and whitening

Without storing samples of earlier tasks, what can be kept is the count, the mean, and the covariance of each class, so they are accumulated once when the class appears. Pre-trained features contain high-variance directions shared by all classes, which dominate the distance to a class mean. Whitening removes this common structure so that the distance measures class-specific deviations. The whitening is shared by all classes, and each class corrects only the four directions that deviate most from the common structure. The effect of each step, as average Final over the 14 settings, is: control 62.51 → class means only 69.77 → shared whitening 77.33 → rank-4 correction 78.25.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.