acceptodds
Under review as a conference paper at ICLR 2027

Reducing Task-Transition Interference for Exemplar-Free Class-Incremental Learning

Abstract

Exemplar-free class-incremental learning (EFCIL) requires models to learn new classes while preserving the decision boundaries of previously learned classes. Many EFCIL methods retain compact statistics that summarize previously learned classes. As new tasks arrive, these statistics are primarily used to preserve or correct existing class structure. Yet this preservation-oriented use leaves open a key question: how should retained knowledge guide adaptation to new classes while preserving the old-class decision structure? Without such guidance, new-class updates may overlap with old-class discriminative directions and cause interference. To answer this question, we propose AnchorScale, which turns retained knowledge into an explicit reference for new-class adaptation. Specifically, Fixed Anchoring extracts current-task prototypes with the frozen previous encoder and projects each prototype onto the orthogonal complement of the subspace spanned by retained old-class classifier directions. The resulting directions remain fixed throughout current-task training while the feature extractor is optimized with classification and distillation objectives. At task boundaries, Probability Feedback calibrates the logit scale for the next task based on the teacher’s probability mass over likely old classes, maintaining effective cross-task supervision as the label space expands. Experiments on standard, fine-grained, and large-scale EFCIL benchmarks show that AnchorScale performs competitively with existing methods. Ablation studies confirm the importance of fixed prototype-guided directions and the complementary contribution of Probability Feedback.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, semantic drift, catastrophic forgetting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Uu6XOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies classifier initialization and knowledge distillation in exemplar-free class-incremental learning. It argues that independently initialized new classifier weights may interfere with old ones, and that a fixed distillation temperature gives inconsistent supervision across incremental stages. The proposed method combines Orthogonal Classifier Initialization, which builds new-class directions from prototypes projected orthogonally to the old-classifier span, and Adaptive-Temperature Knowledge Distillation, which adjusts the temperature based on the teacher’s response concentration.

Critical strengths

None

Critical weaknesses
  1. The abstract and introduction state that orthogonality is “released during subsequent optimization,” allowing the classifier to adapt freely. However, Section 3.3 states that all classifier directions remain fixed during task t and only f_t is optimized, and Table 5 further shows that fixed directions outperform trainable ones. The authors should reconcile this. In addition, the classifier reconstruction at the task boundary is not specified: how are old-class statistics updated to account for feature drift without exemplars?

  2. The authors need to clarify their novelty regarding the fixed geometric classifier, since the idea is very close to the fixed ETF classifier in [1].

[1] Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Torr, and Dacheng Tao. Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learning. In ICLR, 2023.

Questions for authors

Minor

  • CUB-200 results are referenced as Table 1 but appear in Table 2.
  • In Section 4.3, the reported gains of OCI over random initialization (0.57/2.11/1.82) do not match Table 5 (1.77/2.78/2.65).

Questions:

  1. The conclusion states that ATKD outperforms fixed and class-count-based temperature rules, but no such comparison is reported. Could the authors provide it?

  2. Could the authors report standard deviations across class orders?

  3. On ImageNet-1K, the number of classes exceeds the feature dimension, so the old-classifier span covers the whole feature space. How is OCI computed in this regime?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Uu6X

Thank you for your careful review and specific suggestions. References below refer to the revised manuscript.

W1. Classifier Freezing and Reconstruction

Classifier directions remain fixed during incremental training; task-end reconstruction updates them from transported class statistics. Please refer to our global response, Q2 and Q3.

W2. Distinction from Fixed ETF

Fixed ETF prescribes a global feature-classifier geometry for base and future classes. OCI uses retained classifier knowledge and current prototypes to construct task-dependent targets that guide new-class learning. These targets stay fixed within each task, while task-end reconstruction updates the classifier without enforcing its initial orthogonality. Please refer to our global response, Q1(1).

Q1. Fixed and Class-count Comparisons

Table R2 compares ATKD with fixed boundary scales 16, 32 and 48 and a class-count strategy under five matched runs. ATKD achieves the highest mean final and average incremental accuracy and the lowest mean forgetting among these tested strategies. Please refer to our global response, Q1(2) and Table R2.

Q2. Variation across Class Orders

Table R4 reports means ± sample SDs from five runs per setting, varying both class order and training seed. Alast/AincA_{\rm last}/A_{\rm inc} denote final/average incremental accuracy; means match Tables 1–3.

Table R4. IACE main results

SettingAlastA_{\rm last} (%)AincA_{\rm inc} (%)
CIFAR-100 10T54.04±0.3165.93±1.22
CIFAR-100 20T43.38±0.4556.76±0.57
TinyImageNet 10T38.64±0.7250.98±0.64
TinyImageNet 20T31.76±0.8545.75±0.77
CUB-200 5T30.82±0.6739.65±0.60
CUB-200 10T58.33±0.4266.11±0.38
CUB-200 20T46.35±0.5556.20±0.50
ImageNet-100 10T54.60±0.6466.87±0.67
ImageNet-100 20T45.95±0.6259.51±0.58
ImageNet-1K 10T36.59±0.5355.64±0.79

CUB-200 10T/20T use ImageNet pretraining; other settings use random backbone initialization. Sample SDs range from 0.31–0.85 points for AlastA_{\rm last} and 0.38–1.22 for AincA_{\rm inc}.

Q3. OCI Capacity on ImageNet-1K

ResNet-18 with 512-dimensional features is the common backbone setting for all methods in our main comparison. Finite feature dimension also constrains classifiers built from learned weights, class prototypes or fixed directions: their classifier or prototype matrices have rank at most 512. OCI still computes the polar factor from the projected prototypes' reduced SVD when the residual is rank-deficient or zero (Eqs. (8)–(9)).

The implementation retains zero-singular-value components, so SVD completion supplies nonzero directions. At zero residual, their orientation depends on nonunique SVD completion or numerical residuals. For 100 incoming classes in 512 dimensions, new directions can remain mutually orthonormal; exact old–new orthogonality is not guaranteed when the projected prototypes lack full row rank (Sec. 3.3).

Under the ten-task ImageNet-1K protocol, the full IACE pipeline achieves final accuracy of 36.59±0.53% and average incremental accuracy of 55.64±0.79% across five runs, exceeding the reported DPCR results by 1.10 and 1.42 points, respectively (Table 3). These results describe the full pipeline under the shared backbone setting; they do not isolate late-stage OCI's contribution.

Minor Corrections

The CUB-200 paragraph now refers to Table 2. Sec. 4.3 reports gains of 1.77, 2.78 and 2.65 points over ordinary random initialization for CIFAR-100, CUB-200 and TinyImageNet, respectively, matching Table 5.

I thank the authors for the detailed revision and responses.

My concern about the limitation of OCI when the number of classes exceeds the feature dimension of the backbone remains. I suggest that this limitation be stated more simply and explicitly in Section 3.3. In addition, the paper lacks a formal definition of LATKD\mathcal{L}_{\text{ATKD}}. Moreover, the two ablations show only marginal gains over much simpler alternatives: OCI improves over random orthonormal initialization by only 0.52, and ATKD improves over the class-count temperature by only 0.61. For these reasons, I have decided to keep my original score.

AuthorsOpenReview ↗

Response-2 to Reviewer Uu6X

Thank you for the follow-up. We address the capacity limitation, the loss definition, and the experimental evidence below.

1. OCI's capacity limitation

The core objective of OCI is to learn well-separated new-class representations while limiting interference with old-class decisions. Orthogonality serves as a tool for constructing fixed learning targets from incoming-class prototypes and retained classifier knowledge.

On ImageNet-1K, IACE and the comparison methods use the same ResNet-18 backbone with 512-dimensional features. With 100 incoming classes per task, our retained reduced-SVD basis leaves only 12 complementary dimensions at 500 old classes and none from 600 onward. Exact old–new orthogonality is no longer guaranteed when the remaining complementary space is insufficient for the incoming classes.

The SVD construction yields mutually orthonormal new-class targets in exact arithmetic, provided their number does not exceed the feature dimension.

2. Explicit definition of the ATKD loss

The implemented ATKD loss is soft-target cross-entropy over the old classes. For task t≥2t\ge2, let stT(x)s_t^{\mathrm T}(x) and stS(x)s_t^{\mathrm S}(x) denote the frozen teacher's and student's cosine scores over C1:t−1\mathcal C_{1:t-1}. Define

q_t^{\mathrm S}(x)=\operatorname{softmax}(s_t^{\mathrm S}(x)/\tau_t).$$ $$\mathcal L_{\mathrm{ATKD}} =-\mathbb E_{(x,y)\sim\mathcal D^t} \sum_{c\in\mathcal C_{1:t-1}}q_{t,c}^{\mathrm T}(x)\log q_{t,c}^{\mathrm S}(x).$$ Here $\mathcal D^t$ is the current-task training set, and $\sigma_{t-1}$ is the common classifier scale assigned after task $t-1$. The temperature $\tau_t=2/\sigma_{t-1}$ stays fixed during task $t$ and is updated at task boundaries. The objective is new-class cross-entropy plus $10\mathcal L_{\mathrm{ATKD}}$, without a temperature-squared multiplier. ## 3. Component contributions and simpler alternatives **Removing either OCI or ATKD lowers mean final accuracy under both the 10- and 20-task settings, while using both achieves the highest accuracy (Table 4).** | Comparison | CIFAR-100 10T | CIFAR-100 20T | |---|---:|---:| | Remove OCI from IACE | -1.58 | -2.45 | | Remove ATKD from IACE | -0.85 | -1.71 | | OCI + ATKD versus the baseline | +4.24 | +4.66 | Changes in mean final accuracy (percentage points), averaged over five class orders. R1 and R2 (Tables 6 and 7) compare OCI's target construction and ATKD's temperature rule with simpler alternatives, with settings matched within each CIFAR-100 10T comparison. R1 compares OCI with random orthonormal targets while retaining task-wise freezing and reconstruction. Using incoming-class prototypes and the retained classifier to construct targets gives OCI 0.52 points higher mean final accuracy; random orthonormal initialization has lower mean forgetting. R2 keeps the rest of IACE fixed and tests the temperature rule. Compared with class-count scaling, ATKD improves mean final and average incremental accuracy by 0.61 and 0.91 points, respectively, and reduces mean forgetting by 0.98 points.
Reviewer_YJ3COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates Exemplar-Free Class-Incremental Learning (EFCIL) under an expanding decision space. The authors argue that newly added classifier weights suffer from initialization interference with existing classes, whereas enforcing persistent geometric constraints restricts subsequent representation adaptation. To resolve this, the authors propose Interference-Aware Classifier Expansion (IACE), consisting of two main components: Orthogonal Classifier Initialization (OCI) and Adaptive-Temperature Knowledge Distillation (ATKD). Experiments are conducted on CIFAR-100, TinyImageNet, CUB-200, ImageNet-100, and ImageNet-1K using a ResNet-18 backbone.

Critical strengths
  1. Identifying the transient geometric interference between new and old classifier weights at the moment of task expansion is a sound and intuitive research question in EFCIL.
  2. The paper is well-organized, and Fig.1 clearly communicates the conceptual distinction between standard initialization and orthogonal initialization.
Critical weaknesses
  1. The technical contributions are quite incremental. Projecting class prototypes onto the null space of a matrix using Gram-Schmidt / thin SVD polar decomposition is standard linear algebra and widely used in subspace-based continual learning. Adapting temperature using softmax entropy or top-KK probability concentration has been extensively explored in distillation and continual learning works.
  2. Incomplete method details. The paper doesn't provides any algorithmic explanation, equations, or pseudocode detailing how this classifier reconstruction is performed. For example, which class statistics are stored or updated (means, covariances, transport plans)? How are old-class classifiers reconstructed without exemplars? Is it done via feature hallucination (like FeTrIL), analytic ridge regression (like ACIL), or Gaussian sampling (like SLCA/DPCR)?
Questions for authors
  1. Please clarify why the Abstract, Introduction, and Contributions claim that orthogonality is "released during subsequent optimization, allowing the classifier to adapt freely," while Section 3.2, Section 3.3, and Table 5 explicitly specify and demonstrate that classifier directions must remain frozen during training to avoid performance collapse.
  2. Sensitivity of ρt\rho_t. What exact value or scheduling rule was used for ρt\rho_t in Eq. (10)? Please provide an ablation study showing performance sensitivity across varying values of ρt\rho_t.
1 comment
AuthorsOpenReview ↗

Response to Reviewer YJ3C

We thank the reviewer for recognizing task-transition geometry as a sound research question and for the constructive suggestions.

W1. Novelty beyond Standard Techniques

OCI extends retained classifier knowledge from preserving old decisions to guiding new-class acquisition: anchors constructed from current prototypes and retained geometry remain fixed while the backbone learns. ATKD derives one shared temperature for the next task from aggregated old-class response concentration and a fixed target.

R1 supports OCI's construction, with mean final accuracy rising from 51.64% to 52.23% to 53.79% as projection and polar are added. In R2, ATKD exceeds class-count scaling by 0.61 final-accuracy points and reduces forgetting by 0.98 points. Please refer to our global response, Q1(1)–(2) and Tables R1–R2.

W2. Classifier Reconstruction

Classifiers are reconstructed by analytic joint ridge regression from transported class statistics. Please refer to our global response, Q3.

Q1. “released during subsequent optimization” and Classifier Freezing During Training

Classifier directions remain fixed during each incremental task while the backbone learns and are updated by task-end reconstruction. Please refer to our global response, Q2.

Q2. Target Concentration and Sensitivity

We use a fixed target concentration ρ=0.7\rho=0.7 across tasks and datasets. Mean final accuracy spans 0.22 points over ρ=0.6\rho=0.6–0.80.8, but is 1.60 points lower at 0.9 than at 0.7. Please refer to our global response, Q4 and Table R3.

Reviewer_b4WYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes IACE to mitigate old–new classifier interference in exemplar-free class-incremental learning. Orthogonal Classifier Initialization (OCI) projects current-class prototypes onto the orthogonal complement of the retained classifier’s row space and takes the polar factor of the projected prototypes to construct incoming-class directions. Adaptive-Temperature Knowledge Distillation (ATKD) adjusts the next incremental task’s effective distillation temperature based on teacher-response concentration on current-task inputs. Experiments cover five datasets and multiple incremental protocols, with CIFAR-100 ablations examining component gains and forgetting.

Critical strengths
  1. The paper addresses old–new classifier interference without access to old training samples. Classifier geometry supports new-class integration, while teacher-signal calibration supports old-knowledge preservation. The design is clearly motivated by the problem.

  2. The experiments provide broad coverage and demonstrate consistent gains from the proposed components. Tables 1–3 cover five datasets and different task partitions, with the proposed method achieving higher final accuracy than all listed baselines in every setting. Table 4 further shows that OCI and ATKD are individually effective, while their combination further improves accuracy and reduces forgetting.

Critical weaknesses
  1. The technical advance of the proposed method is not sufficiently justified. The method includes specific designs for geometric initialization and adaptive-temperature distillation, but their added value over existing ideas is not sufficiently established. The abstract highlights releasing the orthogonality constraint after initialization, whereas Sections 3.2–3.3 specify that classifier directions remain fixed during each incremental task. This inconsistency also makes the central contribution less clear.

  2. The internal mechanisms and hyperparameter sensitivity are not analyzed in sufficient depth. Tables 4–5 evaluate the overall components and the effect of keeping classifier directions fixed, but do not sufficiently disentangle the benefits of prototype guidance, projection onto the orthogonal complement of the retained classifier’s row space, and taking the polar factor. The comparisons with random orthonormal initialization and alternative temperature rules discussed in the paper are also not clearly presented. Although the distillation settings are kept fixed across the remaining datasets, key parameter values and sensitivity analyses remain insufficiently documented, making it difficult to assess whether the gains depend on particular settings.

Questions for authors
  1. To clarify the training procedure and the claimed contribution, please confirm whether classifier directions are kept fixed throughout each incremental task in the main experiments and explain what the abstract means by orthogonality being “released during subsequent optimization.” How are class statistics updated and the classifier reconstructed at the end of each task?

  2. To isolate the contributions of the individual design choices, could you compare direct prototype initialization, projection without the polar-factor step, and full OCI under the same policy for fixing classifier directions? Could you also provide the comparisons with random orthonormal initialization and with fixed and class-count-based temperature rules mentioned in the paper?

  3. Please clarify the hyperparameter settings and their sensitivity by explaining how the distillation weight, target response concentration, initial effective temperature, and bounds on the temperature update are set. Could you also analyze how changes in the key hyperparameters affect performance in representative settings?

2 comments
AuthorsOpenReview ↗

Response to Reviewer b4WY

Thank you for your careful review and constructive suggestions.

Q1. Training Procedure and Reconstruction (W1)

Classifier directions remain fixed during each incremental task while the backbone learns; task-end joint ridge regression reconstructs them from transported class statistics. Please refer to our global response, Q2 and Q3.

Q2. Technical Advance and Controlled Comparisons (W1, W2)

Under matched settings, R1 shows higher mean final accuracy as projection and polar are added to prototype initialization; R2 shows that teacher-response feedback outperforms the tested fixed and class-count scale rules on all three reported means. Please refer to our global response, Q1(1)–(2) and Tables R1–R2.

Q3. Hyperparameter Settings and Sensitivity (W2)

(1) Distillation weight, initial temperature and update bounds.

We retain λ=10\lambda=10 from the DPCR reference configuration across all reported settings (Eq. (4), Sec. 4.1). The base-task scale σ1=16\sigma_1=16 and fixed KD divisor of 2 give the initial effective temperature τ2=2/16=0.125\tau_2=2/16=0.125 (Eq. (10)).

Subsequent temperature updates use the proportional rule without numerical clipping bounds (Eq. (13), Appendix B). The positive-temperature conditions for Eq. (13) held in every reported run (zero violations); classifier scales are not updated by gradients.

(2) Target concentration and sensitivity.

R3 varies the target concentration ρ\rho, while R2 compares boundary-scale strategies; the distillation weight and initial scale remain fixed in these comparisons. Please refer to our global response, Q4 and Table R3, and Q1(2) and Table R2.

Thank you for your detailed response and the additional experiments. My question Q1 was answered very well, and the authors' response to Q2 also largely resolved my concerns. However, I found that the average incremental accuracy of the complete OCI has a relatively small advantage over random orthonormal initialization. Q3 has been partially answered. The newly added target concentration sensitivity experiment provides some robust evidence, but key parameters such as distillation weight still lack independent sensitivity analysis. After carefully considering the authors' full response, the supplementary experiments, and the feedback from other reviewers, I have decided to maintain my current rating of 2: Weak rejection.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response (2/2)

Q1(2). ATKD (continued)

Distinction from Prior Methods

DTKD (Wei and Bai, 2024) sets sample-wise teacher/student temperatures from their logit sharpness; CIST (Luong et al., 2026) uses their maximum logits. ATKD instead uses aggregated old-class probability mass to set one shared temperature for the following task. LwHT (Gao et al., 2024) changes teacher checkpoints to reduce overconfidence. DynLearn (Luo et al., 2025) adjusts feature-distillation weights using task complexity and schedules a separate supervised contrastive temperature.

Temperature strategy comparison

R2 (Table 7) tests whether teacher-response feedback improves on fixed scales or scaling determined solely by class count, under matched training settings.

After reconstruction, all seen classes receive a common cosine-score multiplier σt\sigma_t, setting next-task KD temperature τt+1=2/σt\tau_{t+1}=2/\sigma_t (Eq. (10)). Fixed SS assigns scale SS at each boundary; Class-count sets σt=16log⁡Nt/log⁡N1\sigma_t=16\log N_t/\log N_1, where NtN_t and N1N_1 count seen classes after task tt and the base task. During training, old classes retain their boundary scale; incoming classes use 16.

Table R2. Classifier scales set at task boundaries

StrategyAlast↑A_{\rm last}\uparrowAinc↑A_{\rm inc}\uparrowFlast↓F_{\rm last}\downarrow
Fixed 1646.01±0.4349.26±1.0628.36±0.67
Fixed 3250.76±0.5252.78±1.4921.01±0.89
Fixed 4853.20±0.6764.26±1.3013.13±0.96
Class-count53.43±0.3765.02±1.3413.52±0.43
ATKD54.04±0.3165.93±1.2212.54±0.92

ATKD gains 0.84 mean final-accuracy points over the best tested fixed scale, 48. Against class-count scaling, mean final and average incremental accuracy increase by 0.61 and 0.91 points, respectively, while forgetting decreases by 0.98 points.

Joint evidence

OCI defines the targets for learning incoming classes, while ATKD adjusts the old-class supervision used during the same backbone adaptation. Table 4 shows that each module improves the baseline and their combination gives the largest gains: +4.24/+4.66 final-accuracy points on CIFAR-100 10T/20T.

Q2. “released during subsequent optimization” and Classifier Freezing During Training (b4WY W1/Q1; YJ3C Q1; Uu6X W1)

“Released” refers to task-end reconstruction, which need not preserve OCI's initial orthogonality. Classifier directions remain fixed while the backbone adapts within each incremental task.

Base-task directions are trained jointly with the backbone; task-end reconstruction follows Eq. (6).

We now state this sequence in the Abstract and Introduction, consistent with Secs. 3.2–3.3.

Q3. Class Statistics and Classifier Reconstruction (b4WY Q1; YJ3C W2; Uu6X W1)

We reconstruct classifiers by analytic joint ridge regression from transported statistics, now detailed in Sec. 3.2 and Appendix B.

Following DPCR, we retain counts, raw-feature means, uncentered second-moment sums and support projectors per class. Paired teacher/student features of identical current-task inputs estimate the drift map StS_t without old images (Eq. (17)).

Old means and second moments are mapped then projected onto retained class support spaces; counts stay unchanged and projectors are recomputed (Eq. (5)). Joint ridge regression uses transported old and current new-class statistics; weight rows are normalized, with scales assigned separately (Eq. (6)).

Q4. Target Response Concentration and Sensitivity (b4WY W2/Q3; YJ3C Q2)

The target concentration ρ\rho specifies the desired probability mass assigned to the effective top-ktk_t old classes in the teacher's predictions, where ktk_t denotes the effective number of competing old classes. We selected ρ=0.7\rho=0.7 on CIFAR-100 validation data before test evaluation and kept it fixed across tasks and datasets.

Table R3 (Table 9 in the paper) reports the subsequent sensitivity analysis, varying only ρ\rho using three matched class orders and training seeds (42, 1234 and 3456) for every target value. The ρ=0.7\rho=0.7 setting matches Full OCI in R1, but the results come from independent run batches.

Table R3. Target concentration

ρ\rhoAlast↑A_{\rm last}\uparrowAinc↑A_{\rm inc}\uparrowFlast↓F_{\rm last}\downarrow
0.553.31±0.1266.00±2.8114.12±1.04
0.653.97±0.3966.45±3.1712.90±0.56
0.753.98±0.3466.63±3.0213.53±0.36
0.853.76±1.1765.91±3.4112.44±1.27
0.952.38±1.0665.45±3.2312.74±0.38

IACE's mean final accuracy shows low sensitivity to ρ\rho over the tested range of 0.6–0.8, spanning 0.22 percentage points, with only 0.01 points between 0.6 and 0.7. The 1.60-point drop at 0.9 shows that the target still matters; 0.8 gives the lowest mean forgetting among the tested values.

We thank the reviewers and respectfully invite them to reconsider their assessments.

AuthorsOpenReview ↗

Global Response (1/2)

We thank the Area Chair and reviewers. All references are to the revised paper.

Q1. Novelty of IACE (b4WY W1–W2/Q2; YJ3C W1; Uu6X W2/Q1)

IACE addresses two complementary challenges at task transitions: integrating new classes while mitigating interference with old-class decisions, and maintaining informative old-knowledge supervision as the label space expands.

(1) OCI: Technical Advance and Distinction from Fixed ETF

Technical Advance.

Existing methods based on independent initialization or predefined geometry do not adapt new-class learning targets to both incoming-class semantics and the learned old-class structure. New-class directions guide backbone updates that can affect old-class recognition, while old examples are unavailable for direct correction.

OCI extends the role of retained classifier knowledge from old-class preservation to actively guiding new-class learning. OCI constructs task-dependent semantic anchors by projecting incoming-class prototypes away from the retained classifier span and applying polar decomposition. The OCI classifier directions remain fixed throughout incremental training, providing stable learning targets while the backbone adapts.

Distinction from Fixed ETF.

NC-FSCIL (Yang et al., 2023) imposes a predefined global ETF geometry for base and future classes. In contrast, OCI constructs task-dependent classifier directions from current-class semantics and retained classifier geometry, without prescribing a persistent global structure. Task-end reconstruction subsequently updates these directions without enforcing their initial orthogonality. Moreover, unlike gradient-projection methods such as GPM (Saha et al., 2021), which constrain parameter updates, OCI directly shapes the learning targets that guide representation adaptation.

Initialization strategy comparison

R1 (Table 6) adds projection and polar sequentially to prototypes, with a random orthonormal control. Variants share the base-task checkpoint, data streams, freezing policy and other training settings.

Table R1. OCI decomposition

InitializationPrototypeProjectionPrototype polarAlast↑A_{\rm last}\uparrowAinc↑A_{\rm inc}\uparrowFlast↓F_{\rm last}\downarrow
Direct prototypes✓——51.64±0.2264.40±2.1215.24±1.25
Projection without polar✓✓—52.23±0.5764.63±2.2514.84±1.49
Full OCI✓✓✓53.79±0.4066.07±1.8413.59±0.81
Random orthonormal———53.27±0.3466.03±1.5812.84±1.08

Standard settings (R1–R3): CIFAR-100 10T, ResNet-18 from scratch; mean ± sample SD over three runs for R1/R3 and five for R2, with paired class orders and training seeds within each comparison.

Alast/AincA_{\rm last}/A_{\rm inc}: final/average incremental accuracy (%); FlastF_{\rm last}: final forgetting from task-end evaluations (percentage points), with peaks computed per seed and task before averaging. The random control uses the polar factor of a Gaussian matrix without projection, not the prototype polar step.

Projection and polar progressively improve mean final accuracy (R1). Keeping OCI directions fixed also outperforms training them on all three tested datasets (Table 5).

Compared with random orthonormal initialization, OCI has 0.52 points higher final accuracy despite greater forgetting. On the first nine tasks, its mean peaks (67.53% vs. 66.39%) and final accuracy (53.95% vs. 53.54%) are higher, but so is its decline (13.59 vs. 12.84 points). Forgetting measures this decline; overall final accuracy covers all ten tasks.

(2) ATKD: Technical Advance

Technical Advance

Existing sample-wise adaptive-temperature methods adjust the softness of individual predictions. EFCIL adds a task-level requirement: maintaining informative old-class supervision as the teacher and old-class output space change. Without old examples, distillation relies on teacher responses: overly sharp predictions suppress alternative old-class relationships, while overly flat predictions weaken class distinctions. Less informative old-class supervision can therefore increase forgetting during new-class learning.

ATKD compares the mean probability mass assigned to the effective top-ktk_t old classes on current-task inputs with a fixed target. Concentration above the target raises the next task's temperature; concentration below the target lowers it. Eq. (13) determines the proportional correction under a two-level response approximation. Teacher and student share the updated temperature, fixed throughout the next task.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.