- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies classifier initialization and knowledge distillation in exemplar-free class-incremental learning. It argues that independently initialized new classifier weights may interfere with old ones, and that a fixed distillation temperature gives inconsistent supervision across incremental stages. The proposed method combines Orthogonal Classifier Initialization, which builds new-class directions from prototypes projected orthogonally to the old-classifier span, and Adaptive-Temperature Knowledge Distillation, which adjusts the temperature based on the teacher’s response concentration.
Critical strengths
None
Critical weaknesses
-
The abstract and introduction state that orthogonality is “released during subsequent optimization,” allowing the classifier to adapt freely. However, Section 3.3 states that all classifier directions remain fixed during task t and only f_t is optimized, and Table 5 further shows that fixed directions outperform trainable ones. The authors should reconcile this. In addition, the classifier reconstruction at the task boundary is not specified: how are old-class statistics updated to account for feature drift without exemplars?
-
The authors need to clarify their novelty regarding the fixed geometric classifier, since the idea is very close to the fixed ETF classifier in [1].
[1] Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Torr, and Dacheng Tao. Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learning. In ICLR, 2023.
Questions for authors
Minor
- CUB-200 results are referenced as Table 1 but appear in Table 2.
- In Section 4.3, the reported gains of OCI over random initialization (0.57/2.11/1.82) do not match Table 5 (1.77/2.78/2.65).
Questions:
-
The conclusion states that ATKD outperforms fixed and class-count-based temperature rules, but no such comparison is reported. Could the authors provide it?
-
Could the authors report standard deviations across class orders?
-
On ImageNet-1K, the number of classes exceeds the feature dimension, so the old-classifier span covers the whole feature space. How is OCI computed in this regime?
3 comments
I thank the authors for the detailed revision and responses.
My concern about the limitation of OCI when the number of classes exceeds the feature dimension of the backbone remains. I suggest that this limitation be stated more simply and explicitly in Section 3.3. In addition, the paper lacks a formal definition of . Moreover, the two ablations show only marginal gains over much simpler alternatives: OCI improves over random orthonormal initialization by only 0.52, and ATKD improves over the class-count temperature by only 0.61. For these reasons, I have decided to keep my original score.
Response-2 to Reviewer Uu6X
Thank you for the follow-up. We address the capacity limitation, the loss definition, and the experimental evidence below.
1. OCI's capacity limitation
The core objective of OCI is to learn well-separated new-class representations while limiting interference with old-class decisions. Orthogonality serves as a tool for constructing fixed learning targets from incoming-class prototypes and retained classifier knowledge.
On ImageNet-1K, IACE and the comparison methods use the same ResNet-18 backbone with 512-dimensional features. With 100 incoming classes per task, our retained reduced-SVD basis leaves only 12 complementary dimensions at 500 old classes and none from 600 onward. Exact old–new orthogonality is no longer guaranteed when the remaining complementary space is insufficient for the incoming classes.
The SVD construction yields mutually orthonormal new-class targets in exact arithmetic, provided their number does not exceed the feature dimension.
2. Explicit definition of the ATKD loss
The implemented ATKD loss is soft-target cross-entropy over the old classes. For task , let and denote the frozen teacher's and student's cosine scores over . Define
q_t^{\mathrm S}(x)=\operatorname{softmax}(s_t^{\mathrm S}(x)/\tau_t).$$ $$\mathcal L_{\mathrm{ATKD}} =-\mathbb E_{(x,y)\sim\mathcal D^t} \sum_{c\in\mathcal C_{1:t-1}}q_{t,c}^{\mathrm T}(x)\log q_{t,c}^{\mathrm S}(x).$$ Here $\mathcal D^t$ is the current-task training set, and $\sigma_{t-1}$ is the common classifier scale assigned after task $t-1$. The temperature $\tau_t=2/\sigma_{t-1}$ stays fixed during task $t$ and is updated at task boundaries. The objective is new-class cross-entropy plus $10\mathcal L_{\mathrm{ATKD}}$, without a temperature-squared multiplier. ## 3. Component contributions and simpler alternatives **Removing either OCI or ATKD lowers mean final accuracy under both the 10- and 20-task settings, while using both achieves the highest accuracy (Table 4).** | Comparison | CIFAR-100 10T | CIFAR-100 20T | |---|---:|---:| | Remove OCI from IACE | -1.58 | -2.45 | | Remove ATKD from IACE | -0.85 | -1.71 | | OCI + ATKD versus the baseline | +4.24 | +4.66 | Changes in mean final accuracy (percentage points), averaged over five class orders. R1 and R2 (Tables 6 and 7) compare OCI's target construction and ATKD's temperature rule with simpler alternatives, with settings matched within each CIFAR-100 10T comparison. R1 compares OCI with random orthonormal targets while retaining task-wise freezing and reconstruction. Using incoming-class prototypes and the retained classifier to construct targets gives OCI 0.52 points higher mean final accuracy; random orthonormal initialization has lower mean forgetting. R2 keeps the rest of IACE fixed and tests the temperature rule. Compared with class-count scaling, ATKD improves mean final and average incremental accuracy by 0.61 and 0.91 points, respectively, and reduces mean forgetting by 0.98 points.
Response to Reviewer Uu6X
Thank you for your careful review and specific suggestions. References below refer to the revised manuscript.
W1. Classifier Freezing and Reconstruction
Classifier directions remain fixed during incremental training; task-end reconstruction updates them from transported class statistics. Please refer to our global response, Q2 and Q3.
W2. Distinction from Fixed ETF
Fixed ETF prescribes a global feature-classifier geometry for base and future classes. OCI uses retained classifier knowledge and current prototypes to construct task-dependent targets that guide new-class learning. These targets stay fixed within each task, while task-end reconstruction updates the classifier without enforcing its initial orthogonality. Please refer to our global response, Q1(1).
Q1. Fixed and Class-count Comparisons
Table R2 compares ATKD with fixed boundary scales 16, 32 and 48 and a class-count strategy under five matched runs. ATKD achieves the highest mean final and average incremental accuracy and the lowest mean forgetting among these tested strategies. Please refer to our global response, Q1(2) and Table R2.
Q2. Variation across Class Orders
Table R4 reports means ± sample SDs from five runs per setting, varying both class order and training seed. Alast/Ainc denote final/average incremental accuracy; means match Tables 1–3.
Table R4. IACE main results
CUB-200 10T/20T use ImageNet pretraining; other settings use random backbone initialization. Sample SDs range from 0.31–0.85 points for Alast and 0.38–1.22 for Ainc.
Q3. OCI Capacity on ImageNet-1K
ResNet-18 with 512-dimensional features is the common backbone setting for all methods in our main comparison. Finite feature dimension also constrains classifiers built from learned weights, class prototypes or fixed directions: their classifier or prototype matrices have rank at most 512. OCI still computes the polar factor from the projected prototypes' reduced SVD when the residual is rank-deficient or zero (Eqs. (8)–(9)).
The implementation retains zero-singular-value components, so SVD completion supplies nonzero directions. At zero residual, their orientation depends on nonunique SVD completion or numerical residuals. For 100 incoming classes in 512 dimensions, new directions can remain mutually orthonormal; exact old–new orthogonality is not guaranteed when the projected prototypes lack full row rank (Sec. 3.3).
Under the ten-task ImageNet-1K protocol, the full IACE pipeline achieves final accuracy of 36.59±0.53% and average incremental accuracy of 55.64±0.79% across five runs, exceeding the reported DPCR results by 1.10 and 1.42 points, respectively (Table 3). These results describe the full pipeline under the shared backbone setting; they do not isolate late-stage OCI's contribution.
Minor Corrections
The CUB-200 paragraph now refers to Table 2. Sec. 4.3 reports gains of 1.77, 2.78 and 2.65 points over ordinary random initialization for CIFAR-100, CUB-200 and TinyImageNet, respectively, matching Table 5.