- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes a logit adjustment strategy to help select the task ID for exemplar-free class-incremental learning. Experiments on multiple datasets supports the efficacy of the proposed strategy. While theoretical analysis is provided trying to justify the proposed strategy, possibly serious issues exist in the theoretical analysis.
Critical strengths
The main contribution lies in the logit adjustment strategy. Specifically, class-wise Gaussian distributions are used to estimate the probability of any test image belong to each task, and such task-level probablity is used to adjust the logit of each task's classifier head for task ID determination.
Critical weaknesses
- Crucial justification of the proposed method is missing. Particularly, why are the mixture of class-wise Gaussian distributions used to estimate the probability of the test input belonging to each task, and why is such estimate resonable? Why is the adjustment of logit helpful for task ID selection? For a test image which is visually similar to two classes from two different tasks, how can such adjustment help select the correct task?
- The theoretical analysis is questionable. First, some key variables and concepts are introduced without clear definition and elaboration. For example, what does the C-dim vector l_j look like, and how does it represent the classes of task j? Isn't the span of l_1, l_2, ..., l_t the whole R^C space? The descriptions of the span (Lines 151-155) are unclear, with new notations and conclusions introduced but without explanation. Second, proofs of propositions and lemma lack enough elaborations and might be partly erroneous. For example, for the proof of Proposition 1, why does a vector v take a constant value vu for all logit elements? where is v from, and why is it a constant? For the proof of Proposition 2, what is g(x) and where is it from? why are elements of all g(x) identical? If g(x) is funciton of the test input, it is morel likely a vector containing various element values. For Lemma 2, pi(x) appears and is not used afterward. For Lemma 3, what is Eq (4) used for?
- The empirical evaluations are not sufficient and one key setting is questionable. Considering that the ViT backbone is pre-trained on the natural image dataset ImageNet-21k, continual learning tasks should include image dataset from other domains (e.g., medical image dataset) to confirm the generalization of the proposed method. For the hyper-parameter setting, usign 5 epoches for task learning is not a widely adopted protocol.
Questions for authors
- Why is the adjustment (Eq. 5) reasonable and effective?
- Please provide clear and detailed proofs, with every notation explicitly elaborated.
- Please provide additional empirical evaluation to suppor the generalizaiton of the method.
2 comments
Response to Reviewer ahEt (2/2)
W3. Other domains and the 5-epoch protocol
Both points are covered by the experiments, and the results show that the proposed method is most effective on domains far from the pre-training data.
VTAB combines five datasets from different domains, and CUB-200 is a fine-grained benchmark. The gain in final accuracy over the control, ordered by domain, is:
| ImageNet-R | CIFAR-100 | ImageNet-A | CUB-200 | VTAB |
|---|---|---|---|---|
| +6.77 | +6.81 | +11.63 | +26.49 | +32.60 |
When the posterior is added to other classifiers, the gain is again largest on VTAB (APER-SSF +6.64, SimpleCIL +8.39, EASE +8.54). Since the posterior uses only statistics of frozen features, its role grows as the domain gap makes the training of the classifier harder.
Regarding the 5-epoch protocol, the 13 baselines were re-run under the settings of their own papers (up to 50 epochs), 910 runs reported in the supplement, and the ranking of Table 1 is preserved under those settings. The posterior involves no training, so the number of epochs affects only the base classifier.
Q1–Q3
Q1 is answered in W1, Q2 in W2, and Q3 in W3.
Response to Reviewer ahEt (1/2)
W1. Justification of the method
The three questions are answered in order, and a direct measurement of the case raised by the reviewer, visually similar classes that belong to different tasks, has been added.
(i) Why a mixture of class-wise Gaussians. A task is a union of distinct classes and is multimodal in feature space. One Gaussian per task gives a TP gain of +4.61 points, whereas the class mixture gives +7.49. The effect of each component, as average Final over the 14 settings, is:
(ii) Why adjusting the logits helps task selection. By Lemma 2, the predicted task is argmaxj[mj(x)+βγj(x)]. The offsets among the block maxima mj are not determined by training (Proposition 1), so the task posterior decides this comparison.
(iii) Similar classes from two tasks. The head logits of two such classes are never trained against each other, so their difference is arbitrary, whereas the Gaussian likelihoods are computed in a whitened space shared by all classes and are directly comparable. Classes were ranked by the similarity to their nearest class in another task, and the test samples of the top 10% were compared with the rest.
The case raised by the reviewer is precisely where the posterior is most effective: on samples with a similar class in another task the TP gain reaches 57 points, and 27–81% of the predictions that fall into a wrong task are corrected. This analysis has been added to the manuscript.
W2. Theoretical analysis
All propositions and proofs hold, and the points raised are resolved by making the notation explicit.
The propositions were also verified numerically (float64, 104 random logit vectors).
In actual training, the relative deviation between the recursion of Corollary 1 and the measured task sums is at most 3.7×10−4, and ΔWP is 0.00 in all 56 method–setting pairs to which the posterior was added. A notation table, the two-task example above, and proofs expanded line by line have been added. The theory is exact and agrees with the experiments to machine precision; a reconsideration of the soundness assessment is respectfully requested.