- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents LAFA, a framework for post-deployment learning that integrates out-of-distribution (OOD) detection with online class-incremental learning. The proposed approach employs a frozen pre-trained model for feature extraction, followed by random projection and Angular Linear Discriminant Analysis (aLDA) to update class statistics incrementally without retaining previous training data. It also incorporates test-time magnitude augmentation, using unnormalized features for inference while constructing class statistics from normalized features. Experiments on eight datasets with three pre-trained backbones report consistent improvements over existing continual learning baselines, particularly PLDA.
Critical strengths
-
The paper considers the integration of OOD detection and incremental classification under a streaming data setting, where new classes may emerge after the initial training stage.
-
The experimental evaluation covers multiple datasets and pre-trained backbones. The paper also provides ablation studies examining the effects of normalization, random projection, and test-time magnitude augmentation.
Critical weaknesses
-
Limited methodological novelty. The proposed method primarily combines existing techniques, including feature normalization, random projection, and incremental LDA. In particular, Angular LDA largely amounts to normalizing class mean directions, while test-time magnitude augmentation modifies the inference rule without introducing a new learning objective or optimization procedure. Angular representations and normalized classifiers have already been explored in continual learning. Overall, the proposed method appears closer to an empirical refinement of existing approaches than a substantial methodological contribution.
-
Unclear distinction from existing continual learning settings and incomplete framework. The proposed learning-after-deployment setting substantially overlaps with existing open-world and online class-incremental learning frameworks. Although label acquisition is presented as an essential component, the paper does not implement it and instead assumes that ground-truth labels are available for detected OOD samples. Consequently, the proposed system essentially reduces to OOD detection combined with supervised online class-incremental learning. This also weakens the claims regarding practical deployment.
-
Insufficient baseline comparisons. Several closely related continual learning methods, including SimpleCIL and RanPAC, are missing from the experimental comparisons, despite their relevance to normalized classifiers, random projection, and analytic incremental learning. Furthermore, first-session adaptation is applied selectively across baselines, making it difficult to assess the fairness of the comparisons. The comparison against joint fine-tuning is also insufficient to support the claimed performance advantage, as a joint analytic classifier using matched feature representations would provide a more appropriate reference.
-
Insufficient analysis of the proposed components and their mathematical justification. Although ablation studies are provided, they do not isolate the contribution of angular class mean normalization from conventional feature normalization. Similarly, the effectiveness of test-time magnitude augmentation is demonstrated empirically without sufficient justification for using normalized features during training and unnormalized features during inference. The incremental covariance update in Equation (6) does not appear to be mathematically equivalent to the covariance definition in Equation (5). In particular, the update does not fully account for changes in class mean directions and uses the total sample count rather than the class-specific count in its correction factor. Although Appendix A discusses the angular covariance approximation, it does not adequately justify this incremental update or analyze its approximation error.
-
Insufficient evaluation of OOD detection and practical deployment. The paper mainly reports aggregate F-scores for OOD detection without sufficiently analyzing how detection performance and annotation requirements evolve during deployment. The claim that AUROC and AUPRC cannot be applied to dynamic OOD detection is also insufficiently justified, as these metrics could be evaluated at individual deployment stages. Furthermore, the experiments do not consider annotation errors or realistic labeling costs, despite emphasizing practical deployment.
Questions for authors
-
What is the key methodological novelty beyond combining existing normalization, random projection, and LDA-based incremental learning techniques?
-
How does the proposed learning-after-deployment framework differ from existing open-world continual learning settings, particularly without implementing label acquisition?
-
Can the authors provide comparisons with closely related methods such as SimpleCIL and RanPAC under consistent adaptation and evaluation settings?
-
Can the authors isolate the contribution of Angular LDA from standard feature normalization and clarify the mathematical correctness of Equation (6)?
-
Can the authors provide a more detailed evaluation of dynamic OOD detection, annotation costs, and robustness to imperfect labels?
7 comments
Q3:
Can the authors provide comparisons with closely related methods such as SimpleCIL and RanPAC under consistent adaptation and evaluation settings?
Response:
We ran SimpleCIL, RanPAC and SLDA on exactly the same adapted backbone features and the same streams as LAFA and PLDA (3 runs). Each method is evaluated in two ways. In the first, we assume a perfect OOD detector: every novel-class sample is detected and its label is given. In the second, each method runs with its own OOD detector inside the same post-deployment loop as LAFA. Since SimmpleCIL, RanPAC and SLDA does not have an OOD detector, we use their own highest classification scores (which is also what our LAFA system uses) as the OOD score, the same emerging-class rule, and the same well-learned threshold (30 labels); PLDA uses its own OOD score.
The results are given in the table below. Accuracy (%) is measured on the test set of all classes (base and novel) at the end of the stream, and the four dataset columns are with each method's own OOD detector. "Queries" is the average number of label requests per stream (the average stream has 4521 samples).
| Method | CIFAR100 | TinyImageNet | ImageNet-R | CUB | Average accuracy (%), own OOD detector | Average accuracy (%), assuming a perfect OOD detector | Average number of queries |
|---|---|---|---|---|---|---|---|
| SLDA | 89.02 | 84.45 | 80.91 | 82.63 | 84.25 | 84.25 | 4517 |
| RanPAC | 81.37 | 78.59 | 79.79 | 82.17 | 80.48 | 80.66 | 2308 |
| SimpleCIL | 87.97 | 84.31 | 81.91 | 80.11 | 83.58 | 84.42 | 1809 |
| PLDA | 87.33 | 84.26 | 81.66 | 82.05 | 83.83 | 84.61 | 1931 |
| LAFA (ours) | 90.41 | 86.04 | 83.84 | 82.88 | 85.79 | 86.41 | 1846 |
LAFA is the most accurate method on every dataset, both with its own OOD detector and assuming a perfect OOD detector. With its own OOD detector (85.79%), it is even more accurate than every other method that is given a perfect OOD detector (at most 84.61%). Note that in our paper, we compared our method with a more recent method, LoRanPac, which improved RanPAC.
First-session adaptation (FSA) fine-tunes the pre-trained model once before deployment, with a light adapter, using only base-class data. Following LoRanPac, the paper applies FSA to the analytic methods that keep the backbone fixed afterwards (LoRanPac, PLDA and LAFA). The replay-based baselines (A-GEM, GDumb, ER, iCaRL, MIR) already adapt the backbone themselves, by training adapters during the stream on all classes, including the novel ones. All methods in the table above share exactly the same FSA backbone, and without FSA, LAFA also outperforms PLDA (Appendix I: 83.01% vs. 80.66% average accuracy with DINOv2).
As a reference with matched features, we also trained joint analytic classifiers offline, on the same projected features and with all labels of the same data given at once. This is the data LAFA would learn assuming a perfect OOD detector: the base-class training data plus the novel-class samples of the stream. Accuracy (%) is on the test set of all classes, averaged over 3 runs.
| Method (same features) | CIFAR100 | TinyImageNet | ImageNet-R | CUB | Average accuracy (%) |
|---|---|---|---|---|---|
| Joint LDA (ordinary LDA, offline, all labels) | 88.11 | 84.13 | 76.06 | 81.16 | 82.36 |
| Joint angular LDA (offline, all labels) | 91.61 | 87.08 | 83.88 | 83.80 | 86.59 |
| LAFA (ours), from the stream, assuming a perfect OOD detector | 91.27 | 86.45 | 83.91 | 84.00 | 86.41 |
| LAFA (ours), from the stream, own OOD detector | 90.41 | 86.04 | 83.84 | 82.88 | 85.79 |
LAFA learns one sample at a time, with no stored data and with its own OOD detector, and still stays within 0.8 points of the same classifier trained offline with all labels (85.79% vs. 86.59%). Even offline, angular LDA is 4.2 points more accurate than ordinary LDA (86.59% vs. 82.36%), so the angular statistics help independently of streaming.
Q4:
Can the authors isolate the contribution of Angular LDA from standard feature normalization and clarify the mathematical correctness of Equation (6)?
Response (part 1 of 3: angular LDA vs. standard feature normalization):
If we understand correctly, standard feature normalization normalize each feature based on the feature of all samples. This is not applicable in our setting: after deployment, samples are encountered one at a time from a stream and are not stored, so such statistics cannot be computed over the data the system learns. LAFA uses L2-normalization, which uses only the sample itself, together with angular class statistics.
The contribution of the angular statistics is isolated by the LAFA−N variant in Table 1, which is the same system with standard (non-angular) LDA on the raw projected features. Removing the angular statistics lowers average accuracy from 85.40% to 80.15%, and the F-score drops from 92.51% to 65.07% on ImageNet-R and from 88.35% to 66.69% on CUB.
Response (part 2 of 3: unnormalized features at inference, TMA):
Let be the projected feature of a test sample, its length, and its direction. For class , is the mean direction and is the precision matrix (the inverse of the shared covariance). The LDA score is
where the first term measures how well the direction of the sample matches class , and is a class-dependent bias (one number per class). Because the shared covariance is estimated mostly from base-class data, is larger for the few-shot novel classes (up to 3.2× on CIFAR100), which pushes predictions away from them. Using the raw feature multiplies the first term by , which rebalances it against ; on CIFAR100, novel-class accuracy rises from 68.7% to 88.2%. Without TMA (normalized test features, the −M variant) average accuracy is 77.39%, and with TMA it is 85.79%.
Response (part 3 of 3: Eq. 6):
Eq. (6) defines a separate online estimator; it is not meant to reproduce Eq. (5) exactly at every step. It is a running average of the shared covariance over all samples: each new sample adds its residual , and the residuals of past samples are kept as they are. Because is shared by all classes, the correction factor uses the total count , not the class count . The first sample of a new class is measured against , since the class has no mean yet.
We also implemented an update that reproduces Eq. (5) exactly at every step, including the changes of the class mean directions. If is the sum of the normalized features of class , the within-class scatter equals with , so it can be updated exactly from the class sums that LAFA already stores. We chose Eq. (6) because it is as good on three datasets and better on CUB.
To measure the approximation error and its effect on decisions, we kept both estimators side by side on the same stream, with only Eq. (6) driving the decisions, and compared the predicted classes and the OOD decisions on the test set at the end of the stream, see the first two rows in the table below. We also ran LAFA end to end with each estimator. Results are averaged over 3 runs, and accuracy (%) is on the test set of all classes.
| CIFAR100 | TinyImageNet | ImageNet-R | CUB | |
|---|---|---|---|---|
| Same predicted class (% of test samples) | 99.3 | 99.4 | 98.6 | 96.7 |
| Same OOD decision (% of test samples) | 99.5 | 99.6 | 99.8 | 98.2 |
| Accuracy (%), our covariance update (Eq. 6) | 90.41 ± 0.51 | 86.04 ± 0.42 | 83.84 ± 0.20 | 82.88 ± 0.40 |
| Accuracy (%), exact update of Eq. (5) | 90.56 ± 0.24 | 85.98 ± 0.35 | 83.80 ± 0.15 | 82.14 ± 0.51 |
On CIFAR100, TinyImageNet and ImageNet-R, which have tens of samples per novel class, the two estimators make the same decision on 98.6 to 99.8% of test samples, and their accuracies differ by less than one standard deviation. On CUB, with about 3 samples per novel class, Eq. (6) is 0.74 points more accurate. In Eq. (6), the first sample of a new class is measured against , and the first residuals of every class receive full weight. Both increase the variance along the directions of the new class, which works like a prior variance for a class with very few samples and lowers the class bias of new classes. With the exact update, novel-class accuracy on CUB falls from 76.27% to 74.84%, while base-class accuracy is unchanged. The exact update and this comparison are now in Appendix A of the revised paper.
Thank you for the thorough review. We address each weakness and question below, and a revised PDF is uploaded.
Q1:
Response:
First, one key difference is the mean direction and angular statistics used in our system rather than the arithmetic mean as used in LDA, which results in major improvements. "Random projection + incremental LDA + the post-deployment pipeline" without the angular statistics is exactly the LAFA−N variant in Table 1. Its average accuracy is 80.15%, lower than PLDA (82.90%) and 5.25 points below LAFA (85.40%), and its F-score falls from 92.51% to 65.07% on ImageNet-R and from 88.35% to 66.69% on CUB. With few samples per novel class, the mean direction is what makes the same LDA score usable for both classification and OOD detection.
Second, at test time, test-time magnitude augmentation (TMA) uses the raw, unnormalized feature, which rebalances the class-dependent bias of the LDA score in favor of the few-shot novel classes (see our response to Q4). For example, on CIFAR100, novel-class accuracy rises from 68.7% to 88.2%.
Finally, a single score is used for both classification and OOD detection, so detection improves as new classes are learned.
Also, importantly, the proposed LAFA achieves accuracy comparable to joint training (i.e., retraining from scratch using all available data). We believe this level of performance demonstrates its readiness for practical deployment.
To the best of our knowledge, no existing method combines angular LDA and TMA, both of which are instrumental in enabling effective learning from limited data—a defining characteristic of post-deployment learning. In our search, we found that SimpleCIL (Zhou et al., 2023) normalize its features, but does so during both training and at test time and it also does not detect OOD samples and does not decide which samples to learn from a stream. LAFA normalizes only when building the class statistics, uses the raw feature at test time (i.e., TMA), and uses one score for both classification and OOD detection. SimpleCIL is included as a baseline in our response to Q3, where we will see that LAFA is significantly more accurate on every dataset.
Q2:
Response:
Our setting is described in the Introduction (Section 1), and its differences from related settings are discussed in Related Work (Section 2). In online class-incremental learning (online CIL), every incoming sample arrives with its label, and the learner never decides whether to learn it. In our setting, a label exists only if the system decides to ask for it, so the system must first detect that a sample is out-of-distribution (OOD). If it asks for labels it does not need, the labels are wasted (low precision); if it misses novel samples, it never learns them (low recall). Its detection errors therefore decide what it learns. The set of known classes also grows sample by sample, as a novel class goes from unseen, to emerging (1 to 29 labels received), to well-learned (30 labels or more).
Our setting is similar to that of open-world continual learning (OWCL)(e.g., Bendale & Boult, 2015). However, in OWCL, novel samples are usually detected against a fixed set of known classes, collected, and learned later in a batch. In our setting, detection and learning are interleaved for every single sample, with no buffer of stored samples, and the detector must also recognize emerging classes that it is currently learning.
As stated in Introduction section, the post-deployment system performs three tasks: (1) classify incoming samples and detect OOD samples, (2) obtain the label of a detected sample from a human or an AI agent, and (3) learn it incrementally. This paper focuses on tasks (1) and (3); task (2) usually involves a natural-language dialogue, and we leave it to future work. In the experiments, the human is simulated by giving the true label of each queried sample, as in active learning and in PLDA. The system still does not just reduce to OOD detection plus supervised online CIL, because it decides which samples receive labels: what it learns and how many labels it asks for both depend on its own detection decisions.
Since a real annotator can make mistakes, we also ran LAFA with a noisy oracle that returns wrong labels (see our response to Q5): with 20% wrong labels, its accuracy drops by only about one point, and it still stays more accurate than PLDA.