- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Test-Time Error Minimization (TTEM) for test-time open-set recognition (TOSR), where models must distinguish known from unknown classes using an unlabeled test stream. The central insight is that, under a fixed known-class recall, minimizing the total classification error is equivalent to maximizing the number of negative predictions. Based on this observation, the authors introduce a discrete-boundary ensemble that learns candidate decision boundaries using kernel SVMs, greedily selects boundaries to increase negative predictions, and combines them through an OR rule. An adaptive detector further switches between a prior detector and the test-time detector to mitigate the cold-start problem. Experiments verify the advantages of the proposed method.
Critical strengths
- The idea of reformulating error minimization as negative prediction maximization under a fixed known-class recall is interesting.
- The method achieves strong results on several benchmarks, particularly with CLIP features, and also works with conventional backbones.
Critical weaknesses
- The main concern is whether the surrogate objective is valid in the actual test-time setting. The equivalence requires the same known-class recall on the test stream, but the method calibrates recall on a held-out validation set. The paper does not establish why this calibration should remain reliable on unlabeled test data, especially under distribution shift. The reported results are encouraging, but do not resolve this issue.
- The threshold calibration in Eq. (5) appears inconsistent with the stated goal of retaining 99.9% of known-class samples. Given the decision rule in Eq. (6), the use of the upper Gaussian quantile may lead to the opposite behavior. The authors should clarify this discrepancy and explain how the target recall is maintained by the final ensemble.
- TTEM relies on the prior detector to filter candidate unknown samples for boundary learning, potentially inheriting its detection biases. Unknown patterns missed by the prior detector may therefore remain underrepresented during adaptation. The authors should evaluate the robustness of TTEM to prior detector quality and clarify whether it can effectively discover unknown patterns beyond those identified by the initial detector.
- The computational cost is not sufficiently analyzed. The method repeatedly trains kernel SVMs and reranks boundaries using accumulated prediction records, yet provides no systematic complexity analysis or detailed runtime comparison. The memory budget alone does not establish computational efficiency, especially since the paper acknowledges that adaptation is relatively slow.
Questions for authors
- Under what assumptions does validation-set recall calibration transfer to the unlabeled test stream, and how is the resulting uncertainty quantified?
- Does the number of negative predictions reliably track actual test error under known-class distribution shifts or non-stationary test streams?
- How are the boundary thresholds and ensemble size (K) calibrated? Please clarify the apparent quantile-direction issue in Eq. (5) and explain how recall is maintained by the final detector.
- What are the runtime and computational costs of SVM training, prediction-record maintenance, and greedy boundary ranking as the test stream grows?
We sincerely thank the Area Chair and all reviewers for their time and constructive feedback. Your insightful comments have been extremely helpful in identifying ways to improve the quality and clarity of our work. We would like to take this opportunity to clarify several concerns regarding the weaknesses raised during the review.
(1) How does the algorithm guarantee the recall on the test set?
One concern raised about our algorithm is how it guarantees the recall on the test set. We would like to point out that this problem has already been well addressed and that standard practices for handling it have been established in the OSR and OOD communities. Our method follows these established practices. A straightforward indication is that FPR@TPR95 is a standard evaluation metric in this field, where TPR95 explicitly refers to a recall of 95%. If controlling the recall were an unsolvable problem, the practical significance of the entire field would be questionable. Fortunately, since each algorithm produces a one-dimensional detection score, we only need to estimate the probability density in this one-dimensional space and determine an appropriate threshold. Density estimation in a one-dimensional space is a trivial problem. By simply constructing an empirical cumulative distribution function (CDF), we can readily achieve an estimation error of less than 1%, even without relying on any specific AI-based algorithms.
(2) How does the algorithm preserve the classification accuracy on known classes?
Another concern raised about our algorithm is how it guarantees the classification accuracy on known classes. We would like to point out that this problem has also been well addressed, with standard practices established in the OSR and OOD communities. Our method follows these established practices. Specifically, the standard approach is to keep the classification-related parameters fixed. This broad family of methods is commonly referred to as post-hoc methods.
(3) Computational efficiency.
Regarding computational efficiency, few algorithms can achieve breakthroughs in both performance and speed simultaneously. In most cases, techniques are progressively improved over time. We acknowledge that computational efficiency is not a strength of our current algorithm. We will provide more detailed runtime experiments in the revised version to offer readers a better understanding of the computational cost of our method.
Once again, we sincerely thank the Area Chair and all reviewers for their time and effort, and constructive feedback.