- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PROBE, a post-hoc open-set recognition method for frozen classifiers, with a particular focus on fine-grained recognition. The method combines two components: (1) class-conditional isotropic Gaussian energies computed from unit-normalized features, and (2) an observation-aware covariance adjustment estimated from paired feature displacements induced by mild image corruptions. To avoid excessively enlarging class support along discriminative directions, the authors introduce a between-class scatter penalty when estimating the observation subspace. The resulting rejection score is computed using a class-conditional energy function without retraining the classifier or modifying its predictions.
Critical strengths
- The paper addresses an important challenge in fine-grained open-set recognition: distinguishing observation-induced feature variations from semantic differences between visually similar categories. The motivation is intuitive, and the discussion of why uniformly expanding known-class support may negatively affect unknown-class rejection is reasonable.
- PROBE does not require retraining the backbone or classification head, and the subspace estimation has a closed-form solution.
Critical weaknesses
- The proposed method primarily combines existing techniques, including feature normalization, Gaussian energy modeling, perturbation-based subspace estimation, and between-class scatter regularization. In particular, the penalized eigendecomposition of (S_{\mathrm{obs}}-\beta S_B) and the subsequent covariance adjustment appear to be relatively straightforward extensions of established ideas. The paper does not sufficiently clarify its fundamental methodological novelty beyond this combination.
- PROBE assumes that directions separating known-class means can serve as a proxy for directions important for rejecting unknown classes. However, this assumption is not necessarily valid, especially in fine-grained settings where unseen categories may differ along directions not captured by known-class scatter. While the ablations demonstrate the effectiveness of the penalty, they do not directly validate this underlying assumption.
- The observation subspace is estimated using synthetic corruptions at a fixed severity level, which may not adequately represent real-world variations such as viewpoint changes, occlusion, and background shifts. Additionally, the experiments mainly consider fixed data partitions and probe realizations, leaving the robustness of the method across different acquisition conditions and experimental configurations insufficiently explored.
- The covariance adjustment derived from the positive spectrum of the penalized scatter matrix is largely heuristic. The paper lacks a principled explanation of why this particular construction should improve unknown-class rejection, or under what conditions the proposed between-class protection may become ineffective or even detrimental.
Questions for authors
- Can the authors clarify the fundamental methodological distinction between PROBE and existing approaches based on nuisance-invariant subspaces, discriminative scatter analysis, and covariance-based open-set scoring? What is the key new principle beyond combining these components?
- Is there empirical evidence that directions separating known-class means are also informative for distinguishing known classes from unseen, semantically similar classes? Could the authors directly analyze the projections of unknown-class feature differences onto the protected and unprotected subspaces?
- How sensitive is the method to the choice and sampling of corruption operators? In particular, does the improvement persist under genuine acquisition shifts involving viewpoint changes, background variation, or occlusion?
- Could the authors provide stronger evidence of the practical significance of the relatively small gains over the strongest baselines on several SSB benchmarks, including uncertainty estimates for paired method comparisons and additional probe realizations or data partitions?
4 comments
Official Response to Reviewer stRp (Part 2/2)
W3/Q3. Sensitivity to corruption choice; real acquisition shifts. Operator choice. Using all 19 operators gives the best AUROC on four of six datasets and is within 0.4 points of the best single family on the other two. The best family varies by dataset (weather on Aircraft, CUB, and DTS; digital on Cars; blur on VisioDECT; noise on ICG-Drone-3c). Uniform sampling is therefore robust and avoids per-dataset tuning.
Severity. Severity 1 is best on five of six datasets, and higher severity lowers AUROC (Aircraft 84.00 → 82.53, DTS 84.61 → 79.77 at severity 5). Stronger corruptions change predictions more often (Aircraft label preservation falls from 95.7% to 40.1%) and overlap more with between-class directions. Severity 1 keeps probes on the observation side, as intended.
Real acquisition shifts. On the UAV videos, we fit M2 using crops of the same target in consecutive frames. These pairs capture genuine changes in viewpoint, scale, background, and illumination.
Table R3: AUROC (%) with different probe sources.
| M1 only (paper Table 4) | Real pairs | Real + corruption pairs | Corruption pairs (paper) | |
|---|---|---|---|---|
| DTS | 78.62 | 82.42 | 84.14 | 84.61 |
| VisioDECT | 92.03 | 92.61 | 93.63 | 93.61 |
| ICG-Drone-3c | 73.89 | 73.37 | 74.67 | 75.88 |
- With real frame pairs alone, M2 improves over M1 by 3.80 points on DTS and 0.58 on VisioDECT. The mechanism therefore also works with genuine acquisition changes.
- Combining real and corruption pairs improves over M1 on all three datasets.
- Corruption probes alone match or beat real pairs, so synthetic probes suffice without video data.
Q4. Practical significance and uncertainty of the SSB margins. All post-hoc methods share one frozen classifier, so we computed paired bootstrap 95% CIs (2,000 test-set resamples) of PROBE's gain over the strongest other post-hoc method on each dataset.
Table R4: AUROC gain of PROBE (points) [paired bootstrap 95% CI].
| Strongest post-hoc | PROBE gain | |
|---|---|---|
| Aircraft | MLS | 3.18 [2.61, 3.72] |
| Cars | KNN | 0.46 [0.20, 0.72] |
| CUB | MSP | 0.61 [0.14, 1.04] |
| DTS | KNN | 3.40 [3.08, 3.71] |
| VisioDECT | ViM | 0.67 [0.54, 0.80] |
| ICG-Drone-3c | WDiscOOD | 1.48 [0.41, 2.57] |
- No interval includes zero, so the gains are statistically significant on all six datasets, including the smaller SSB margins on Cars and CUB.
- The strongest baseline changes from dataset to dataset (MLS, KNN, MSP, ViM, WDiscOOD), whereas PROBE is best on all six. In practice, one method works reliably across benchmarks without selecting a detector per dataset.
- SSB uses its standard benchmark splits. The UAV protocols use leave-one-class-out evaluation over 7, 6, and 3 folds, so every drone model serves as the unknown class once. For probes, uniform sampling over all 19 operators (W3) removes the dependence on any single operator subset.
W4. A principled explanation, and when protection fails. We derived the following proposition for the appendix. Let a known class be , an unknown class , and the scoring covariance with . Because the trace and log-determinant terms cancel, the expected energy gap is
- The gap shrinks only through : added tolerance costs separability exactly to the extent that it overlaps the unknown's displacement . Tolerance orthogonal to costs nothing, while it can still reduce false rejections of corrupted known samples.
- When lies in the span of between-class scatter, the -penalty limits this overlap. The analysis in W2 (Table R2) shows that this condition holds on all five datasets tested (94–98%).
- Failure condition. If is orthogonal to the between-class span but aligned with observation directions, the penalty provides no protection. With on all measured datasets, this case does not arise in our benchmarks; we will state it in the Limitations paragraph. Tables 12–15 of the paper support the remaining design choices: positive-spectrum clipping, the isotropic tail , and the additive form.
Thank you again; we are happy to clarify any remaining points.
I appreciate the authors’ responses. I still have two questions:
- The real acquisition-shift experiments use consecutive frames of the same UAV targets. In addition, the VisioDECT dataset is largely split into adjacent temporal blocks from the same recordings, with no gap between splits. This raises concerns about temporal correlation and recording-specific bias. Can the authors evaluate the method on test data from entirely separate recordings to verify that the improvements generalize beyond the recordings used for training and validation?
- The paired bootstrap appears to resample individual test samples, although UAV crops from the same recording may be highly correlated. Could the authors perform a recording-level or sequence-level bootstrap to check whether the reported confidence intervals and statistical significance still hold?
Follow-up Response: Separate-Recording Evaluation and Recording/Sequence-Level Bootstrap
Thank you for following up. We evaluated the frozen models without retraining or refitting, and quantified uncertainty using synchronized recording-level and temporal-block bootstrap analyses.
1. Generalization to separate recordings
We evaluated the frozen Sunny-trained ResNet-50 classifiers and Sunny-fitted detectors from Table 2 on the official Cloudy and Evening recordings, disjoint from training, validation, and detector fitting. All 6,768 Cloudy and 6,940 Evening crops (13,708 crops; 12 recordings; six classes) were evaluated under the same six-fold leave-one-class-out protocol. Combined metrics pool both subsets within each fold.
Table F1. Means across folds and three seeds; AUROC / OSCR (%), except for shared known-class accuracy (Acc).
| Method | Cloudy | Evening | Cloudy + Evening |
|---|---|---|---|
| (Shared-CE Acc) | (70.93) | (80.67) | (75.83) |
| MSP | 62.45 / 48.08 | 68.01 / 56.91 | 65.00 / 52.70 |
| MLS | 58.58 / 44.37 | 63.50 / 51.68 | 60.91 / 48.36 |
| Energy | 58.08 / 43.87 | 63.04 / 51.16 | 60.43 / 47.85 |
| KNN | 66.57 / 52.22 | 74.87 / 62.99 | 70.63 / 57.97 |
| Mahalanobis | 61.94 / 48.70 | 73.59 / 61.81 | 67.69 / 55.50 |
| ViM | 60.87 / 48.09 | 72.87 / 61.10 | 66.88 / 54.88 |
| Mahalanobis++ | 66.20 / 51.93 | 75.73 / 64.29 | 70.72 / 58.30 |
| WDiscOOD | 61.07 / 47.56 | 72.09 / 60.11 | 66.55 / 54.17 |
| PROBE | 68.41 / 54.60 | 77.13 / 65.79 | 72.62 / 60.35 |
| M1 only () | 64.61 / 50.25 | 72.30 / 60.40 | 68.56 / 55.86 |
| PROBE (Sunny real pairs) | 68.67 / 54.19 | 76.55 / 65.05 | 72.25 / 59.70 |
Default PROBE exceeds all eight competing post-hoc baselines in AUROC and OSCR on all three test sets. Its combined gains over Mahalanobis++ are 1.90 AUROC and 2.05 OSCR points; the AUROC gain over ViM is 5.74 points. Relative to M1 (68.56% combined AUROC), PROBE using Sunny real pairs achieves 72.25% (+3.69 points), and PROBE using Sunny corruption probes achieves 72.62% (+4.06 points). Both variants exceed all eight baselines on both metrics in all three test sets. Frame pairs are used for detector fitting on Sunny training data; inference scores individual crops.
2. Recording-level and temporal-block paired bootstrap
We used whole recordings and contiguous blocks defined once within each complete physical recording and mapped to each fold. Each draw assigns one Poisson(1) weight per unique recording or block, shared across all methods, folds, and three seeds. If any fold has zero total known or unknown weight, we discard the entire draw and resample until obtaining 2,000 valid draws. We compute paired AUROC differences within each seed and fold and average them as in Table 2.
Table F2. Synchronized cluster-bootstrap 95% intervals for AUROC gains (percentage points). K/U: mean known/unknown clusters per fold; total: distinct clusters across folds.
| Benchmark | Unit (avg. K/U; total) | vs. strongest competing baseline | vs. Mahalanobis++ | vs. WDiscOOD |
|---|---|---|---|---|
| DTS | Recording (7.7/4.1; 29) | KNN: +3.40 [1.93, 6.92] | +5.00 [3.46, 10.74] | +15.00 [11.39, 26.74] |
| DTS | 20-crop block (28.3/20.9; 146) | KNN: +3.40 [1.39, 5.74] | +5.00 [2.07, 8.54] | +15.00 [9.14, 21.43] |
| VisioDECT Sunny | 10-crop block (117.5/115.5; 693) | ViM: +0.67 [0.22, 1.16] | +2.04 [1.44, 2.63] | +0.73 [0.14, 1.37] |
| VisioDECT Sunny | 20-crop block (60.0/57.8; 347) | ViM: +0.67 [0.04, 1.28] | +2.04 [1.18, 2.85] | +0.73 [-0.14, 1.59] |
| VisioDECT Cloudy+Eve. | 20-crop block (575.8/115.2; 691) | Mah++: +1.90 [1.43, 2.36] | +1.90 [1.43, 2.36] | +6.07 [5.33, 6.87] |
| VisioDECT Cloudy+Eve. | Recording (10.0/2.0; 12) | Mah++: +1.90 [-0.37, 4.31] | +1.90 [-0.37, 4.31] | +6.07 [2.33, 10.63] |
| ICG-Drone-3c | 10-crop block (13.0/35.3; 84) | WDisc: +1.48 [-0.88, 3.77] | +2.84 [0.06, 5.50] | +1.48 [-0.88, 3.77] |
| ICG-Drone-3c | Recording (4.3/11.0; 26) | WDisc: +1.48 [-2.63, 5.62] | +2.84 [-0.71, 7.39] | +1.48 [-2.63, 5.62] |
- DTS: Recording-level and 20-crop intervals exclude zero for all three reported comparisons.
- Sunny: The 10-crop intervals exclude zero for all three comparisons. The 20-crop intervals exclude zero against ViM and Mahalanobis++ and include zero against WDiscOOD.
- Cloudy+Evening: The 20-crop intervals exclude zero against Mahalanobis++, WDiscOOD, KNN (+1.99 [1.50, 2.49]), and ViM (+5.74 [5.08, 6.40]). Recording-level intervals exclude zero against WDiscOOD and ViM (+5.74 [2.73, 9.19]) and include zero against Mahalanobis++ and KNN (+1.99 [-0.57, 4.43]).
- ICG-Drone-3c: The 10-crop interval excludes zero against Mahalanobis++ and includes zero against WDiscOOD. Both recording-level intervals include zero.
We hope Tables F1-F2 thoroughly address your remaining questions. Please let us know if further details would help; otherwise, we hope this new evidence can be taken into account in your final assessment. Thank you again for your insightful feedback.
Official Response to Reviewer stRp (Part 1/2)
Thank you for the careful and insightful review. Your questions led to the new empirical and theoretical analyses below, which strengthen the paper and which we will incorporate into the revised manuscript.
W1/Q1. What is the new principle beyond combining components? The principle is this: a post-hoc Gaussian rejector should add tolerance mainly along directions in which known classes move under observation changes, and penalize the overlap of that tolerance with directions that separate classes. None of the three families you list separates these two roles:
The fixed-spectrum control (Table 5 of the paper) tests this principle in isolation. It keeps eigenvalues, trace, and log-determinant identical and changes only the subspace directions. The protected directions win on all six datasets.
W2/Q2. Are known-class-separating directions informative for unknowns? Projection analysis. Yes. We ran the projection analysis you asked for, at two levels.
Sample level. For a test sample, let r=z−μy^ and define its occupancy of a subspace U as ρ(U)=∥Ur∥2/∥r∥2. The selectivity ratio ρHard unknown/ρknown measures how much tolerance a subspace gives unknown residuals relative to known ones; below 1 is the desired behavior.
Table R1: Selectivity ratio.
Without protection, the observation subspace favors unknown residuals (ratio > 1). With between-class protection, the ratio drops to 0.81–0.92 on every dataset. Protection therefore moves tolerance away from where unknowns lie.
Class level. For each unknown class, let δ be the displacement from its mean to the nearest known-class mean, and let γ be the fraction of ∥δ∥2 that lies in the span of SB.
Table R2: Where unknown-class displacements lie.
The analysis covers five datasets; ICG-Drone-3c has two known classes per fold, so its span(SB) is one-dimensional. We will state the proxy explicitly as a premise in the revised manuscript; W4 (Part 2) gives the exact condition under which it would fail.