- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes TASTE, which constructs a task-aware out-of-distribution (OOD) signal by incorporating the score of the training distribution—along with the input gradients of a fixed predictor—into a Langevin Stein operator. This approach ensures that detection focuses on distribution shifts that impact model performance, rather than on all statistical variations. Furthermore, the authors use the training set to calibrate score errors and extend this signal to both sample-level detection and pixel-level localization.
Critical strengths
-
The proposed methodology offers some novelty. TASTE employs a fixed predictor as a test function for a Stein operator, integrating the score of the training distribution with the model’s sensitivity to its input. It further introduces a projection identity, a score-error correction mechanism, and both sample-wise and pixel-wise residuals.
-
The experiments are comprehensive and provide meaningful insights. The paper evaluates the approach across a diverse range of datasets, including Gaussian, MNIST, CIFAR-10, ImageNet-1K, and MVTec AD—assessing its out-of-distribution (OOD) detection performance while demonstrating its ability to capture the directionality of task-relevant shifts and perform pixel-level localization.
Critical weaknesses
-
The main methodological idea is to combine the score of the training distribution with the input sensitivity of a fixed predictor, with the full operator additionally using the predictor’s Laplacian. This is a meaningful and potentially novel construction. However, the broader goal of task-aware OOD detection has been explored in prior work, including Task-Driven OOD Detection, Sequential Harmful Shift Detection Without Labels, and Towards a Certificate of Trust. The paper should therefore clarify which aspects are genuinely new beyond this existing line of work. The experimental section includes several general-purpose OOD baselines, but the discussion and evaluation of the closest task-aware and joint input/output likelihood methods remain limited.
-
The multiclass experiments use the predicted-class softmax as the default probe. Although a pre-softmax logit probe appears in one auxiliary experiment, the paper does not explain how sensitive the method is to this choice. A brief comparison with a logit probe, or a clearer justification for using the softmax probe, would make the methodology easier to assess. The different variants should also be separated more clearly in the main results. Full TASTE uses the complete Langevin Stein operator; the no-Laplacian version is a cheaper first-order variant that still depends on the predictor gradient; score-only removes this predictor-sensitivity term; and TASTEext uses a classwise first-order construction for ImageNet. These variants have different interpretations and should be labeled consistently in the tables and captions.
-
The paper uses signed residuals in some experiments and in others. It would help to state explicitly how score direction, aggregation, and thresholding are defined for AUROC/FPR95, dataset-level shift detection, and pointwise OOD testing. A compact implementation table would also improve reproducibility by collecting the diffusion timestep, noise sampling, Hutchinson settings, calibration split, random seeds, repetitions, preprocessing, feature layers, and baseline hyperparameter choices. Most of this information may already be present in the appendix; the main need is to make it easier to find. As a minor optional analysis, the authors could report rank correlations between TASTE residuals and task performance in the controlled Gaussian or MNIST experiments.
Questions for authors
See the “Critical Weaknesses” above for details.
Q1. Label-preserving input shifts provide a natural setting for connecting predictor sensitivity to task performance. Sensor noise, blur, compression artefacts, illumination changes and class-preserving rotations or translations replace (X,Y) by (T(X),Y). Adversarial perturbations that preserve the true label fit the same setting: a successful attack on an initially correct prediction creates an error, potentially with high confidence. This makes adversarial examples a concrete motivation for relating input geometry to model sensitivity.
Sensitivity alone, however, does not imply degradation: confidence can change without changing the decision, and a class change can correct an existing error. Additionally, a changed labelling rule can increase error without changing the input distribution. We will make this distinction explicit in the revised submission.
Viewed as a hypothesis test, TASTE asks whether a new image is consistent with the reference distribution (Section 4). For a new image drawn from that distribution, rank-based thresholds calibrated on held-out images control the false-positive rate. Under a shift, the mean TASTE score is the expected inner product of the density-score difference and the model gradient (Eq. (5)). This explains why the same shift can produce different mean residuals across classifiers. We test whether this sensitivity tracks task degradation by asking whether TASTE AUROC falls when augmentation reduces classification error on the same corrupted images.
We add an empirical test using label-preserving MNIST rotations and CIFAR-10 Gaussian noise, with robustness varied through training augmentation. For each dataset, we train three standard/augmented classifier pairs. Within each pair, classifiers share the initialization and training data order; the augmented arm additionally receives rotations sampled uniformly from −30∘,30∘ for MNIST or Gaussian noise for CIFAR-10. We evaluate six rotation angles and five noise severities, giving 18 rotation and 15 noise seed-condition comparisons. We also evaluate translations of 4, 8, 12 and 16 pixels on padded MNIST images as a control. Both arms are evaluated on the same 2,000 clean images and their transformed counterparts. We measure excess classification error (shifted minus clean classification error) and TASTE AUROC for distinguishing shifted from clean images. This tests whether TASTE responds less strongly when the same shift becomes less harmful to the classifier. The table reports the −30∘ rotation, 16-pixel translation and noise-severity-5 results (mean ± sample SD across three training seeds).
Excess error and TASTE AUROC fell together at every rotation angle and noise severity for all three seeds. With the images and density-score estimator held fixed, TASTE separated the same shifts less strongly after the classifier became more robust to them. For the 16-pixel translation control, excess error was close to zero and AUROC was near chance for both classifiers.
Q3. MVTec demonstrates zero-shot localisation: neither the classifier nor the score model was trained or fine-tuned on MVTec data. We have not evaluated specialised anomaly detectors under the same zero-shot conditions and will present these results as a supplementary feasibility study.
Tables 1 and 2 show that performance varies across shifts for every detector evaluated. TASTE leads on both AUROC and FPR95 overall and on adversarial attacks on CIFAR-10. For classical OOD, Energy leads on both metrics on CIFAR-10 and ViM on ImageNet. Even the lowest overall FPR95 on CIFAR-10 corresponds to frequent false alarms on clean inputs. We will revise the phrase “well suited to post-deployment monitoring” in Section 6. TASTE complements existing detectors, particularly for adversarial inputs. Its use for monitoring would require validation on the shifts of interest at a false-alarm rate the application can tolerate.
Q4. Equally weighted ImageNet results.
Giving the classical and adversarial regimes equal weight, using the results from Table 2, changes the leading results as follows:
Under this alternative weighting, TASTEext remains first by AUROC and moves to second by FPR95, behind Mahalanobis. On classical OOD, ViM achieves 0.9428 AUROC versus 0.8485 for TASTEext. We will emphasise that these results should not be interpreted as a regime-independent advantage.