- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a post-hoc approach for failure prediction in object detection and instance segmentation. The proposed FADE framework trains a meta-classifier, a lightweight feedforward network, to predict the probability that a prediction of the base model is correct (IoU with a same-class ground-truth object above a threshold ). Its input combines four families of features: semantic features extracted with CLIP, geometric features, the score of the base model, and optionally task-specific structural features. Experiments are conducted with various classical models and Vision Foundation Models (VFMs) on the public COCO, Pascal VOC and Cityscapes datasets, and on two non-public domain-specific object detection datasets (fish detection and surgical instruments).
Critical strengths
- Failure prediction for detection and segmentation is a highly relevant problem, especially for VFMs used zero-shot.
- The method is simple and relevant.
- Experiments are conducted across different datasets and with a wide range of models: six classical models and five VFMs on COCO, Pascal VOC and Cityscapes.
Critical weaknesses
-
Methodological novelty is limited: FADE essentially combines MetaSeg/MetaDetect-style meta-classification with CLIP crop features and score/geometry recalibration in the spirit of Küppers et al. The contribution is therefore mainly empirical.
-
Weak baselines. Apart from the raw score, all comparisons are ablations of FADE itself. The configuration , presented as the MetaSeg/MetaDetect baseline, is a stripped-down version of these methods: a few size and perimeter statistics. It lacks what makes them effective: the softmax dispersion measures (MetaSeg), the pre-NMS candidate statistics (MetaDetect), and even the score itself. The comparison with therefore does not show that FADE outperforms these methods. I would expect:
- The actual MetaSeg/MetaDetect on models whose internals are accessible (e.g., Faster R-CNN, Mask R-CNN), which would also quantify the cost of model-agnosticism.
- VLM-based post-hoc failure predictors such as ViLU (Lafon et al., 2025, cited in the paper). In particular, FADE only uses the CLIP image embedding of the crop and ignores the text embedding of the predicted class.
- Standard recalibration of the raw score (temperature scaling, Platt, isotonic) and position-dependent calibration (Küppers et al., cited), which is close to the configuration; model-specific estimators where available, e.g., SAM's IoU prediction head or UncertainSAM.
-
Training a separate meta-classifier for each model, dataset and task is a major limitation.
- It requires labeled data from the target domain for every model. This conflicts with the zero-shot motivation for VFMs: with such data, one could adapt the VFM directly. The fish dataset has only 46 images, i.e., about 7 test images.
- The common feature representation is the paper's main argument, but it is never exploited for generalization. There is no transfer across models, datasets or tasks, and no single meta-classifier shared by detection and segmentation, despite the "task-agnostic" claim. In the current design, transfer is not even possible: has the dimension of the dataset vocabulary, and the PCA is refit for each model–dataset pair with a different k, so lives in different spaces. This also weakens the claim that the estimates are "directly comparable across model families" (Fig. 9).
- The meta-classifier is only evaluated on i.i.d. splits, whereas the paper is motivated by distribution shift. Cross-dataset experiments (e.g., train on COCO, test on VOC or Cityscapes) are needed.
-
Training objective and hyperparameters.
- The loss is not specified (presumably binary cross-entropy on ).
- Binarizing at threshold \tau is the key design choice, yet it is set to 0.1 and never ablated. is very lenient: errors become mostly hallucinations and misclassifications rather than localization errors. Moreover, without one-to-one matching, duplicates and fragments count as successes, so the definition of an error differs from the one underlying AP. Results for different are needed.
- More fundamentally, thresholding is unnecessary. One could regress the IoU directly, as in MetaSeg's meta-regression. Better still, one could predict a distribution over IoU (e.g., cross-entropy over discretized IoU bins, or a CRPS loss), which gives P(IoU > ) for any at test time. Since the CRPS equals the Brier score integrated over all thresholds, it is a principled way to remove the dependence on \tau.
- Other choices also need discussion: random oversampling without prior correction (which biases the predicted probabilities, see below), k tuned per model–dataset pair on validation AUROC, and checkpoint selection on an unspecified validation loss.
-
Experimental claims and findings.
-
Cost and robustness of the findings.
- The method is presented as lightweight, but this holds only for the meta-classifier: CLIP feature extraction costs 200–310 ms per object, so FADE's overhead (9.54 s/image) exceeds the cost of the VFM itself (7.69 s/image, Table 14).
- The 10 folds share a fixed test set, so the reported standard errors ignore test-sampling variance, and the domain-specific results rely on tiny test sets
- Random oversampling makes the training distribution differ from the test distribution by a model-dependent factor. Its effect on both AUROC and ECE should be assessed, e.g., against class-weighted training with prior correction, or no rebalancing.
-
Comparing models with failure-prediction and UQ metrics, claimed as a contribution, is potentially misleading. The AUROC and ECE of the meta-classifier \phi characterize the pair (model, meta-classifier) on a model-specific prediction set. They do not measure the reliability of the base model itself.
- AUROC ignores accuracy and depends on the model's error profile. Stronger models leave fewer and harder errors, possibly close to aleatoric uncertainty, while weaker models make gross errors that are easy to detect (cf. Jaeger et al., ICLR 2023). Appendix A.7 illustrates this: fine-tuning Grounding DINO raises AP (0.418 -> 0.525) while AUROC drops (0.853 -> 0.806).
- Prediction sets differ across models: post-processing (top-k vs. score threshold), number of predictions, and positive rate, none of which are reported for COCO, VOC or Cityscapes. Each model thus defines a different failure-prediction task, and many easy low-score negatives inflate AUROC.
-
Calibration evaluation. The claim that VFMs are worse calibrated (ECE 0.034 vs. 0.017) is drawn from a score designed and tuned for failure prediction, not for calibration. Failure prediction (ranking errors below successes) and calibration are distinct, sometimes conflicting objectives (e.g., Zhu et al., ECCV 2022). Several choices explicitly favor discrimination over probability estimation: random oversampling without prior correction, and selection of the PCA dimension on validation AUROC. Differences in ECE therefore reflect how well a discrimination-tuned estimator happens to be calibrated, not the reliability of the underlying VFM.
-
Questions for authors
I find this work interesting and the problem well motivated. My main concerns are the baselines and the genericity of the approach. In the rebuttal, I would mainly expect:
- Comparison with stronger baseline (see weaknesses)
- Transfer: evidence, even preliminary, that a meta-classifier trained on one model or dataset transfers to another (e.g., across models on COCO, or from COCO to VOC/Cityscapes).