- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces BudgetMatch, a framework for documenting and checking differences in reference access when comparing anomaly detectors. Controlled experiments measure the effects of reference-bank size, contamination, and source using frozen features. The authors also audit published comparisons and argue that differences in access can be large enough to undermine the interpretation of reported performance margins.
Critical strengths
- The paper addresses a meaningful evaluation problem. Methods that use a few verified normal images and methods that access the entire unlabeled test cohort operate under different assumptions. Making these differences explicit helps readers interpret comparisons and judge their relevance to deployment.
- The experiments go beyond showing that more references help. The decomposition of reference size, contamination, and source provides useful evidence about why performance changes. Additional scorers, encoders, and datasets strengthen the analysis, and the authors clearly disclose several limitations and negative findings.
Critical weaknesses
-
The contribution beyond existing evaluation practice is limited. The distinction between inductive and transductive methods is established, and prior work such as AnomalyDINO already separates batched zero-shot evaluation from other settings. The main finding that increasing the number of clean references improves performance is somewhat expected. The proposed cards and validator make reporting more systematic, but checking which declared fields differ is straightforward. The paper needs a clearer explanation of what new understanding this framework provides beyond explicit protocol reporting and standard sensitivity experiments.
-
The headline comparison depends heavily on the scoring rule. With nearest-neighbor scoring, one clean reference matches or beats the full test cohort in 15 of 27 categories. Against official MuSc, this drops to only 4 categories, and even 16 clean references are insufficient in 8. This matters because nearest-neighbor scoring is particularly vulnerable to anomalous references: a single similar defect can suppress the anomaly score.
Questions for authors
-
What is the main new insight beyond the known benefit of more clean references and the distinction between inductive and transductive evaluation?
-
Have you tested whether giving existing methods the same reference data changes which method performs best?
3 comments
Response to the authors
Thanks for providing the extra evidence. Seeing how much reference access mattered in the published comparisons helped me understand what the audit adds and the revision addresses my main concern. Based on that I’m raising my score
Thank you
Thank you for reading our response and for raising your score. The evidence on how much reference access mattered in the published comparisons is part of the revision, and it will stay in the final version.
Response to Reviewer iVty
Thank you for the review and for Q2, which we have now answered with new runs.
The new insight is that the access fields a setting label leaves undeclared can be found and priced, and that setting this price against a printed margin shows whether a published ranking survives that field. The inductive/transductive distinction itself is established: the submission cites Bronskill et al. (2020), and the revision adds, in Section 2, their point that transductive methods "must be labeled as such". As you note, AnomalyDINO separates batched zero-shot evaluation; it places MuSc and ACR there, which removes the label ambiguity for them. Its few-shot Table 3, however, still places APRIL-GAN, which trains on another dataset's labeled test set and normalizes over the test set, beside training-free methods such as WinCLIP+ and its own AnomalyDINO-S. What we add is:
We agree. In the source × purity decomposition the cohort's anomalies cost +3.61 and +7.23 points (MVTec AD and VisA), against +0.41 and +0.50 for its source. Measured on a cohort subsample, the purity cost is +3.58 and +7.45 under 1-NN and falls to +2.35 and +4.19 under MuSc's interval average. In the 8 categories where even 16 clean images under our scorer do not reach official MuSc on the test cohort, the comparison changes the scorer as well as the access (Appendix O), so we read them as scorer dependence, not as a single-factor price. The revised Section 5 gives steps for pricing a field in each detector's own pipeline (response to Reviewer erdj, 2/2).
Yes. With access matched on clean reference images, none of the columns we re-ranked in three published tables has a single identifiable leader. Aligned APRIL-GAN runs without its auxiliary-trained branch and test-set normalization. Two rows cannot be told apart when the 95% interval of their difference includes 0 or, against a printed row, when the difference is within the larger of our range across support paths and the printed standard deviation.
For MuSc's Table 1, we restricted the compared methods to four clean training images; MuSc itself cannot run under this restriction, since its scoring needs the test set. Aligned APRIL-GAN cannot be told apart from MuSc's printed row on either benchmark, nor, on MVTec AD, from WinCLIP; on VisA it stays above PatchCore and WinCLIP. Neither column has a single leader.
The other reading of "the same reference data" gives every method the whole test set. Official MuSc against our 1-NN on the same full test set (self-excluded) differs by +1.71 [−1.23, +4.53] on MVTec AD (cannot be told apart) and +4.54 [+1.87, +7.32] on VisA (MuSc leads). The new Appendix Z gives the orderings, the released
R1_aprilgan_analysis.jsonandR1_inctrl_analysis.jsongive the numbers, and the revised Appendix U gives the whole-test-set comparison.We would be glad to answer any further questions.