acceptodds
Under review as a conference paper at ICLR 2027

Computation–Performance Benchmark for Generalized Industrial Anomaly Detection

Abstract

Recent industrial anomaly detection (IAD) research increasingly targets three generalization settings: (1) multi-class unified detection with a single shared model, (2) few-shot detection with limited normal data, and (3) zero-shot detection without a designated target-domain support set before deployment. These settings require models to generalize robustly under data-scarce, cross-scenario conditions. Although recent methods achieve high accuracy on benchmark datasets, substantial differences remain in deployment cost. Here, we present a computational taxonomy and a benchmark of 30 representative methods on MVTec-AD and VisA dataset. Evaluation on a single NVIDIA A100 combines detection and localization metrics with batch-one throughput, peak GPU memory, parameter counts, and profiled query FLOPs. Through analyses of predictive performance, few-shot stability, and computational cost, we characterize the strengths and limitations of existing methods under different generalization settings. Our findings reveal that accuracy rankings alone provide an incomplete account of practical effectiveness, motivating an evaluation perspective that connects detection quality with the state and computation required at deployment.

Primary area:
datasets and benchmarks
Keywords:
Industrial Anomaly Detection, Deployment Efficiency, Computation-Accuracy Tradeoffs

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ufP7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares 30 industrial anomaly detection configurations across multi-class, few-shot, and zero-shot settings on MVTec-AD and VisA. It reports detection and localization scores alongside throughput, GPU memory, parameter counts, and FLOPs. It also examines one-shot ranking stability and the cost of retaining reference features. The goal is to provide a more useful basis for selecting detectors under deployment constraints.

Critical strengths
  1. The comparisons have practical value and provide useful information for deployment. Methods with similar detection performance can differ substantially in inference speed and GPU memory usage, which can affect the choice of method in practice.
  2. The paper provides concrete evidence of the distinction between detection and localization.
  3. The few-shot comparison is reasonably designed. All methods use the same support sets across the five one-shot runs, making small performance differences between methods easier to interpret.
Critical weaknesses
  1. The efficiency measurements are not documented in enough detail to assess their reproducibility and comparability. The timer boundaries, GPU synchronization, numerical precision, and number of timed repetitions remain unclear. Since the paper’s main contribution relies on comparing deployment costs, these details are essential for interpreting the reported differences.
  2. Although the paper broadens the coverage of methods and adds paired few-shot experiments, both of which are useful, much of the analysis still focuses on describing differences between the selected configurations. The additional contribution over existing benchmarks[1] therefore needs clearer justification. Grouping methods by how they compute anomaly scores is mainly descriptive. Measurements showing how different scoring procedures affect computational cost would make its value more convincing.
  3. The deployment and budget conclusion for PromptAD is not fully supported by the reported timing. The paper excludes PromptAD from throughput comparisons because its timing omits post-processing, yet Section 4.4 uses it as an example satisfying the 1,000 MB and 50 images/s budget. It is therefore unclear whether the complete pipeline meets this requirement.

[1] Xie et al. IM-IAD: Industrial Image Anomaly Detection Benchmark in Manufacturing. IEEE Transactions on Cybernetics, 2024.

Questions for authors
  1. Where does the timer start and stop for each method? Are reference retrieval, CPU–GPU transfers, and post-processing included consistently? Please also clarify how GPU synchronization and repeated timing measurements are handled.
  2. Does PromptAD still meet the 50 images/s threshold in Section 4.4 when post-processing is included? If complete timing is unavailable, could the authors revise the deployment-budget example?
  3. For MetaUAS, can the authors confirm whether the reported VisA accuracy was measured at the same input resolution as the throughput? If not, which accuracy result should be paired with the reported efficiency?
  4. Could the authors provide the five paired run-level differences behind the ranking reversals in Section 4.3?
Reviewer_ACLTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper constructs a benchmark for comparing the anomaly detection accuracy (I-AUROC, P-AUROC, AU-PRO) and deployment cost (batch-one throughput, peak GPU memory, parameter counts, FLOPs) of methods under three settings in industrial anomaly detection, including multi-class unified detection, few-shot detection, and zero-shot detection. Under the unified conditions of a single NVIDIA A100 and a batch size of one, experiments on MVTec-AD and VisA with 30 methods, each run in its documented configuration, yield four finds. Methods with comparable accuracy can differ in deployment cost by an order of magnitude. Higher image-level detection accuracy does not ensure higher accurate localization. Small few-shot mean gaps may depend on the sampled support set. Parameter count does not capture peak GPU memory, and lower FLOPs do not consistently imply higher throughput.

Critical strengths
  1. As many as 30 methods from 2023 to 2026 are evaluated on the same A100 GPU, covering CLIP-based, DINO-based, diffusion-based, and LVLM-based methods. The coverage is broad. It is of reference value for the practical application of these methods.
Critical weaknesses
  1. Finding 1 is not an experimental finding. The cost is determined by the architecture and is independent of accuracy. That is, the cost of any two methods is different, and this is always true regardless of accuracy.
  2. Finding 1 and 4 are already present in EfficientAD, Finding 2 is widely known and shown in different methods, and Finding 3 is already present in AnomalyDINO and CAReg. The incremental contribution of this paper is mainly the updated quantification under the same A100 GPU.
  3. No supporting materials were submitted. The absence of the support manifests and the modification code for the methods such as RAD makes it difficult to reproduce.
Questions for authors
  1. Could you add several representative methods' results on MVTec AD 2 or Real-IAD under the multi-class unified setting? This will help verify whether Finding 1 still holds on datasets where accuracy is not saturated. Whether low-cost methods remain competitive on harder data needs verification.
  2. Could you release the code and the support-set manifests for reproducing the results?
  3. Were these methods re-trained, or were the officially released weights used? The I-AUROC of AnomalyCLIP on VisA differs significantly from the original paper (0.888 vs. 0.821). What causes this discrepancy, and would it affect the fairness of the comparison?
  4. Could you clarify what new insights this paper provides, compared with the works mentioned in weakness 2?
Reviewer_jdHXOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper benchmarks 30 industrial anomaly detection methods across multi-class, few-shot, and zero-shot settings on MVTec AD and VisA. It evaluates detection accuracy and computational efficiency, including throughput, GPU memory, parameters, and FLOPs. The results highlight trade-offs between accuracy and efficiency, differences between detection and localization performance, and the limitations of parameter counts in estimating deployment costs.

Critical strengths

The paper addresses an important practical issue in industrial anomaly detection by considering both detection performance and computational efficiency. This evaluation perspective is relevant and valuable for real-world deployment.

Critical weaknesses
  1. Although balancing detection accuracy and computational efficiency is practically important, the paper's main findings are largely predictable and provide limited new insights. Moreover, the trade-off between accuracy and efficiency is highly application-dependent. For example, medical anomaly detection may prioritize accuracy over speed, whereas high-throughput industrial inspection may favor efficiency. Therefore, it is difficult to establish a universally meaningful comparison or determine which trade-off is preferable without considering specific application requirements.

  2. As a benchmark study, the paper has limited potential to guide future research. Although it incorporates computational efficiency into the evaluation, it does not establish a sufficiently compelling and standardized framework for subsequent studies to follow. Researchers may still primarily compare methods based on detection accuracy, making it unclear how this benchmark would meaningfully influence future evaluation practices.

  3. The manuscript lacks the mandatory AI Use Statement required by the ICLR 2027 submission guidelines

Questions for authors

NONE

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.