acceptodds
Withdrawn from ICLR 2027

AnomalyPrompt: LLM-Guided Adaptive Prompting via Visual Perception for Zero-Shot Anomaly Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Industrial visual anomaly detection has long been constrained by the scarcity of anomalous samples and the high diversity of defect appearances. Recent CLIP/VLM-based methods have demonstrated promising cross-category generalization for ZSAD through prompt optimization. Most existing methods rely on static or manually crafted prompts or perform only passive adjustments, lacking mechanisms for active prompt exploration and quality assessment. Consequently, they struggle to capture fine-grained distinctions between normal and abnormal states and ultimately fail to consistently generate reliable prompts for anomaly detection. To address this limitation, we propose AnomalyPrompt, a training-efficient vision-language anomaly detection framework that unifies prompt generation, prompt evaluation, and visual-semantic alignment through a multi-stage prompting process. AnomalyPrompt operates through a structured “perception-reasoning-decision-execution” pipeline: it first uses the image-level embedding and patch embeddings of the input image to retrieve global surface features and fine-grained local features from hierarchical concept banks; then, an LLM-based Prompt Generator produces diverse, CLIP-friendly natural-language candidate prompts under structured constraints; next, an LLM Prompt Evaluator scores each candidate in terms of cue faithfulness, CLIP suitability, and clarity/readability, selecting the optimized prompt; finally, the selected prompt is encoded together with learnable normal/abnormal state vectors for anomaly scoring and localization. In this way, AnomalyPrompt enables image-specific prompt adaptation while preserving the semantic transparency of natural-language prompts. Extensive experiments on seven industrial anomaly detection datasets and four medical benchmarks demonstrate that AnomalyPrompt achieves SOTA performance in zero-shot anomaly detection.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Anomaly Detection, Zero-Shot, Prompt Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EpWNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AnomalyPrompt, an LLM-guided adaptive prompting framework for zero-shot anomaly detection. The method retrieves global and local visual cues from hierarchical concept banks, uses an LLM to generate image-conditioned candidate prompts, and further employs an LLM-based evaluator to select the final prompt for image-level anomaly detection and pixel-level localization. The framework is clearly organized and achieves strong overall performance across multiple industrial and medical anomaly detection benchmarks. However, I still have concerns regarding the methodological novelty, as several components are closely related to existing work on retrieval, dynamic prompting, and LLM-based anomaly reasoning. In addition, the contribution of fine-grained semantic descriptions is not sufficiently isolated, and more targeted ablation studies would help clarify how different semantic cues affect the final performance.

Critical strengths

(1) Well-motivated framework. The paper addresses the limitation of static/passively optimized prompts and introduces image-conditioned prompt generation through global and local visual cues. (2) Coherent and interpretable design. The perception–reasoning–decision–execution pipeline is intuitive, and the generated natural-language anchors provide better semantic interpretability than purely learnable prompt tokens. (3) Strong empirical results. The method achieves strong overall performance across seven industrial datasets and further demonstrates cross-domain generalization on four medical benchmarks. (4) Relatively comprehensive ablations. The paper analyzes visual retrieval, global/local cues, prompt evaluation, candidate number, and different LLM scales.

Critical weaknesses

(1)The novelty is relatively conventional. The overall framework largely combines several existing ideas, including dynamic prompting, fine-grained textual descriptions, retrieval mechanisms, and LLM-based reasoning. For example, MRAD already explores retrieval-based anomaly detection, while AnomalyGPT investigates the use of large language/vision-language models for anomaly reasoning. (2)The effectiveness of the LLM Prompt Evaluator is not sufficiently justified. The same LLM is used for both candidate prompt generation and evaluation, while the evaluation criteria mainly focus on linguistic quality and consistency with visual cues rather than directly measuring actual anomaly discrimination capability. In addition, the evaluator is only compared against random selection, which is a relatively weak baseline. (3)The statistical robustness is unclear. The method uses stochastic LLM decoding, but the paper does not report mean and standard deviation over multiple random seeds. Therefore, it is difficult to assess whether some of the reported performance improvements are statistically reliable. (4)The test-time computational overhead is relatively high. For each test image, the framework performs both LLM-based candidate prompt generation and prompt evaluation. However, the paper does not provide direct comparisons with existing methods in terms of inference speed, latency, or throughput. (5)The performance may depend heavily on the manually designed or LLM-generated Concept Bank. The current paper does not sufficiently analyze the effects of the Concept Bank size, semantic coverage, and bank quality on the final performance. More fine-grained ablation studies would be helpful to clarify this dependency. (6)The SOTA claim is not fully accurate. Although the proposed method achieves strong overall average performance, it does not obtain the best result on every dataset or metric. For example, on MVTec-AD, some competing methods outperform AnomalyPrompt on several image-level and pixel-level metrics.

Questions for authors

(1)Novelty: Could the authors clarify the key methodological novelty of AnomalyPrompt compared with prior work on retrieval-based anomaly detection (e.g., MRAD), dynamic/fine-grained prompting, and LLM-based anomaly reasoning (e.g., AnomalyGPT)? In particular, which part of the proposed framework provides a fundamentally new contribution beyond combining these existing components? (2)Prompt Evaluator: The same LLM is used for both candidate prompt generation and prompt evaluation. Could the authors provide additional justification or evidence that the evaluator score is well correlated with downstream anomaly detection performance? Comparisons with stronger selection strategies, such as CLIP-similarity-based selection, an independent evaluator, or oracle prompt selection, would help clarify the effectiveness of this module. (3)Fine-grained semantic cues / Concept Bank: How sensitive is the performance to the design and semantic coverage of the Concept Bank? In particular, could the authors provide more fine-grained ablations on different types of semantic cues, such as global vs. local descriptions, defect-related vs. neutral concepts, or different Concept Bank sizes? This would help determine whether the gains mainly come from fine-grained semantic descriptions or from the overall prompting pipeline. (4)Statistical robustness: Since the LLM generation process uses stochastic decoding, could the authors report results over multiple random seeds with mean and standard deviation? This is particularly important because some of the improvements over prior methods are relatively small. (5)Inference efficiency: Could the authors provide a direct comparison of test-time latency or throughput with representative competing methods? Since both prompt generation and prompt evaluation are performed for each test image, this information would be important for assessing the practical applicability of the method in industrial inspection settings.

Reviewer_kKF9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AnomalyPrompt for zero-shot anomaly detection. The method retrieves global and local visual concepts using CLIP. An LLM generates candidate prompts and evaluates their quality. The selected prompt is integrated with learnable normal/abnormal representations for anomaly detection and localization. Experiments on industrial and medical datasets show competitive results.

Critical strengths
  1. Reasonable motivation and framework. The paper addresses the limitations of static prompts in capturing diverse visual anomalies.

  2. Competitive experimental results. The method achieves 93.2% mean pixel-level PRO across seven industrial datasets, compared with the strongest reported baseline.

Critical weaknesses
  1. Limited novelty and missing closely related work. ALFA [1] already employs LLM-generated anomaly prompts and per-image contextual scoring. FAPrompt [2] introduces sample-wise abnormality adaptation, while FiLo++ [3] uses runtime prompt filtering. These closely related methods are missing from the comparisons.

  2. Unclear contributions beyond existing prompt-learning frameworks. Section 3.5 retains learnable context vectors, normal/abnormal state vectors, and a prompt bank, closely resembling Bayes-PFL [4]. Table 3 shows a 16% image-level AUROC drop without the prompt bank, but only a 2.2% drop without LLM optimization. This raises concerns about the actual contribution of the proposed modules.

  3. Potentially unfair baseline comparisons. Section 4.1 uses MVTec-AD for auxiliary training on most target datasets. However, Bayes-PFL [4] uses VisA for most non-VisA targets. This difference affects comparisons on BTAD, KSDD2, RSDD, and DTD-Synthetic.

  4. Insufficient ablation of the core LLM modules. Table 3 removes the Generator and Evaluator simultaneously, preventing independent assessment. The complete LLM optimization improves VisA pixel AUROC by only 0.2% over fixed templates. Figure 4 compares evaluator-based selection only against random selection. Stronger baselines should include CLIP similarity ranking, heuristic scoring, and ALFA-style contextual selection [1].

References:

[1] Zhu et al. Do LLMs Understand Visual Anomalies? Uncovering LLM's Capabilities in Zero-shot Anomaly Detection. ACM MM, 2024.

[2] Zhu et al. Fine-grained Abnormality Prompt Learning for Zero-shot Anomaly Detection. ICCV, 2025.

[3] Gu et al. FiLo++: Zero-/Few-Shot Anomaly Detection by Fused Fine-Grained Descriptions and Deformable Localization. TCSVT 2026.

[4] Qu et al. Bayesian Prompt Flow Learning for Zero-Shot Anomaly Detection. CVPR, 2025.

Questions for authors

Q1. What distinguishes the proposed LLM Evaluator from ALFA's contextual scoring and FiLo++'s prompt filtering? Can you provide direct comparisons with ALFA and FAPrompt?

Q2. Were Bayes-PFL and MRAD retrained under the same auxiliary-training protocol? Can you report controlled comparisons using identical training datasets and preprocessing?

Q3. What are the independent contributions of the Generator and Evaluator? Can you compare LLM-based selection with CLIP similarity ranking and ablate the three evaluation criteria separately?

Q4. Do higher LLM scores correlate with better anomaly detection performance? How does the method handle incorrect retrieved concepts and defect-related prompts generated for normal images?

Reviewer_TzC2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an LLM-guided adaptive prompting framework, AnomalyPrompt, for zero-shot anomaly detection. It first retrieves global and local visual concepts using CLIP image-level and patch-level features. An LLM then generates multiple image-conditioned prompt candidates, and a prompt evaluator selects the most suitable one based on visual faithfulness, CLIP suitability, and readability. The selected prompt is integrated with learnable normal and abnormal state vectors for image-level anomaly detection and pixel-level localization. Experiments on seven industrial datasets and four medical benchmarks demonstrate that AnomalyPrompt achieves competitive performance against existing zero-shot anomaly detection methods.

Critical strengths

1.Combining global and local visual concept retrieval with LLM-based prompt generation allows the text representation to adapt to individual images rather than relying entirely on fixed class descriptions.

2.The experimental evaluation is relatively comprehensive, covering seven industrial datasets and four medical benchmarks. The method achieves competitive results against recent CLIP-based anomaly detection approaches.

Critical weaknesses

1.Table 3 removes the Prompt Generator and Prompt Evaluator together. It is difficult to determine how much each component contributes. Although Figure 4 shows that evaluator-based selection outperforms random selection, it remains unclear whether the LLM evaluator offers advantages over simpler alternatives, such as CLIP similarity-based selection.

2.The proposed framework relies on an LLM-based Prompt Evaluator to select the best candidate from multiple generated prompts. However, the evaluation only compares this strategy against random selection, without considering simpler yet potentially competitive alternatives, such as CLIP similarity-based selection or prompt embedding aggregation. Moreover, selecting a single prompt may discard complementary visual information captured by other candidates.

3.In Section 3.5, the same image-conditioned prompt is incorporated into both the normal and abnormal text representations, while their distinction is maintained through learnable state vectors. Do the generated prompts improve anomaly-specific discrimination or primarily provide better image-conditioned class descriptions?

4.In Table 3, the full method improves image-level AUROC on VisA from 84.9% to 87.1% compared with the fixed-template variant, while pixel-level AUROC increases only from 95.7% to 95.9%. The author should provide a direct comparison of runtime against the fixed-template variant to clarify whether the additional per-image LLM generation and evaluation are worthwhile, particularly for anomaly localization.

5.The experimental comparisons omit several relevant zero-shot anomaly detection methods, particularly FiLo[1] and FAPrompt[2]. FiLo leverages LLM-generated fine-grained anomaly descriptions and learnable text prompts, while FAPrompt explores fine-grained abnormality prompt learning and image-specific prompt adaptation. Both are closely related to the proposed framework in terms of prompt construction and adaptation.

[1] FiLo: Zero-Shot Anomaly Detection by Fine-Grained Description and High-Quality Localization [2] Fine-grained Abnormality Prompt Learning for Zero-shot Anomaly Detection

Questions for authors

See comments above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.