acceptodds
Withdrawn from ICLR 2027

Exploring the Potential of MLLMs for Zero-Shot Anomaly Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing multimodal large language model (MLLM)-based methods for zero-shot anomaly segmentation (ZSAS) rely on dedicated segmentation components, leaving open whether a single MLLM can directly segment and describe anomalies. To investigate this question, we empirically examine two potential bottlenecks. First, at the vision encoder level, pretrained visual features exhibit weaker semantic relationships within regions than those of visual foundation models, potentially hindering anomaly localization that requires semantically coherent representations. Second, at the LLM level, contextual [SEG] representations exhibit limited diversity in their variation across input images under repetitive mask supervision, potentially restricting adaptation to different anomaly characteristics. Based on these findings, we propose MIAS (MLLM Itself for Anomaly Segmentation), an MLLM-only framework with two training strategies that address the respective bottlenecks. To strengthen semantic relationships in the vision encoder's features, Patch Affinity Distillation (PAD) transfers pairwise patch affinities from a frozen visual foundation teacher. To encourage more diverse image-conditioned [SEG] representations in the LLM, Complementary Prompt Supervision (CPS) constructs complementary localization tasks from existing anomaly masks without additional manual annotations. Both strategies operate only during training. At inference, MIAS uses a single MLLM to generate anomaly descriptions and predict masks by directly matching contextual segmentation queries with visual features, requiring neither external visual models nor dedicated mask decoders. Experiments across eight industrial and medical datasets show that MIAS achieves the best average segmentation performance among the compared methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
anomaly segmentation, multimodal large language model, zero-shot

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VZo1OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether a multimodal large language model (MLLM) can directly perform zero-shot anomaly segmentation and anomaly description without relying on external visual models or dedicated mask decoders. The authors identify two key representation bottlenecks: weak within-region semantic relationships in the vision encoder and limited diversity of image-dependent contextual [SEG] representations. To address these issues, they propose MIAS, which directly matches the MLLM’s visual features with a contextual [SEG] query to produce anomaly masks, enabling unified anomaly understanding, description, and localization.

The main contributions are two targeted training strategies. Patch Affinity Distillation (PAD) transfers pairwise patch relationships from a frozen visual foundation model to improve region-level visual coherence, while Complementary Prompt Supervision (CPS) constructs complementary localization tasks from existing anomaly masks to encourage more diverse image-conditioned [SEG] representations. Both strategies are used only during training, allowing MIAS to perform anomaly description and segmentation with a single MLLM at inference, while achieving competitive zero-shot segmentation performance across eight industrial and medical datasets.

Critical strengths
  1. The paper addresses a clear and relevant problem by investigating whether a single MLLM can directly perform zero-shot anomaly segmentation without relying on an external mask decoder. The formulation is well motivated, and the paper explicitly analyzes both the visual representation and contextual [SEG] query as two potential bottlenecks.

  2. The proposed framework is lightweight at inference time. Both PAD and CPS are used only during training, while inference relies only on the MLLM’s own visual features and contextual segmentation query, avoiding additional visual teachers or dedicated segmentation decoders.

Critical weaknesses
  1. The current experiments demonstrate the effectiveness of DINOv2-based affinity supervision, but do not sufficiently isolate whether the improvement specifically results from correcting weak within-region semantics, rather than from more general visual structural regularization.

  2. The validation across different MLLM backbones remains limited. The current backbone comparison is conducted only on MVTec-AD, and although InternVL3.5-2B, InternVL3-2B, and Qwen3.5-2B all achieve competitive results, this is insufficient to establish that MIAS generalizes consistently across model architectures and data distributions. The gains from PAD and CPS are noticeably weaker on Qwen3.5-2B, and the full PAD+CPS setting improves AUROC over the baseline by only 0.2 points, while PAD alone achieves a higher AUROC. I would recommend evaluating more datasets across multiple backbones and further analyzing the backbone sensitivity of PAD and CPS.

  3. While CPS substantially increases the diversity of [SEG] representations, the causal connection between this change and the reported segmentation gains is not sufficiently demonstrated. The current analysis mainly contrasts full CPS with repeated supervision, but does not examine whether progressively adding complementary objectives leads to consistent increases in both query diversity and localization performance.

Questions for authors
  1. Can the authors provide more direct evidence that the increased diversity of contextual [SEG] representations is actually associated with the segmentation gains?

  2. How stable are the contributions of PAD and CPS across different MLLM backbones and datasets?

Reviewer_3YBDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors investigate the capabilities of a 2B multimodal LLM for anomaly segmentation with the purpose of industrial quality control and medical imaging. They analyze the shortcomings of existing MLLM for the anomaly segmentation task and find that on the one hand the semantic encodings of the LMM internal ViT are less consistent compared with thorse encodings provided by a state of the art vision foundation model like Dinov2. Secondly, the internal features of the segmentation instruction token are very static under variation of images, which prevents effective segmentation.

Here the autors propose a fine tunig strategy that on the one hand uses Dinov2 as a teacher model and on the other hand existing segmentation maps. After fine tuning, the MLLM performs the zero shot anomaly segmentation task without further auxiliary modules.

In their experiments, the authors find that their method shows consistent but minor improvements over the state of the art in zero shot anomaly segmentation on a number of benchmarks in industrial and Medical applications (Table 1). Nevertheless, the performance still is weaker than supervised methods, which dominate on the anomaly detection task.

The authors perform an analysis of their phenomena and evaluate their initial hypotheses.

Critical strengths
  • The performance of MLLM is investigated in some details and the methods applied to improve its behavior is thoroughly investigated beyond pure ablation studies.
Critical weaknesses
  • The stud is resticed to the class of up to 3.5B MLLM models and it is not clear, whether the findings also generalize to more powerful models.
  • The obtained performance for zero shot anomaly segmentation is impressive, but still rather far away from precision required in industrial or even medical applications. Therefore, ZSAD seems to be a rather academic endeavor that mostly compares to itself. This is not saying that ZSAD is not interesting at all and certainly the field has made some progress. But if it proceeds with the present speed of marginal developments per paper minimally outperforming previous baselines, we have to reserve quite some space in conferences before something applicable in practice comes out. As far as I am aware, supervised methods still clearly outperform ZSAD by a clear margin and are already mature for industrial and to some degree medical use. For ZSAD to catch up, larger steps are required than presented in the present paper.
Questions for authors
  • Is there a case where your method outperforms supervised methods?
Reviewer_jGBmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper considers zero-shot anomaly segmentation, focusing specifically on improving Multi-Modal Large language models MLLMs for this task. The main finding is that MLLMs perform worse than specialized vision foundation models as their visual representations have weaker semantic relationships within connected regions and limited diversity in textual representations across images.

The proposed approach, MLLM Itself for Anomaly Segmentation (MIAS), addresses both of the identified weaknesses. The vision encoder is fin-tuned with LoRA to produce token affinity like a DINOv2-L model in a student-teacher setup. Its visual tokens are projected with a trainable MLP projector and passed into the intern LLM. A prompt-mask pair is created by a Complementary Prompt Supervision (CPS) block. CPS enriches supervision by generating multiple possible prompt mask pairs from a single image (e.g. anomaly region for the prompt "locate anomaly" or the rest of the image as the mask for the prompt "locate normal region"). The intern LLM is fine-tuned with LoRA to produce a contextual [SEG] token based on projected visual tokens and the given prompt. Cosine similarity between visual features and the [SEG] token is used to produce anomaly segmentation masks. The loss is based on the difference between the predicted and the ground truth mask for multiple possible prompt-mask pairs given by CPS.

Critical strengths
  1. Anomaly segmentation is an important safety topic. Enabling MLLMs for this task could enable textual descriptions of anomalies which would help interpretability of results.
  2. Good quantitative results. Evaluation also considers efficiency, with the proposed approach being relatively fast compared to alternatives. Ablation experiments confirm the value visual encoder alignment and COS. The method works with multiple MLLM backbones.
Critical weaknesses
  1. The method requires access to vision foundation models during the training stage, as well as densely labelled anomaly images.
  2. There is no analysis of textual descriptions, so the advantage of MLLMs over purely vision models is not clear at the moment.
  3. CPS analysis focuses on examining spectrum effects of CPS. Simpler approaches, like looking at the effect of using only a subset of prompt-mask pairs might provide more intuitive results (e.g. do "normal" prompts continue working if only "anomaly" prompts are used, and vice-versa, would background segmentation improve overall performance, are both context and core important etc. )
  4. Moving the experiments i which examine the effect of affinity teacher appendix to the main paper would strengthen the contribution (shows flexibilitiy of the proposed approach, examines the impact of teacher/backbone/pretraining on the overall performance).
Questions for authors
  1. Should the mask from the CPS prompt be connected to the ground truth segmentation in Figure 3?
  2. In table 5, what are the baseline MLLM performances?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.