- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether a multimodal large language model (MLLM) can directly perform zero-shot anomaly segmentation and anomaly description without relying on external visual models or dedicated mask decoders. The authors identify two key representation bottlenecks: weak within-region semantic relationships in the vision encoder and limited diversity of image-dependent contextual [SEG] representations. To address these issues, they propose MIAS, which directly matches the MLLM’s visual features with a contextual [SEG] query to produce anomaly masks, enabling unified anomaly understanding, description, and localization.
The main contributions are two targeted training strategies. Patch Affinity Distillation (PAD) transfers pairwise patch relationships from a frozen visual foundation model to improve region-level visual coherence, while Complementary Prompt Supervision (CPS) constructs complementary localization tasks from existing anomaly masks to encourage more diverse image-conditioned [SEG] representations. Both strategies are used only during training, allowing MIAS to perform anomaly description and segmentation with a single MLLM at inference, while achieving competitive zero-shot segmentation performance across eight industrial and medical datasets.
Critical strengths
-
The paper addresses a clear and relevant problem by investigating whether a single MLLM can directly perform zero-shot anomaly segmentation without relying on an external mask decoder. The formulation is well motivated, and the paper explicitly analyzes both the visual representation and contextual [SEG] query as two potential bottlenecks.
-
The proposed framework is lightweight at inference time. Both PAD and CPS are used only during training, while inference relies only on the MLLM’s own visual features and contextual segmentation query, avoiding additional visual teachers or dedicated segmentation decoders.
Critical weaknesses
-
The current experiments demonstrate the effectiveness of DINOv2-based affinity supervision, but do not sufficiently isolate whether the improvement specifically results from correcting weak within-region semantics, rather than from more general visual structural regularization.
-
The validation across different MLLM backbones remains limited. The current backbone comparison is conducted only on MVTec-AD, and although InternVL3.5-2B, InternVL3-2B, and Qwen3.5-2B all achieve competitive results, this is insufficient to establish that MIAS generalizes consistently across model architectures and data distributions. The gains from PAD and CPS are noticeably weaker on Qwen3.5-2B, and the full PAD+CPS setting improves AUROC over the baseline by only 0.2 points, while PAD alone achieves a higher AUROC. I would recommend evaluating more datasets across multiple backbones and further analyzing the backbone sensitivity of PAD and CPS.
-
While CPS substantially increases the diversity of [SEG] representations, the causal connection between this change and the reported segmentation gains is not sufficiently demonstrated. The current analysis mainly contrasts full CPS with repeated supervision, but does not examine whether progressively adding complementary objectives leads to consistent increases in both query diversity and localization performance.
Questions for authors
-
Can the authors provide more direct evidence that the increased diversity of contextual [SEG] representations is actually associated with the segmentation gains?
-
How stable are the contributions of PAD and CPS across different MLLM backbones and datasets?