acceptodds
Under review as a conference paper at ICLR 2027

Is In-Domain Training Enough for Fine-Grained Industrial Anomaly Understanding?

Abstract

A single multimodal large language model (MLLM) struggles to excel simultaneously at detection, localization, description, and reasoning in multimodal industrial anomaly understanding (MM-IAU). We show that in-domain training does not close this gap. On MMAD, a widely adopted MM-IAU benchmark, trained specialists reach at most 75.5% accuracy in defect localization, against 92.3% for human experts, and even detect anomalies less accurately than their untrained base model. Meanwhile, different MLLMs offer complementary strengths but share this weakness in fine-grained perception, so combining them alone cannot remove it. We therefore propose SiGMA, a spatially grounded multi-agent framework that divides labor between heterogeneous MLLM agents and a dedicated visual defect expert. A multimodal searcher supplies industrial knowledge and normal references, the defect expert turns query–reference comparison into calibrated anomaly evidence, and a label-free reliability controller weighs each source by task-wise competence and query-level evidence quality. SiGMA reaches 85.2% average accuracy on MMAD, 4.0% above the strongest trained specialist and Gemini-2.5-Pro and within 1.5% of human experts. Even with three agents of at most 9B parameters, it reaches 84.4%, and new MLLMs join without retraining.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Industrial anomaly understanding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uBwhOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether in-domain training is sufficient for fine-grained multimodal industrial anomaly understanding (MM-IAU). The authors argue that supervised specialization of a single MLLM improves certain semantic capabilities but does not solve shared fine-grained perceptual weaknesses. They analyze MMAD and show that trained specialists still lag behind human experts in defect localization and anomaly detection, motivating a division-of-labor strategy. The authors propose SiGMA, a training-free spatially grounded multi-agent framework that combines heterogeneous frozen MLLM agents with a reference-based defect expert. The framework includes a multimodal searcher for retrieving domain knowledge and normal references, a defect expert for producing calibrated anomaly evidence, and a reliability controller for adaptive aggregation based on source competence and query-level evidence quality. Experiments on MMAD show that SiGMA achieves 85.2% average accuracy, outperforming existing trained specialists and large-scale MLLMs.

Critical strengths

Instead of only proposing another model, the paper identifies an important bottleneck: fine-grained industrial perception remains weak even after domain training. The analysis showing trained methods still underperform base models on anomaly detection is interesting and potentially valuable.

The proposed combination of heterogeneous MLLMs with a dedicated defect expert is conceptually reasonable. Ablations show that the defect expert mainly contributes to anomaly detection and localization, while MLLM diversity improves semantic tasks.

Critical weaknesses
  1. The paper argues that in-domain training is insufficient for fine-grained industrial anomaly understanding, but the conclusion is drawn from one benchmark. A stronger claim requires broader experimental evidence.

  2. Although the authors report that a three-agent system reaches 84.4% accuracy, SiGMA still requires multiple MLLM calls and external visual expert processing. 57640_Is_In_Domain_Training_En The practical advantage over a carefully designed single-model system is not fully established.

Questions for authors

How does SiGMA perform when the reliability controller cannot access the entire unlabeled test batch and must operate in an online streaming scenario?

Have the authors evaluated whether the estimated source reliability transfers across different industrial domains or product categories?

How sensitive is the performance to the choice of the defect expert (e.g., DINOv3/RAD)? Would replacing it with another anomaly detector substantially change the conclusions?

Compared with simply using a strong vision anomaly detector plus an LLM-based explanation module, what is the unique advantage of the proposed multi-agent aggregation strategy?

5 comments
AuthorsOpenReview ↗

Response to Reviewer uBwh (1/4)

W1: The conclusion is drawn from one benchmark.

A: We thank the reviewer for this important point. To verify that our conclusion holds beyond MMAD, we froze the agent team and all hyper-parameters and evaluated SiGMA on MANTA [1], an additional benchmark of tiny objects from five domains, each captured from five viewpoints, which was never used in any design choice. Without any training or tuning, SiGMA outperforms all reported methods, improving over the strongest trained specialist SAGE [2] by 4.6, while the trained JUDO [3] even falls below its own untrained base model. A plausible reason is that SFT and RL fit the model weights to the training distribution, which limits their generalization under distribution shift, whereas SiGMA keeps all MLLMs frozen and re-estimates source reliabilities label-free on each new benchmark. This supports the main finding of our paper beyond MMAD. Moreover, on MMAD itself, SiGMA is insensitive to all key hyper-parameters, with performance varying by at most 0.5. This indicates that SiGMA's performance stems from the framework itself without the need for careful hyper-parameter tuning on MMAD.

(a) Additional unseen benchmark. MANTA [1] focuses on tiny objects, such as grains, pills and small parts, from five domains (agriculture, electronics, groceries, mechanics, medicine). Each image stitches five viewpoints, and five binary items are asked about each object. Besides item accuracy over all 10,000 items, we report the stricter all-five-correct rate, where all five items about an object must be answered correctly. We froze the team, the readers, all hyper-parameters and the decoding budgets, and no label was read before scoring.

SystemMANTA item acc. (%)All five correct (%)
LLaVA-1.5 [1]56.83.7
LLaVA-NeXT [1]57.43.4
InternVL2 [1]53.31.1
Molmo [1]54.14.5
BLIP-2 + LoRA [1]66.58.7
LLaVA-1.6 [2]49.04.5
Qwen2-VL [2]54.35.3
DeepSeek-VL2 [2]50.4–
GPT-4o [2]63.73.9
InternVL2 (SFT) [2]71.3–
SAGE (trained specialist) [2]73.523.8
Qwen3.5-9B67.020.4
Qwen2.5-VL-7B, same context73.131.7
JUDO, same context70.030.2
JUDO + same defect-expert evidence71.831.9
SiGMA without defect expert77.537.8
SiGMA (full)78.137.3

As shown in the table, SiGMA improves over SAGE [2] by 4.6, over the fine-tuned InternVL2 by 6.8, and over GPT-4o by 14.4. The gain is even larger under the all-five-correct metric, where SiGMA improves over SAGE by 13.5. In contrast, in-domain training does not transfer: given the same context, JUDO falls 3.1 below its own untrained base model, Qwen2.5-VL-7B, and remains below it in item accuracy even with the same defect evidence.

(b) Sensitivity on MMAD. On a stratified subset of 1,024 images (4,981 questions) with the full 8 base agents and 3 readers, we vary each key hyper-parameter while fixing the others at their defaults.

Hyper-parameterRoleDefaultValues testedMMAD avg. (%)Max change
βevidence weighting, ρ = 1 + βt80, 2, 4, 8, 1683.8 – 84.30.5
wglobal vs. task-wise reliability0.350, 0.175, 0.35, 0.7, 184.3 – 84.40.05
p_lownoise quantile for map calibration0.990.95, 0.975, 0.99, 0.99584.3 – 84.40.05
δmargin of the spatial gate0.010, 0.005, 0.01, 0.02, 0.03, 0.0584.1 – 84.30.2

As shown in the table, every value stays within 0.5 of the default. The largest change comes from switching off evidence weighting entirely (β = 0), and w and p_low change the result by at most 0.05. The reported results therefore do not rely on finely tuned hyper-parameters. We will report these results in the camera-ready version (CR).

[1] MANTA: A Large-Scale Multi-View and Visual-Text Anomaly Detection Dataset for Tiny Objects. CVPR 2025.

[2] SAGE: A Visual Language Model for Anomaly Detection via Fact Enhancement and Entropy-aware Alignment. ACM MM 2025.

[3] JUDO: A Juxtaposed Domain-Oriented Multimodal Reasoner for Industrial Anomaly QA. ICLR 2026.

AuthorsOpenReview ↗

Response to Reviewer uBwh (2/4)

W2: The practical advantage over a carefully designed single-model system is not fully established.

A: We thank the reviewer for this fair comment. We provide evidence that SiGMA outperforms carefully designed single-model systems in two aspects, namely at a matched model size and in generalization. First, at a similar total scale, single large MLLMs of 72B/76B given domain knowledge still fall at least 5.0 below SiGMA on MMAD. Second, the strongest trained specialist, JUDO [3], given exactly the same retrieved context and defect-expert evidence as SiGMA, still falls 2.9 below SiGMA on MMAD. Moreover, its training does not generalize, as on the unseen MANTA [1] it falls below its own untrained base model and 6.3 below SiGMA.

(a) Single models of similar scale. The 9 checkpoints of SiGMA total about 70B parameters. We therefore compare with single large MLLMs of similar size on MMAD, each given domain knowledge.

ModelParamsAvg.(%)ADCls.Loc.
InternVL2-Llama3-76B76B76.266.771.060.6
Qwen2.5-VL-72B-Instruct72B80.271.673.969.1
SiGMA~70B (9 × ≤9B)85.282.679.279.6

As shown in the table, at a similar total size SiGMA outperforms the strongest 72B model by 5.0 on average, by 11.0 in anomaly detection and by 10.5 in defect localization. The advantage therefore does not come from more parameters, but from dividing the labor among heterogeneous agents and a defect expert.

(b) A carefully designed trained specialist. We gave JUDO, the strongest trained specialist in Table 1, exactly the information SiGMA uses: first the same retrieved context, and then additionally the same defect-expert overlay and spatial hint.

SettingMMAD avg. (official, %)MANTA item acc. (%)
Qwen2.5-VL-7B (JUDO's base model), same retrieved context76.973.1
JUDO, same retrieved context81.870.0
JUDO + same defect-expert evidence82.371.8
SiGMA85.278.1

As shown in the table, the same defect evidence improves JUDO by only 0.5, and SiGMA still outperforms it by 2.9 on MMAD. On the unseen MANTA, the gap widens to 6.3, and JUDO falls 3.1 below its own untrained base model, so the benefit of its in-domain training does not transfer (W1). We will add these comparisons to the CR.

[1] MANTA: A Large-Scale Multi-View and Visual-Text Anomaly Detection Dataset for Tiny Objects. CVPR 2025.

[2] SAGE: A Visual Language Model for Anomaly Detection via Fact Enhancement and Entropy-aware Alignment. ACM MM 2025.

[3] JUDO: A Juxtaposed Domain-Oriented Multimodal Reasoner for Industrial Anomaly QA. ICLR 2026.

AuthorsOpenReview ↗

Response to Reviewer uBwh (3/4)

Q1: How does SiGMA perform when the controller cannot access the full unlabeled test batch and must operate online?

A: We thank the reviewer for this practical question. SiGMA operates effectively online without access to the full test batch. Its controller uses unlabeled test data only to estimate the task-wise reliability of each source and the distribution of anomaly-map strength, and both can equally be estimated from previously seen images. Specifically, when MMAD is replayed as an online stream, with each image answered using statistics from the most recent 512 images and re-estimated every 256 images, accuracy decreases by only 0.2 compared with the full batch, averaged over 10 arrival orders. When the statistics are estimated once from the first 512 images and then frozen, the decrease is only 0.1.

What the controller estimates. Without any label, the controller estimates two statistics from unlabeled test data: (i) the task-wise reliability of each source, and (ii) the distribution of anomaly-map strength, which turns each map into its evidence score t (the within-task percentile). In the paper, both are estimated from the full test batch.

Online protocol. We replayed MMAD as a stream of images, using the frozen model outputs. When an image arrives, all its questions are answered with statistics from previous images only, and only then is the image added to the history, so no question influences its own weights. The statistics use the last w images and are updated every u images. For the first images, before any history exists, the controller uses uniform voting, and these images are included in the score. We tested a random image order and a product-blocked order, in which all images of one product arrive before the next product starts, so each new product begins with statistics from other products only.

ProtocolMMAD avg. (%)Δ vs. batch
Full unlabeled batch (paper)85.2—
Streaming, random order, w = 512, u = 256 (mean of 10 orders, SD < 0.1)85.0−0.2
Streaming, product-blocked order (mean of 10 orders, SD < 0.1)85.0−0.2
Statistics frozen after the first 512 images (mean of 3 seeds; scored on the remaining images)85.1−0.1
Range over all 12 (w, u) settings, w ∈ {128,…,1024}, u ∈ {64, 256, 512}84.9 – 85.1−0.3 – −0.1

As shown in the table, the change is −0.2 in both orders (95% bootstrap CI over images: [−0.3, −0.1]), and it stays between −0.3 and −0.1 across all 12 choices of w and u. SiGMA-Agent (see our response to Reviewer 787k, W1 & Q1) likewise estimates reliabilities online, from previously answered questions only.

Q2: Does the estimated source reliability transfer across industrial domains or product categories?

A: We thank the reviewer for this insightful question. The estimated reliability transfers across both industrial domains and product categories. MMAD itself spans 4 industrial sources and 38 product categories, from industrial parts and electronic boards to food and retail goods, which allows us to test this transfer directly. Specifically, we hold out one source (e.g., VisA) or one product (e.g., bottle) at a time, estimate the controller statistics only on the remaining data, and apply them unchanged to the held-out one, so the target is never seen during estimation. The overall accuracy remains 85.2 and the accuracy on each held-out source changes by at most 0.1.

Protocol. We hold out targets in two ways. In leave-one-source-out, each of the four sources is held out in turn, and the controller statistics of Q1 (source reliabilities and the evidence-score distribution) are estimated on the other three sources only. In leave-one-product-out, each product is held out in turn and the statistics are estimated on all other products. In both cases, the statistics are frozen and applied to the held-out target without any update. Since DS-MVTec reuses MVTec-AD images, the two are always held out together.

Where the statistics are estimatedMMAD avg. (%)
All of MMAD, including the target (paper)85.2
Other sources only (leave-one-source-out)85.2
Other products only (leave-one-product-out)85.2

The table below shows the leave-one-source-out result for each held-out source.

Held-out sourceStatistics from the other sources (%)Statistics from all of MMAD (%)
MVTec-AD93.893.8
MVTec-LOCO80.380.2
VisA82.682.6
GoodsAD84.084.1

As shown in the tables, estimating the statistics without the target leaves the overall accuracy unchanged, and changes the accuracy on each held-out source by at most 0.1. Thus, the statistics can be estimated once on existing data and reused for new domains or products.

Reviewer_i54zOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether in-domain training is sufficient for fine-grained industrial anomaly understanding and proposes SiGMA, a training-free multi-agent framework. The method combines heterogeneous frozen MLLMs with a dedicated visual defect expert that uses query–reference feature matching to generate calibrated anomaly maps. A multimodal searcher provides domain knowledge and normal references, while expert reader agents interpret the visual evidence. A reliability controller then combines the predictions based on task-wise source competence and query-level evidence quality. SiGMA is evaluated on MMAD, achieving an average accuracy of 85.2%.

Critical strengths

The division of labor between heterogeneous MLLMs and a dedicated visual defect expert makes sense for industrial anomaly understanding. The results on MMAD are encouraging, particularly for anomaly detection and defect localization. Table 2 provides useful evidence that the MLLM agents and defect expert contribute to different tasks. The calibration experiments are also convincing, reducing the false-positive rate from 24.2% with raw reader maps to 9.4% with calibration and gating.

Critical weaknesses

W1. The reported configuration was selected using the same MMAD answers used for evaluation. Sec. 4.1 states that the agent team and hyperparameters were selected on MMAD, including the calibration and reliability-controller settings. The expert-reader team was also selected based on benchmark results. Although the authors report 84.9% using all ten base agents, this does not address the other configuration choices made using evaluation labels. The controller itself does not require labels once configured, but the reported 85.2% still lacks an independent evaluation with these choices fixed beforehand.

W2. The experiments do not isolate the effect of in-domain training from the additional models and visual evidence available to SiGMA. The proposed framework combines multiple MLLMs with a dedicated defect expert, whereas the trained baselines in Table 1 are primarily single-MLLM approaches. Table 2 shows that the different components of SiGMA contribute to performance, but it does not establish whether a trained MLLM with access to the same calibrated defect evidence would perform similarly. The results support the effectiveness of SiGMA, but I think the broader conclusion about the limitations of in-domain training is not fully supported by these comparisons.

W3. The query-level evidence weighting is not fully validated. In Sec. 3.5, the controller increases the influence of expert readers based on the strength or spatial extent of the anomaly response. However, the paper does not establish whether higher evidence scores identify queries where the expert readers are more reliable than the base agents. Sec. 4.4 shows that adding query-level weighting improves average accuracy from 84.8% to 85.2%, with a larger gain in anomaly detection. Still, it is unclear whether these scores indicate when the expert readers should actually be trusted more, particularly when their predictions disagree with the base agents.

Overall, the component ablations support the benefit of combining heterogeneous MLLMs with calibrated defect evidence. My main reservations are the use of evaluation answers for configuration selection and the lack of a controlled comparison supporting the broader conclusion about in-domain training.

Questions for authors

Q1. Can the authors evaluate SiGMA on genuinely unseen data, with the agent team, reader team, and hyperparameters fixed beforehand? This would help determine how much of the reported improvement generalizes beyond MMAD, which was already used for configuration selection.

Q2. How would a trained method such as JUDO or OmniAD perform within an otherwise identical SiGMA pipeline, using the same retrieved context, defect expert, and expert readers? This would help clarify whether the advantage comes from the multi-agent framework and additional visual evidence rather than the absence of in-domain training.

Q3. Have the authors examined how expert-reader accuracy compares with base-agent accuracy across different evidence-score ranges, particularly on queries where their predictions disagree?

3 comments
AuthorsOpenReview ↗

Response to Reviewer i54z (1/3)

W1 & Q1: The configuration was selected with MMAD answers; can SiGMA be evaluated on genuinely unseen data with the agent team, reader team and hyper-parameters fixed beforehand?

A: We thank the reviewer for this precise question. With the agent team, reader team and hyper-parameters fixed beforehand, SiGMA generalizes well to genuinely unseen data and does not rely on careful tuning on MMAD. Since SiGMA trains no model weights, its risk of overfitting lies only in the choice of hyper-parameters, and we verify its generalization in two ways. First, we froze the team and all hyper-parameters beforehand and evaluated on MANTA [1], an additional benchmark of tiny objects from five domains, each captured from five viewpoints, which was never used in any design choice. Without any training or tuning, SiGMA outperforms all reported methods, improving over the strongest trained specialist SAGE [2] by 4.6, while the trained JUDO [3] even falls below its own untrained base model. A plausible reason is that SFT and RL fit the model weights to the training distribution, which limits their generalization under distribution shift, whereas SiGMA keeps all MLLMs frozen and re-estimates source reliabilities label-free on each new benchmark. Second, on MMAD itself, SiGMA is insensitive to all key hyper-parameters, with performance varying by at most 0.5. Together, these results indicate that SiGMA's performance stems from the framework itself rather than from tuning on MMAD.

(a) Additional unseen benchmark. MANTA [1] focuses on tiny objects, such as grains, pills and small parts, from five domains (agriculture, electronics, groceries, mechanics, medicine). Each image stitches five viewpoints, and five binary items are asked about each object. Besides item accuracy over all 10,000 items, we report the stricter all-five-correct rate, where all five items about an object must be answered correctly. We froze the team, the readers, all hyper-parameters and the decoding budgets, and no label was read before scoring.

SystemMANTA item acc. (%)All five correct (%)
LLaVA-1.5 [1]56.83.7
LLaVA-NeXT [1]57.43.4
InternVL2 [1]53.31.1
Molmo [1]54.14.5
BLIP-2 + LoRA [1]66.58.7
LLaVA-1.6 [2]49.04.5
Qwen2-VL [2]54.35.3
DeepSeek-VL2 [2]50.4–
GPT-4o [2]63.73.9
InternVL2 (SFT) [2]71.3–
SAGE (trained specialist) [2]73.523.8
Qwen3.5-9B67.020.4
Qwen2.5-VL-7B, same context73.131.7
JUDO, same context70.030.2
JUDO + same defect-expert evidence71.831.9
SiGMA without defect expert77.537.8
SiGMA (full)78.137.3

As shown in the table, SiGMA improves over SAGE [2] by 4.6, over the fine-tuned InternVL2 by 6.8, and over GPT-4o by 14.4. The gain is even larger under the all-five-correct metric, where SiGMA improves over SAGE by 13.5. In contrast, in-domain training does not transfer: given the same context, JUDO falls 3.1 below its own untrained base model, Qwen2.5-VL-7B, and remains below it in item accuracy even with the same defect evidence.

(b) Sensitivity on MMAD. On a stratified subset of 1,024 images (4,981 questions) with the full 8 base agents and 3 readers, we vary each key hyper-parameter while fixing the others at their defaults.

Hyper-parameterRoleDefaultValues testedMMAD avg. (%)Max change
βevidence weighting, ρ = 1 + βt80, 2, 4, 8, 1683.8 – 84.30.5
wglobal vs. task-wise reliability0.350, 0.175, 0.35, 0.7, 184.3 – 84.40.05
p_lownoise quantile for map calibration0.990.95, 0.975, 0.99, 0.99584.3 – 84.40.05
δmargin of the spatial gate0.010, 0.005, 0.01, 0.02, 0.03, 0.0584.1 – 84.30.2

As shown in the table, every value stays within 0.5 of the default. The largest change comes from switching off evidence weighting entirely (β = 0), and w and p_low change the result by at most 0.05. The reported results therefore do not rely on finely tuned hyper-parameters. We will report these results in the camera-ready version (CR).

[1] MANTA: A Large-Scale Multi-View and Visual-Text Anomaly Detection Dataset for Tiny Objects. CVPR 2025.

[2] SAGE: A Visual Language Model for Anomaly Detection via Fact Enhancement and Entropy-aware Alignment. ACM MM 2025.

[3] JUDO: A Juxtaposed Domain-Oriented Multimodal Reasoner for Industrial Anomaly QA. ICLR 2026.

AuthorsOpenReview ↗

Response to Reviewer i54z (2/3)

W2 & Q2: How would a trained method such as JUDO or OmniAD perform within an otherwise identical SiGMA pipeline, with the same retrieved context, defect expert and readers?

A: We thank the reviewer for this insightful suggestion. Given exactly the same retrieved context and defect-expert evidence, only SiGMA makes full use of this additional evidence. The same evidence improves the trained JUDO [3] by only 0.5 on MMAD, whereas it improves SiGMA by 3.2 (Table 2 in our paper, row f vs. row e). As a result, JUDO still falls 2.9 below SiGMA on MMAD, and the gap widens to 6.3 on the unseen MANTA [1]. This indicates that the advantage of SiGMA comes from how the multi-agent framework uses the evidence, through expert readers weighted by evidence quality, rather than from the additional evidence alone.

Specifically, we used JUDO, the strongest trained specialist in Table 1, in two increasingly complete settings: (a) with the same retrieved context as SiGMA, and (b) additionally with the same defect-expert overlay and spatial hint, i.e., JUDO reads the same evidence as the expert readers.

SettingMMAD avg. (official, %)MANTA item acc. (%)
Qwen2.5-VL-7B (JUDO's base model), same retrieved context76.973.1
(a) JUDO, same retrieved context81.870.0
(b) JUDO + same defect-expert evidence82.371.8
SiGMA85.278.1

As shown in the table, in-domain training does not transfer to unseen data: on MANTA, JUDO falls 3.1 below its own untrained base model with the same context, and remains below it with the same defect evidence. These results support our main finding that in-domain training is not enough for fine-grained industrial anomaly understanding, and we will add them to the CR.

[1] MANTA: A Large-Scale Multi-View and Visual-Text Anomaly Detection Dataset for Tiny Objects. CVPR 2025.

[2] SAGE: A Visual Language Model for Anomaly Detection via Fact Enhancement and Entropy-aware Alignment. ACM MM 2025.

[3] JUDO: A Juxtaposed Domain-Oriented Multimodal Reasoner for Industrial Anomaly QA. ICLR 2026.

AuthorsOpenReview ↗

Response to Reviewer i54z (3/3)

W3 & Q3: Do higher evidence scores identify queries where the expert readers are more reliable than the base agents, particularly when they disagree?

A: We thank the reviewer for this valuable question. Higher evidence scores do identify the queries where expert readers are more reliable than the base agents, including when they disagree. Specifically, on defect localization, the reader's accuracy and its gain over the same model without the map both rise steadily with the evidence score. Moreover, when a reader disagrees with the base team, following the reader loses 135 questions at low scores but gains 630 questions at high scores, which is exactly the behavior that our evidence weighting implements. For anomaly detection, this selection is already made by the calibrated gate: readers vote only when it fires, and are then correct in 80–90% of disagreements.

Mechanism. Expert readers answer only spatial questions, i.e., all defect-localization questions and the anomaly-detection questions on which the calibrated gate fires. For each such question, the defect expert gives an evidence score t ∈ [0, 1], the within-task percentile of the anomaly-map strength (t = 1: strongest map). The controller scales a reader's vote by ρ = 1 + βt (β = 8). Thus, a reader with weak evidence counts as about one ordinary vote, whereas a reader with strong evidence can outvote the 8-agent base team. The question is therefore whether readers are indeed more reliable than the team exactly where ρ gives them more weight.

Analysis. We use all 6,629 questions with active readers (4,878 localization, 1,751 anomaly detection) and the frozen predictions, counting each of the 3 readers separately (19,887 reader–question pairs). For each pair, we compare (i) Reader, which sees the anomaly-map overlay; (ii) Same model without map, the same MLLM answering as a base agent, which isolates the effect of the map; and (iii) Base team, the aggregated answer of the 8 base agents. The decisive subset is disagreement, since only there does the reader's weight change the final prediction. The analysis is post hoc and changed no parameter.

Defect localization (14,634 pairs), accuracy (%) per evidence bin.

Evidence score t[0, .2)[.2, .4)[.4, .6)[.6, .8)[.8, 1]
Reader59.770.277.582.284.6
Same model without map54.163.563.262.969.2
Base team62.771.873.171.478.2
When reader and team disagree:
Reader correct36.140.050.263.256.8
Team correct42.945.235.727.328.7
Net questions gained by following the reader−86−49+127+315+188

As shown in the table, reader accuracy rises steadily with t, so the score reflects map quality. Moreover, the map itself helps: the reader's gain over the same model without the map grows from 5.6 in the lowest bin to 15–19 in the top two. Most importantly, on disagreements the reader is less reliable than the team when t < 0.4, but far more reliable when t ≥ 0.4, by up to 35.9. This is exactly what ρ = 1 + βt implements: at low t the team prevails, and at high t the reader can override it.

Anomaly detection (5,253 pairs, only questions where the gate fires), accuracy (%).

Evidence score t[0, .2)[.2, .4)[.4, .6)[.6, .8)[.8, 1]
Reader96.597.395.595.496.9
Base team82.282.486.085.792.0
Reader correct, when reader and team disagree90.389.983.380.087.0

As shown in the table, readers vote only on the 1,751 of 8,297 AD questions where the gate fires. On these questions, they exceed 95% accuracy in every bin and are right in 80–90% of disagreements, so up-weighting them is safe at any t.

We note that t is a within-task rank rather than a calibrated probability of correctness, and that the localization advantage peaks in [.6, .8) rather than in the top bin. We will add these tables to the CR and clarify this in Sec. 3.5.

Reviewer_787kOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The primary contribution of this paper lies in its in-depth analysis and revelation of the current capability boundaries of Multimodal Large Language Models (MLLMs) in the context of industrial anomaly understanding. While both domain-fine-tuned expert models and state-of-the-art closed-source LLMs (such as Gemini-2.5-Pro) demonstrate strengths in semantic description and analysis, they consistently exhibit significant shortcomings in fine-grained visual perception tasks, specifically regarding anomaly detection and defect localization.

To address this core challenge, the authors propose a meticulously designed, training-free industrial pipeline called SiGMA. This framework leverages a multimodal retriever to obtain priors from normal samples, incorporates a DINOv3-based, training-free spatial grounding visual expert, and employs a noise-robust calibration mechanism to precisely extract anomalous regions from query images. Subsequently, the framework utilizes these highlighted visual cues to guide the downstream MLLM in its decision-making process, with an enhanced statistical voting algorithm to aggregate predictions from multiple sources. Experimental results demonstrate that this purpose-built toolchain is highly effective, substantially mitigating the perceptual limitations of MLLMs and achieving superior performance on benchmark tests.

Critical strengths
  1. The authors begin by analyzing the performance of current Multimodal Large Language Models (MLLMs) across a wide range of industrial anomaly detection benchmarks. They demonstrate that existing models—whether open-source, closed-source, or domain-fine-tuned—suffer from a common weakness: poor fine-grained visual perception. This finding confirms that relying solely on fine-tuning (SFT/RL) or scaling up model size is insufficient to address the core challenges of industrial quality inspection.

  2. The authors propose a sophisticated pipeline for industrial anomaly detection. This approach enhances detection capabilities by integrating anomaly heatmaps—generated via specialized tools—and aggregating outputs from multiple MLLMs.

  3. The experimental evaluation is comprehensive, achieving state-of-the-art (SOTA) performance across several datasets while also delivering faster inference speeds.

Critical weaknesses
  1. Although the pipeline presented in this paper is highly efficient, it lacks true "agentic" intelligence; the model itself has not learned to autonomously perceive uncertainty, invoke tools on demand, or engage in multi-turn interactive reasoning. Consequently, it is not a strong fit for the themes typically explored at ICLR.

  2. The authors acknowledge in the appendix that the selection of the subset of 8 base models, as well as all critical hyperparameters (such as quantile thresholds and weighting coefficients), were determined by directly comparing results against the ground-truth answers in the MMAD test set. This approach lacks validation regarding generalization to unseen, independent test sets.

  3. The full-scale version requires the concurrent invocation of up to 11 MLLMs (8 base models + 3 readers), in addition to CLIP and DINOv3, resulting in a peak VRAM usage of 85–105 GiB (necessitating 4 L20 GPUs). Comparing such a massive ensemble—which exerts overwhelming computational dominance—against a single 7B fine-tuned model (requiring only 15.2 GiB of VRAM) is inherently unfair. Furthermore, the substantial VRAM requirements pose significant challenges for practical deployment in industrial settings.

Questions for authors

I suggest that the authors consider how to train an intelligent agent capable of autonomously determining when and how to invoke these tools—engaging in multi-turn interactions—to effectively address the ICLR theme.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 787k (1/3)

W1 & Q1: The pipeline lacks true agentic intelligence; the model does not autonomously perceive uncertainty, invoke tools on demand, or engage in multi-turn reasoning.

A: We thank the reviewer for this constructive suggestion. We agree that our original presentation, which described SiGMA more like a fixed pipeline, did not make its agentic nature explicit. Importantly, our main contributions do not depend on this framing, and the novelty of SiGMA does not come from being an agentic system. In fact, SiGMA can be naturally framed as a complete agentic system, since its modules already act as tools and its reliability controller already maintains a label-free belief over the answers. Without any training, SiGMA already outperforms the strongest trained specialist on MMAD by 4.0. Nevertheless, the agentic framing leaves room for further optimization. Taking a further step toward a fully autonomous agent only requires adding a lightweight orchestrator on top of the unchanged modules, which decides, turn by turn, which tool to call, whom to ask and how to ask. On the full MMAD, the resulting SiGMA-Agent reduces MLLM calls by 45% at a cost of only 0.1 in accuracy. Code and details are in https://anonymous.4open.science/r/SiGMA-Agent-20EC, and we will add this agentic formulation to the camera-ready version (CR).

Specifically, SiGMA-Agent consists of the following components.

  • Orchestrator. A frozen Qwen3-VL-8B. Each episode starts from only the query image, the question and the orchestrator's own first look. In each turn, it writes <think> and then either one <tool_call> or <answer>.
  • Tools. reference(k) returns k similar normal samples, zoom(bbox) crops a region, defect_map() returns the defect expert's anomaly map and calibrated gate, read_map() calls the expert readers (available only after defect_map), and ask(model, view) queries any base MLLM in one of four views:
    • glance: the official prompt with the whole image and one normal template, answered directly (default, cheap);
    • reason: step-by-step comparison against the template (expensive);
    • tiles: the whole image plus four high-resolution quadrant crops;
    • hotspot: a zoom on the defect-map hotspot, side by side with the same region of the template (defect questions only).
  • Deciding when and how to ask. The orchestrator first lets peers glance. Only when they disagree does it ask a peer to reason, to inspect high-resolution details, or to focus on the region flagged by the defect map. Thus, it decides not only whom to ask and when, but also where and how closely a peer should look, i.e., active perception delegated to peers.
  • Perceiving uncertainty. After every observation, the orchestrator sees a belief: the reliability-weighted posterior over options, the margin of the leading option, and which sources conflict. A verifier asks for more evidence while the leading option is below a threshold. The final prediction is the top option of the belief, so the answer is grounded in the gathered evidence.

Results on the full MMAD. Here, source reliabilities are estimated online and label-free from previously answered questions only.

SystemMMAD avg. (official, %)MLLM calls / question
SiGMA85.28.50
SiGMA-Agent (adaptive tool use)85.14.66

As shown in the table, SiGMA-Agent reduces MLLM calls by 45% at a cost of only 0.1 in accuracy. The savings come from adaptive tool use: SiGMA-Agent stops after one or two glances when peers agree, and spends its budget only on questions where they conflict.

Notably, SiGMA-Agent achieves this agentic behavior without any training: the frozen orchestrator already decides when, whom and how to ask, while keeping accuracy close to SiGMA. As the next step, we will carefully consider learning the orchestration policy with SFT and RL.

AuthorsOpenReview ↗

Response to Reviewer 787k (2/3)

W2: The 8-model team and key hyper-parameters were selected with MMAD test answers; generalization to unseen test sets is not validated.

A: We thank the reviewer for this important point, and we agree that the original team and key hyper-parameters were developed on MMAD. Nevertheless, SiGMA generalizes well beyond MMAD and does not rely on careful tuning. Since SiGMA trains no model weights, its risk of overfitting lies only in the choice of hyper-parameters, and we verify its generalization in two ways. First, we froze the team and all hyper-parameters and evaluated on MANTA [1], an additional benchmark of tiny objects from five domains, each captured from five viewpoints, which was never used in any design choice. Without any training or tuning, SiGMA outperforms all reported methods, improving over the strongest trained specialist SAGE [2] by 4.6, while the trained JUDO [3] even falls below its own untrained base model. A plausible reason is that SFT and RL fit the model weights to the training distribution, which limits their generalization under distribution shift, whereas SiGMA keeps all MLLMs frozen and re-estimates source reliabilities label-free on each new benchmark. Second, on MMAD itself, SiGMA is insensitive to all key hyper-parameters, with performance varying by at most 0.5. Together, these results indicate that SiGMA's performance stems from the framework itself rather than from tuning on MMAD.

(a) Additional unseen benchmark. MANTA [1] focuses on tiny objects, such as grains, pills and small parts, from five domains (agriculture, electronics, groceries, mechanics, medicine). Each image stitches five viewpoints, and five binary items are asked about each object. Besides item accuracy over all 10,000 items, we report the stricter all-five-correct rate, where all five items about an object must be answered correctly. We froze the team, the readers, all hyper-parameters and the decoding budgets.

SystemMANTA item acc. (%)All five correct (%)
LLaVA-1.5 [1]56.83.7
LLaVA-NeXT [1]57.43.4
InternVL2 [1]53.31.1
Molmo [1]54.14.5
BLIP-2 + LoRA [1]66.58.7
LLaVA-1.6 [2]49.04.5
Qwen2-VL [2]54.35.3
DeepSeek-VL2 [2]50.4–
GPT-4o [2]63.73.9
InternVL2 (SFT) [2]71.3–
SAGE (trained specialist) [2]73.523.8
Qwen3.5-9B67.020.4
Qwen2.5-VL-7B, same context73.131.7
JUDO, same context70.030.2
JUDO + same defect-expert evidence71.831.9
SiGMA without defect expert77.537.8
SiGMA (full)78.137.3

As shown in the table, SiGMA improves over SAGE [2] by 4.6, over the fine-tuned InternVL2 by 6.8, and over GPT-4o by 14.4. The gain is even larger under the all-five-correct metric, where SiGMA improves over SAGE by 13.5. In contrast, in-domain training does not transfer: given the same context, JUDO falls 3.1 below its own untrained base model, Qwen2.5-VL-7B, and remains below it in item accuracy even with the same defect evidence.

(b) Sensitivity on MMAD. On a stratified subset of 1,024 images (4,981 questions) with the full 8 base agents and 3 readers, we vary each key hyper-parameter while fixing the others at their defaults.

Hyper-parameterRoleDefaultValues testedMMAD avg. (%)Max change
βevidence weighting, ρ = 1 + βt80, 2, 4, 8, 1683.8 – 84.30.5
wglobal vs. task-wise reliability0.350, 0.175, 0.35, 0.7, 184.3 – 84.40.05
p_lownoise quantile for map calibration0.990.95, 0.975, 0.99, 0.99584.3 – 84.40.05
δmargin of the spatial gate0.010, 0.005, 0.01, 0.02, 0.03, 0.0584.1 – 84.30.2

As shown in the table, every value stays within 0.5 of the default. The largest change comes from switching off evidence weighting entirely (β = 0), and w and p_low change the result by at most 0.05. The reported results therefore do not rely on finely tuned hyper-parameters.

[1] MANTA: A Large-Scale Multi-View and Visual-Text Anomaly Detection Dataset for Tiny Objects. CVPR 2025.

[2] SAGE: A Visual Language Model for Anomaly Detection via Fact Enhancement and Entropy-aware Alignment. ACM MM 2025.

[3] JUDO: A Juxtaposed Domain-Oriented Multimodal Reasoner for Industrial Anomaly QA. ICLR 2026.

AuthorsOpenReview ↗

Response to Reviewer 787k (3/3)

W3: Up to 11 MLLMs plus CLIP and DINOv3 need 85–105 GiB (4 L20 GPUs); comparing with a single 7B model (15.2 GiB) is unfair and hard to deploy.

A: We thank the reviewer for raising this practical concern, and we address deployment difficulty and model-size fairness separately. For deployment, SiGMA shares checkpoints and caches repeated inputs across roles, loads models in turn while each answers its questions in parallel batches, and quantizes their language weights to NF4. With these steps, the full system needs only 18.4 GiB, which fits on a single consumer GPU such as the RTX 4090 (24 GB), and is less than the trained 7B specialist JUDO [3] in BF16, while being 3.1 more accurate. For fairness, against single MLLMs of similar total size (72B/76B), SiGMA is still 5.0 higher on MMAD.

(a) Deployment difficulty. SiGMA uses only open ≤9B MLLMs served with standard vLLM, without any training. We reduce its deployment cost in three steps. First, roles that use the same model, such as the base and reader roles of Qwen3-VL-8B and GLM-4.1V, share one checkpoint, and repeated image inputs are cached. Second, since the controller only needs each model's votes, the models are loaded in turn while each answers its questions in parallel batches. Third, the language weights are quantized to NF4.

We measured this on a fixed subset of 368 MMAD questions that covers all four sources and seven question types, regenerating all answers. Times include model loading, inference and aggregation, and all rows use 2× L20.

ConfigurationPeak memory (GiB)Time / question on 2× L20 (s)Acc. on subset (%)
JUDO (7B), BF1620.61.6885.6
JUDO (7B), NF411.51.4784.7
SiGMA, original BF16 serving81.54.7089.1
+ shared checkpoints and caching68.53.7589.0
+ loading in turn (5 BF16 + 4 NF4 checkpoints)26.92.9788.9
+ NF4 language weights for all18.41.9988.7

As shown in the table, with NF4 language weights SiGMA uses 77% less memory and runs 2.4× faster than its original serving, at a cost of only 0.4 in accuracy. Compared with JUDO in BF16, SiGMA needs less memory and only 0.3 s more per question, while being 3.1 more accurate.

(b) Fairness in model size. The 9 checkpoints of SiGMA total about 70B parameters. We therefore compare with single large MLLMs of similar size on MMAD, each given domain knowledge.

ModelParamsAvg.(%)ADCls.Loc.
InternVL2-Llama3-76B76B76.266.771.060.6
Qwen2.5-VL-72B-Instruct72B80.271.673.969.1
SiGMA~70B (9 × ≤9B)85.282.679.279.6

As shown in the table, at a similar total size SiGMA outperforms the strongest 72B model by 5.0 on average, by 11.0 in anomaly detection and by 10.5 in defect localization. The advantage therefore does not come from more parameters, but from dividing the labor among heterogeneous agents and a defect expert.

(c) Matched information. Given exactly the same retrieved context and defect-expert evidence as SiGMA, JUDO reaches 82.3 on MMAD, still 2.9 below SiGMA, and the gap widens to 6.3 on the unseen MANTA (W2). We will report memory, latency, parameters and calls side by side in the CR.

[3] JUDO: A Juxtaposed Domain-Oriented Multimodal Reasoner for Industrial Anomaly QA. ICLR 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Opening

We sincerely thank all reviewers for the time and care they devoted to evaluating our work and for their thoughtful, constructive feedback. We appreciate your recognition of the fine-grained perception bottleneck in industrial anomaly understanding and the value of combining heterogeneous MLLMs with a dedicated visual defect expert in SiGMA. Your comments have helped us identify where clearer explanations and stronger evidence are needed, and have guided the additional experiments and analyses presented in our responses.

Below, we address each reviewer’s concerns in detail. We hope our responses clarify the motivation, design, and evaluation of SiGMA, and we welcome further questions and discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.