acceptodds
Withdrawn from ICLR 2027

UniMRE: Modality-Specialized Retrieval and Reasoning for Unified Multimodal Anomaly Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Unified multi-class anomaly detection has attracted growing interest as an efficient alternative to the conventional one-class-one-model paradigm, reducing training and maintenance costs for industrial deployment. However, real-world industrial defects span both appearance defects, such as scratches and contamination, and geometric defects, such as protrusions and deformations. Most existing unified methods rely primarily on RGB images and therefore lack explicit geometric cues for detecting structural anomalies. In this paper, we propose Unified Multimodal Retrieval-guided Experts (UniMRE), a unified framework that preserves category-specific normality and complementary appearance–geometry evidence through dual-stream retrieval-guided reasoning. UniMRE constructs class-conditional normal memories in separate RGB and joint RGBD feature spaces, where stream-wise retrieval provides stream-specific normal references and matching evidence for anomaly reasoning. Retrieval-guided expert pathways progressively reason over the stream-specific evidence, while lightweight bidirectional cross-stream conditioning enriches feature reasoning without altering the matching volumes. The resulting predictions are integrated by a learnable class-aware pixel fusion conditioned on local expert responses and nearest-normal retrieval cues. Extensive experiments on MVTec-3D, Eyecandies, and MulSen-AD demonstrate improved image-level anomaly detection over the existing unified multimodal baseline under the full-shot setting, together with strong few-shot performance and competitive results against specialized one-class-one-model approaches. Code is available at https://anonymous.4open.science/r/unimre-iclr-anon-62EA.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Anomaly Detection, Unified Anomaly Detection, Multimodal Representation Learning, Normality Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UVbxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates multi-class anomaly detection and localization using RGB and depth information. UniMRE establishes separate RGB and joint RGBD normal feature libraries for each class, then processes the retrieval results through a two-stage expert network, and combines cross-branch conditional information and learnable pixel-level fusion to obtain the final prediction. During testing, the model first estimates the class of the input and then selects the corresponding feature library. Experiments on three datasets show that UniMRE outperforms UniMMAD in image-level detection, with a more significant advantage when there are fewer normal training samples, but the improvement in localization performance is not consistent.

Critical strengths
  1. Retaining a separate RGB retrieval branch while using a combined RGBD branch is a reasonable choice, balancing aesthetics and geometry. The ablation results in Table 3 also support this design: the I-AUROC of RGB+RGBD is 93.3%, higher than 92.2% for RGB+Depth and 88.1% for using only RGB.
  2. The improvement achieved with a small number of samples is significant. Table 2 shows that on the MVTec-3D dataset, the I-AUROC is 85.2% when using five normal samples per class, compared to 72.5% for the reproduced UniMMAD baseline. Appendix E.1 also shows that the I-AUROC is 84.1 ± 0.8% in three sample subsets, strongly demonstrating that the results are not dependent on a single favorable subset.
  3. This evaluation covers three datasets and tests an important practical assumption: selecting the correct category during inference. Appendix E.3 reports that routing accuracy reached 100% on the MVTec-3D and Eyecandies datasets, and 95.3% on the MulSen-AD dataset. This demonstrates that memory for selecting a specific category can be achieved without providing category labels at test time.
Critical weaknesses
  1. The comparisons do not sufficiently separate the effect of the feature encoders from that of the proposed architecture. UniMRE uses DINOv3-S and DUNE-S, while Appendix E.2 matches input resolution without establishing a comparison using the same encoders. This matters particularly for the relatively small full-shot I-AUROC gains on MVTec-3D and Eyecandies.
  2. I am unsure which version of the fusion module was used in the experiments. Equations (8) through (10) describe a convolutional gate with class bias that is trained using a focus loss, but the version in Figure 5 also includes class embedding, FiLM residual refinement, and image-level loss. The paper should clarify whether these additional components were used to obtain the reported results.
  3. The advantages of the two-stage expert architecture need to be demonstrated with stronger evidence. As shown in Table 4, the retrieval function alone achieves 92.2% I-AUROC and 95.0% I-F1, while the corresponding figures for the full model are 93.3% and 95.0%, respectively. Comparison with simpler learned predictive heads helps determine whether the increased complexity of expert routing and the two-stage structure is worthwhile.
  4. The efficiency comparison does not directly establish an advantage over the main unified baseline. Table 5 omits UniMMAD and takes the measurements for M3DM and MulSen-TripleAD from another paper, without establishing that the measurement conditions are comparable. Reporting UniMMAD under the same hardware and timing protocol would make the deployment comparison more convincing.
Questions for authors
  1. Which RGB and geometry encoders were used in the UniMMAD reproductions in Tables 2 and 19? Were they the same as those used by UniMRE?
  2. Which fusion components and losses were used for the reported results? Specifically, were the class embeddings, FiLM residual refinement, and image-level loss shown in Figure 5 included?
  3. What components are retained in the “Retrieval Only” setting in Table 4, particularly for fusion and image scoring? Are there any existing comparisons with a simpler prediction head that help establish the benefit of the two-stage expert architecture?
  4. Does the reported latency of 0.185 s include feature extraction, category assignment, retrieval, and fusion? Are UniMMAD measurements available under the same hardware and timing protocol?
Reviewer_boagOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

UniMRE addresses multimodal industrial anomaly detection with one model per dataset shared across object categories. Frozen DINOv3-S and DUNE-S encoders produce RGB and joint RGBD representations. Separate class-conditional normal memories provide nearest-neighbor matching volumes and centroid-based normal references. Two-stage expert pathways process these references, with bidirectional conditioning confined to the feature pathways. A class-aware spatial gate combines the stream predictions, augmented at inference by direct retrieval distances. Training uses synthetic appearance and depth anomalies, and category assignment at inference uses normal-feature prototypes.

Experiments cover MVTec-3D, Eyecandies, and MulSen-AD, including full-shot and 5/10/50-shot settings. The paper reports higher image-level AUROC than its selected UniMMAD baseline, particularly in low-shot settings, with mixed localization results. The supplement provides category-level results, support-set variation, a resolution-matched comparison, and category-routing measurements. The main contribution is the integration of modality-specialized retrieval, cross-stream expert processing, and spatial fusion into a shared detector.

Critical strengths

S1. A coherent design for sharing a detector while preserving complementary normality evidence

The distinction between stream-specific matching evidence and cross-stream feature reasoning is well motivated. Keeping separate RGB and RGBD memories allows the two streams to retain different normal correspondences, while conditioning and spatial fusion allow their predictions to interact. The reported MVTec-3D ablation supports combining modalities: RGB and depth alone achieve 88.1 and 84.5 I-AUROC, RGB+Depth achieves 92.2, and RGB+RGBD achieves 93.3. This is a useful design contribution for a practical setting in which appearance and geometric defects coexist.

S2. Substantial empirical coverage and promising low-shot behavior

The evaluation covers three datasets and several training-set sizes, with detailed category-level results. On MVTec-3D, the reported 5-shot I-AUROC improves from 72.5 for the reproduced UniMMAD baseline to 85.2 for UniMRE; the corresponding Eyecandies result improves from 70.7 to 78.4.

Critical weaknesses

W1. The released implementation leaves the independence of evaluation and the reported configuration unresolved

In the anonymous release, test_dual_fusion loads the MVTec-3D test samples and their labels (train_mvtec3d.py, lines 1253 and 1265-1268). The training loop repeatedly calls this evaluator and saves the checkpoint that maximizes its fused image-level AUROC (lines 1777-1832); the README also describes this selection rule. Selecting the reported checkpoint this way would use test labels for model selection and can produce optimistic estimates. The manuscript does not specify an independent validation-based or predetermined checkpoint-selection rule.

The released defaults also differ materially from Appendix A. Five categories retain their entire memory banks instead of the stated 0.1 keep ratio (lines 112-128 and 455-467). The fusion bias is initialized from manually specified category weights (lines 360-368 and 1574-1580), and foam receives unequal branch-loss multipliers (lines 384-393), beyond the global equal weights described in the paper. These settings may be legitimate, but their selection procedure and use in the reported experiments are not documented. Together, these discrepancies prevent attributing the reported gains confidently to the stated method. The authors should identify the exact configurations and provide results with a checkpoint rule fixed independently of test performance.

W2. The main baseline comparison does not establish an advantage under matched inputs and training conditions

The largest full-shot gain, 91.2 versus 85.4 I-AUROC on MulSen-AD, is not a controlled comparison of two RGB+geometry detectors. UniMMAD's original paper explicitly evaluates MulSen-AD using infrared only [Appendix C and Table 8], whereas this submission uses RGB and geometry (Appendix B.3). Its 85.4 baseline and category-level values correspond to that infrared evaluation [Table 19]. Pixel metrics likewise require matching the annotation modality, rather than directly comparing infrared localization with RGB localization.

There are additional differences in the original full-shot comparison: UniMMAD jointly trains across datasets and uses WideResNet50 features [Secs. 4.1-4.2], whereas UniMRE trains separately per dataset with DINOv3-S/DUNE-S. Appendix E.2 usefully controls resolution on MVTec-3D, but does not remove these other differences. The same-feature retrieval-only ablation is informative, yet already obtains 92.2 versus 93.3 I-AUROC on that dataset (Table 4), and is not extended to the other datasets or few-shot settings. Matched-input baselines and broader same-feature controls are therefore necessary to establish how much improvement comes from the proposed expert interaction and fusion.

Questions for authors

Q1. Which model-selection rule and configurations produced the reported results?

Please provide the commands/configurations and checkpoint identifiers underlying Tables 1-4, and clarify whether the checkpoint was selected by test-set AUROC as in the release. How were the category-specific fusion initializations, memory ratios, and loss multipliers selected? If test performance influenced these choices, please report results using a predetermined final checkpoint or an independent validation rule, with the configuration fixed before test evaluation.

Q2. Does the advantage persist under the same modality, labels, and training scope?

Please clarify the exact input modalities, image labels, and pixel-mask modality used for every MulSen-AD baseline, including the few-shot reproduction. Can UniMMAD and a simple retrieval baseline using the same frozen features be compared with UniMRE under the same RGB+geometry inputs, per-dataset training, test samples, and checkpoint-selection rule? In particular, a comparison should account explicitly for the infrared-only baseline identified in W2. Consistent gains under these controls would materially strengthen the contribution.

Reviewer_bA9XOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates unified multi-category RGB/RGBD anomaly detection and proposes UniMRE. The framework maintains modality-specific normal memories for different categories, independently retrieves and processes anomaly evidence in the RGB and RGBD streams, exchanges cross-modal context through lightweight bidirectional conditioning, and employs category-dependent pixel-wise gated fusion to combine predictions from the two streams. The paper conducts experiments on MVTec-3D, Eyecandies, and MulSen-AD under both full-shot and few-shot settings.

Critical strengths
  1. The paper addresses two practical deployment challenges: unified multi-category anomaly detection and RGB/geometric multimodal anomaly detection. It independently preserves RGB and RGBD retrieval evidence and then enables interaction during inference, aligning the research motivation with the method design.

  2. The experiments cover three multimodal benchmarks under full-shot and 5/10/50-shot settings. The appendix also reports robustness to random support set sampling and inference-time category routing accuracy, providing a relatively comprehensive experimental evaluation.

Critical weaknesses
  1. The so-called “unified” nature of the proposed method still relies on separate dual memories, category centroids, and category biases for each category. Storage and retrieval requirements grow linearly with the number of categories, and new or unseen categories also require dedicated memories. In fact, the proposed method is not fundamentally different from using a separate model for each object category. The paper also does not evaluate the method's performance on unseen categories. Therefore, the method is more accurately characterized as category-conditioned retrieval with shared experts.

  2. The modality specialization lacks causal evidence at the defect-type level. The final RGBD stream also contains RGB information, the gains from cross-stream interaction are relatively small, and the paper does not separately report the performance of the two streams, fusion gates, or failure cases for appearance and geometric defects. Furthermore, UniMRE does not outperform competing methods on several pixel-level metrics, and the existing comparisons are insufficient to demonstrate its general superiority.

  3. The paper provides insufficient explanation of how synthetic anomalies maintain physical consistency across RGB and depth modalities during training. If the two modalities contain unnatural synchronized artifacts, the experts may learn shortcuts associated with the anomaly generator rather than genuine defects.

Questions for authors
  1. As the number of categories continues to increase, how do the storage requirements and inference latency of the dual memories, category centroids, and category biases scale?

  2. How does performance change for unseen categories, incorrect category routing, and scenarios where category labels are unavailable?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.