acceptodds
Withdrawn from ICLR 2027

MTL-MAD: Multi-Task Learners are Effective Medical Anomaly Detectors

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Anomaly detection in medical images is a challenging task, since anomalies are not typically available during training. Recent methods leverage a single pretext task coupled with a large-scale pre-trained model to reach state-of-the-art performance. Instead, we propose to learn multiple self-supervised and pseudo-labeling tasks from scratch, using a joint model based on Mixture-of-Experts (MoE). By carefully integrating multiple proxy tasks, the joint model effectively learns a robust representation of normal anatomical structures, so that anomaly scores can be derived based on how well the multi-task learner (MTL) solves each task during inference. We perform comprehensive experiments on BMAD, a recent benchmark that comprises a broad range of medical image modalities. The empirical results indicate that our multi-task learner is an effective anomaly detector, outperforming all state-of-the-art competitors on BMAD. Moreover, our model produces interpretable anomaly maps, potentially helping physicians in providing more accurate diagnoses. To reproduce the reported results, we publicly release our code at https://anonymous.4open.science/r/MTL-MAD.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
anomaly detection, medical anomaly detection, multi-task learning, mixture-of-experts

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_m7JjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MTL-MAD, a multi-task learning method for unsupervised medical image anomaly detection. It combines five heterogeneous anomaly-detection objectives to address a key limitation: a training set cannot practically represent every possible abnormality. The experiment is comprehensive and MTL-MAD outperforms the baselines across all metrics on the BMAD benchmark.

Critical strengths
  • This paper addresses a practical limitation in the medical anomaly detection: the anomaly classes may be very limited, and expensive to annotate. The motivation for combining 5 self-supervised objectives is reasonable, particularly given the heterogeneous appearance of anatomical structures and pathological abnormalities across different medical imaging modalities.

  • The results are quite strong over baselines with ablation showing the benefit of the task combinations.

Critical weaknesses
  • The novelty is limited. This paper combines 5 existing tasks (Masked image modeling, Jigsaw prediction, DeMixUp, augmentatino-based classification, generation-based classification). It does not sufficiently establish a substantial methodological advance beyond these existing approaches. In addition, the literature is also limited, It lacks the discussion and performance comparison on recent methods, such as InvAD [1], PDD [2], UniMMAD [3].

  • The ablation is insufficient. The main contribution is the intergation of the task-specific experts inside a shared transformer. However, the there is no ablation to demonstrate the benefits of this design. In addition, Section 3.1 states that routing is deterministic, with each task assigned a fixed subset of experts whose outputs are averaged. This differs substantially from conventional learned MoE routing.

  • Other weakness: The writing is difficult to follow. The research problem should be clearly described in the abstract, Figure 1 also does not highlight the main contributions of the paper. And the code is not provided.

Reference: [1] InvAD: Inversion-based Reconstruction-Free Anomaly Detection with Diffusion Models, CVPR 2026

[2] PDD: Manifold-Prior Diverse Distillation for Medical Anomaly Detection, CVPR 2026

[3] MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detection, NeurIPS 2024

Questions for authors
  1. In the evaluation, why only image-level scores are reported? How about the pixel-level scores?
  2. In the formulation of jigsaw prediction objective, Equation (3) is described as binary cross-entropy but includes only the positive-label term. Is the negative-label term omitted from the equation, or is the implementation based on a different classification formulation?
Reviewer_nJhUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a somewhat novel method entitled MTL-MAD (for Multi-Task-Learners-for-Medical-Anomaly-Detection) for use of unsupervised medical anomaly detection. A vision transformer-based model with Mixture-of-Experts (MoE) layers for five different “proxy” tasks to capture different notions of normality/typicality. The five tasks include a masked image modeling, a jigsaw puzzle, a patch-mixup (which is a novel task), inpainting of pseudo anomalies made by well-known augmentations, and finally classification of pseudo-anomalies generated from domain-specific text prompts. Each task use a fixed set of experts. At test time, per-task anomaly score (the nature of the scores vary by task) are combined by a weighted sum. The paper reports better than SOTA performance on the BMAD benchmark that spans six heterogeneous anomaly datasets. The reported ablations cover the tasks, transformer depth, dimension, number of experts, expert dropout mainly on the BRATS2021 data. A visual/quality comparison of the anomaly maps is performed by a physician who rated the proposed method as “substantially more accurate”.

Critical strengths

The general idea makes sense and is well motivated. The proposed method could also easily be extended to include further tasks to capture additional notions of “normality”.

The idea is interesting for discussion at ICLR.

The new task is straightforward to implement and works well on its own as can be seen from Table 2, adding 8.62 AUROC on top of masked-image-modeling and jigsaw.

Evaluation covers all BMAD datasets.

Ablations are fairly extensive.

Code has been released (anonymously).

Critical weaknesses

The main issue: no pre-training/fairness of the comparison: Authors stress that there is no pre-training needed, which is somewhat misleading as task 5 relies on a large pre-trained model with knowledge of medical anomalies. Its pseudo-anomalies stem from generative models that were prompted with text that describes the medical pathology. In particular, these prompts (Table 9) include “Brain tumor” and “Brain lesion”. This is akin a supervised tumor detection training that directly impacts the performance on the BRATS data (which is also the dataset where most of the ablations are performed). Similarly, there are prompts that match directly to the RSNA anomaly types. The use of task 5 clearly biases the evaluation (in favor of the proposed model) and it is hard to truly consider the proposed method an anomaly detection method for BRATS and RSNA-related data. Not surprisingly, this task alone reaches 90.72 AUROC on BRATS (Table 2), which is close to the full model. An appropriate evaluation for BRATS could have been performed without using the prompt related to tumors and lesions (i.e. using only generic or pathology agnostic prompts). As a consequence of the above, the method is really “anomaly-type-informed” than fully unsupervised, and this should be appropriately stated in the paper.

Novelty: Multi-task self supervised methods exist for videos and 2D images. The idea of using per-task-experts in a shared transformer is a somewhat modest step beyond that.

Baselines do not seem to be tuned the way the paper says: While the authors state that all baselines received equivalent effort in hyperparameter tuning, table 1 largely matches the performance to those listed in the BMAD paper (with rounding), i.e. the baseline performance is copied from the BMAD paper. That paper though states that it used default hyperparameters and that the reported performance is not optimal for the methods. In comparison, the proposed method was heavily tuned (as expected). Please correct the statement about baseline tuning.

No significance tests to look at difference to baselines: Please add significance information to the comparison table, as several differences to baselines seem to be too small to be significant.

Questions for authors
  • What is the performance when using pathology-agnostic-only prompts for task 5 (or prompts that do not overlap with the nature of the BMAD datasets)? (needed for a fair evaluation)
Reviewer_hYpXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MTL-MAD proposes medical anomaly detection through a shared vision transformer with deterministic task-specific expert allocation and five objectives covering masked reconstruction, jigsaw prediction, donor-patch identification and two forms of synthetic-anomaly classification. The detector uses progressive task activation and FAMO, incorporates training anomalies produced by pretrained generators, and combines task-specific outputs during inference. The paper reports image-level AUROC on six BMAD datasets, component and hyperparameter ablations, lesion-size stratification and qualitative localization comparisons.

Critical strengths
  1. Table 1 reports the highest mean image-level AUROC among the listed competitors on all six BMAD datasets across five imaging domains, including a 10.4 percentage-point gain on BTCV+LiTs under dataset-specific training. The small-lesion analysis in Table 4 additionally reports 87.94 AUROC compared with 85.95 for QFAE, providing evidence on a difficult subset beyond aggregate benchmark performance.

  2. The component ablations in Tables 2 and 8 support the practical value of integrating complementary objectives with task-specific experts and adaptive optimization. On BraTS2021, the complete task combination reaches 95.49 AUROC compared with 90.72 for the strongest individual task, while the architectural and optimization ablation reports an increase from 93.0 for base multi-task learning to 95.5 for the final MoE/FAMO configuration.

Critical weaknesses
  1. The methodological contribution described in Sections 2, 3.1 and 3.2 is primarily an adaptation and combination of established mechanisms, while the claimed new DEMIXUP principle is insufficiently distinguished from prior work. [ref.1] already provides task-dependent expert routing, and [ref.2] already predicts corresponding-location foreign content and includes binary replacement. [ref.3] also learns dense foreign-region predictions using Poisson blending and pixel-level supervision. MTL-MAD introduces fixed expert averaging and token-grid origin classification, but Tables 2 and 8 establish benefits within the proposed combination without isolating the advantages of these changes over the closest existing formulations.
  2. The empirical evidence does not yet adequately establish the comparative and localization superiority claimed in Section 4. The three-run reporting in Table 1 and the equivalent-tuning statement remain unreconciled with matching published five-run BMAD statistics, leaving the comparability of the evaluation uncertain. Localization claims rely on eight selected examples in Figure 2 and a physician’s qualitative interpretation, while Table 4 measures image-level classification rather than localization accuracy and neither Section 4 nor Appendix C reports full-test-set pixel-level or region-level results.

REF:

  1. M³vit: Mixture-of-experts vision transformer for efficient multi-task learning with model-accelerator co-design. Advances in Neural Information Processing Systems, 2022
  2. Detecting Outliers with Foreign Patch Interpolation, Machine Learning for Biomedical Imaging 2022
  3. Natural synthetic anomalies for self-supervised anomaly detection and localization. ECCV 2022
Questions for authors
  1. Does the jigsaw head use independent sigmoid outputs with binary cross-entropy or a per-tile softmax with categorical cross-entropy, and which implemented objective produced Table 2? Equation 3 in Section 3.2 omits the negative binary cross-entropy term and would admit a trivial all-one prediction under independent outputs.
  2. What exact inference procedure produced Table 1 and Figure 2, including task-fusion weights, their selection rule and the masking coverage or repeated evaluations used to construct complete reconstruction maps? These details should be clearified in Section 3.4 and the supplied appendices.
  3. Which Table 1 baseline statistics were newly measured and which were taken from published results, and what run counts and validation-tuning procedures apply to those comparisons?
  4. What full-test-set pixel-level or region-level results on the three mask-annotated datasets support the localization superiority claims associated with Figure 2 in Section 4?
  5. What evidence isolates the benefit of DEMIXUP’s token-grid formulation in Section 3.2 over FPI’s binary replacement task when the backbone and joint-training setting are held fixed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.