acceptodds
Withdrawn from ICLR 2027

MaskMed: Decoupled Mask and Class Prediction for Medical Image Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Medical image segmentation methods typically use point-wise convolutional heads that map output channels to predefined semantic classes. While effective, this formulation imposes a rigid channel-to-class correspondence that limits flexible spatial-semantic reasoning and feature reuse across classes. We propose MaskMed, a volumetric segmentation framework that decouples mask generation from class prediction. Instead of directly predicting multi-class logits, MaskMed produces class-agnostic binary masks together with their semantic labels using a shared set of object queries. This formulation removes fixed class-channel bindings and enables more adaptive semantic reasoning over anatomical structures. To improve multi-scale feature integration, we introduce a Full-Scale Deformable Transformer that enables deformable cross-scale interaction between encoder and decoder features across the full feature hierarchy while preserving spatial alignment. We further design a masked multi-scale segmentation head with shared query propagation and finest-scale bipartite supervision for consistent coarse-to-fine refinement across decoder stages. Extensive experiments on AMOS22, BTCV, ACDC, and BraTS23 demonstrate state-of-the-art performance. MaskMed improves Dice scores over nnU-Net by +2.0 and +1.3 points on AMOS22 Task 1 and Task 2, respectively, as well as by +1.7 on ACDC, +0.9 on BTCV, and +1.0 on BraTS23.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Segmentation Head, Deformable, Medical Image Segmentation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_i3dROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposed a method called MaskMed. It brings Mask2Former-style mask classification into an nnU-Net framework, with three novel contributions: (1) Masked Multi-Scale SegHead, (2) Finest-scale bipartite matching, and (3) FSD-Transformer. The method is comprehensively evaluated across four benchmarks; it reports gains over nnU-Net on AMOS22 Task 1 (+2.0) and Task 2 (+1.3), ACDC (+1.7), BTCV (+0.9), and BraTS23 (+1.0).

Critical strengths
  1. The paper has a comprehensive evaluation on four benchmarks across different modalities CT, MRI, cardiac and brain tumor data, with official leaderboard submissions for AMOS and BraTS.

  2. Sharing the finest-scale matching across stages is a good fix for deep supervision under bipartite matching.

  3. The 3D MaskFormer and Mask2Former baselines isolate the effect of the mask-classification paradigm itself. This is good to clarify the novelty.

  4. The paper discusses efficiency and lists limitations, which is very informative.

Critical weaknesses
  1. The novelty is somehow overstated. Mask2Former-style query decoders with masked cross-attention for 3D medical segmentation already exist. 3D TransUNet (Chen et al., 2023) uses learnable queries with coarse-to-fine masked attention on a U-Net backbone. MaskSAM (you already cited) is also mask classification for volumetric segmentation. Deformable multi-scale attention bridging CNN encoder features was introduced for 3D medical segmentation by CoTr (Xie et al., MICCAI 2021).

  2. Its not clear if each baseline number comes from re-run under the same protocol, or copied from leaderboards and papers. Please clarify that.

  3. MaskSAM appears for AMOS Task 1 but is dropped from Task 2. Its Task 1 margin (0.905 vs 0.913) is the smallest in the paper. Please be consistent and have apple-to-apple comparison with benchmarks and existing methods.

  4. Missing efficient transformer baselines on BTCV and ACDC (Tables 2 and 3). Please add these efficient hybrid transformers designed for volumetric segmentation, which are missing from both tables: UNETR++ [1], Slim UNETR [2], and MedNeXt [3].

[1] A. Shaker, M. Maaz, H. Rasheed, S. Khan, M.-H. Yang, and F. S. Khan. UNETR++: Delving into Efficient and Accurate 3D Medical Image Segmentation. IEEE Transactions on Medical Imaging,

[2] Y. Pang, J. Liang, T. Huang, H. Chen, Y. Li, D. Li, L. Huang, and Q. Wang. Slim UNETR: Scale Hybrid Transformers to Efficient 3D Medical Image Segmentation Under Limited Computational Resources. IEEE Transactions on Medical Imaging,

[3] S. Roy, G. Koehler, C. Ulrich, M. Baumgartner, J. Petersen, F. Isensee, P. F. Jaeger, and K. H. Maier-Hein. MedNeXt: Transformer-driven Scaling of ConvNets for Medical Image Segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2023.

Questions for authors
  1. Which split does each of Tables 1 and 5–11 use?

  2. Were any design or hyperparameter choices made using test-leaderboard results?

  3. How were the 3D MaskFormer and Mask2Former baselines tuned, and did they get the differential learning rates?

  4. How does MaskMed compare with 3D TransUNet, UNETR++, nnU-Net, and MedNeXt?

1 comment
Reviewer_i3dROpenReview ↗

Follow-up with the authors

Dear authors,

I have read the other reviewers’ comments and would appreciate a rebuttal addressing both the concerns I raised and the ones identified by the other reviewers. If these concerns remain unaddressed, I may lower my score.

Thanks!

Reviewer_EHbvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes MaskMed, a 3D medical image segmentation model that replaces the traditional multi-class segmentation head with a decoupled mask-class prediction framework. Its main innovations include Decoupled Mask–Class Prediction, a Masked Multi-Scale Segmentation Head, and a Full-Scale Deformable Transformer. The experimental results cover the AMOS22, BTCV, ACDC, and BraTS23 datasets. Overall, the paper is well-written, the model is clearly described, and the figures and tables are of high quality.

Critical strengths
  1. The paper involves abdominal CT, mixed CT/MRI modalities, cardiac MRI, and brain tumor MRI, demonstrating a certain degree of task diversity.
  2. It analyzes variants of the segmentation head, shared queries, masked attention, and feature fusion modules, and reports parameter counts, FLOPs, latency, and memory usage. These experiments help evaluate the effectiveness of the proposed method.
  3. The descriptions of different components are clear and easy to understand.
Critical weaknesses
  1. There appears to be a conflict between the experimental results reported in the main text and those in the appendix. The average results for MaskMed on the ACDC, BraTS23, and AMOS22 datasets presented in the main table are inconsistent with the corresponding results in the appendix. Additionally, the numbers for nnU-Net should also be checked for correctness and consistency.
  2. The overall method consists of three main components, and the related work discusses MaskFormer, Mask2Former, and Deformable DETR. The paper itself explicitly acknowledges being inspired by these methods. However, the newly proposed approach does not seem to introduce any substantial innovation. For ICLR, simply combining several existing designs into a new architecture typically requires clearer technical novelty or more solid mechanistic analysis.
  3. Although the experimental results are competitive, I believe there is a lack of sufficiently fair and complete comparative experiments to support the independent contribution of each module. For example, the main table for AMOS22 explicitly adopts a five-fold cross-validation ensemble, but it remains unclear whether all compared methods follow the same ensemble protocol; this needs to be more clearly stated. The paper compares MaskMed with 3D-adapted versions of MaskFormer and Mask2Former, yet lacks sufficiently detailed implementation and hyperparameter tuning information to confirm whether the comparison is fair.
Questions for authors
  1. There are clear numerical discrepancies between the main experimental tables for ACDC, AMOS22, and BraTS23 and the statistical tables in the appendix. The authors should clarify the data splits, evaluation protocols, and model ensemble strategies used for each table, and explain the source of these discrepancies.
  2. The core components of MaskMed are closely related to existing methods such as MaskFormer, Mask2Former, or deformable attention. The authors should clearly articulate the substantive technical innovations of MaskMed relative to these prior works, rather than merely presenting an architectural combination tailored for 3D medical segmentation.
  3. The inference latency of MaskMed is 182.04 ms with a peak GPU memory of approximately 4.95 GB, whereas nnU-Net achieves 32.13 ms and 1.23 GB, respectively. The authors should explain whether the additional computational overhead is justified.
  4. The main ablation experiments are conducted solely on AMOS22 Task 1, which is insufficient to demonstrate the effectiveness of shared matching supervision and the FSD-Transformer across other modalities and tasks.
Reviewer_NYmzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MaskMed for 3D medical image segmentation. It decouples binary mask prediction from semantic classification and introduces a multi-scale segmentation head with shared queries and finest-scale bipartite matching supervision. A Full-Scale Deformable Transformer is further used for multi-scale feature fusion. Experiments on AMOS22, BTCV, ACDC, and BraTS23 show improvements over several existing methods. Ablation studies are provided to evaluate the main components.

Critical strengths
  1. The method achieves competitive performance across several medical image segmentation benchmarks involving different anatomical structures and imaging modalities. The improvements over nnU-Net on AMOS22 and ACDC are encouraging.
  2. The ablation studies investigate different segmentation head designs, query sharing strategies, and feature fusion modules. These comparisons provide useful evidence for the effectiveness of the proposed components.
Critical weaknesses
  1. The methodological novelty is somewhat limited. Decoupled mask and class prediction with bipartite matching has been explored in MaskFormer, Mask2Former, and MaskSAM. Although the proposed multi-scale supervision is interesting, its contribution is not sufficiently isolated. In particular, there is no direct comparison between shared finest-scale matching and independent matching at different decoder stages.
  2. The computational cost is relatively high compared with the performance gain. According to Table 10, MaskMed requires substantially more inference time and GPU memory than nnU-Net. The FSD-Transformer further increases inference latency from 67.38 ms to 182.04 ms while improving DSC by only 0.004. This trade-off raises concerns about the practical value of the additional complexity.
Questions for authors
  1. Could the authors directly compare finest-scale shared matching with independent matching at each decoder stage? This would help clarify the contribution of the proposed supervision strategy.
  2. Table 8 shows that one query per class gives the best performance. Could the authors clarify how this setting provides more flexible semantic assignment than conventional class-specific prediction?
  3. In the BTCV statistical analysis (Table 11), three random seeds are evaluated on the same six validation cases. How was the dependence between repeated evaluations handled when calculating the p-values and confidence intervals?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.