acceptodds
Under review as a conference paper at ICLR 2027

THEME: Task-Hierarchical Evidence Completion for In- complete Multimodal Brain Tumor Segmentation

Abstract

An unavailable MRI sequence removes information needed for brain tumor segmentation, but its effect differs across tumor regions. We propose THEME, which predicts a separate learned summary for each missing sequence and each of three region tasks. Modality-specific encoders extract spatial features and task summaries; a completion module estimates the summaries of missing sequences from those observed. One router fuses only acquired spatial features, and a second combines acquired and predicted summaries for each task before decoder modulation. We evaluate every one of the 15 non-empty subsets of four MRI sequences. On BraTS 2018, THEME obtains 88.44/81.16/64.32% Dice for whole tumor/tumor core/enhancing tumor, improving on the strongest baseline by 1.50/0.35/1.87 points. The corresponding BraTS 2020 scores are 89.65/83.04/68.16%, with gains of 1.49/0.36/1.87 points. Removing direct summary supervision reduces enhancing-tumor Dice by 5.78 points. These experiments show the benefit of the complete architecture under simulated missing-sequence evaluation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Missing-modality Learning, Multimodal MRI, Brain Tumor Segmentation, Task-conditioned Feature Completion, Modality Routing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SXHYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

THEME addresses brain tumour segmentation when 1,2 or 3 of the four MRI sequences (FLAIR, T1, T1ce, T2) are missing. Instead of synthesising missing images or completing a shared latent code, each modality expert produces a spatial feature map and three 64d summaries z_{m,t}, one per region task (ET, TC/ET, WT/TC), via query-based pooling biased by region masks (GT in training, detached expert predictions at inference). A cross-attention module predicts the summaries of missing modalities, with heads evaluated in order.

A global router fuses observed spatial maps; a task router fuses observed and completed summaries per task; each summary drives a spatial gate and residual in a shared decoder whose outputs are summed into ET/TC/WT logits.

Training uses two nested availability subsets per subject, with Dice+BCE on both views, an L1+cosine reconstruction loss against detached full-input summaries, ranking losses on outputs and evidence, and gate regularizers.

Evaluation covers all 15 non-empty modality subsets on BraTS 2018/2020 with 5 seeds, cumulative and leave-one-out ablations, an efficiency audit, and a Prostate158 study.

Critical strengths

THEME predicts a small summary per missing sequence and tumor region, supervised by what the full-input expert would produce. Spatial detail comes only from acquired sequences, so the model never relies on synthesized voxels. Indexing by region fits the problem, since a missing sequence affects ET and WT differently. The added cost is small (0.74M parameters, 40.7 GFLOPs).

Moreover the authors contribute with an informative evaluation design. All 15 subsets, two releases, five seeds, separate WT/TC/ET scores and a severe-missing summary. Seed and subset variation are kept apart, and the per-subset results show that which sequence is available matters, not only how many.

Critical weaknesses

W1: The paper does not test its main idea directly. A test of this kind would replace the three per-task summaries with one summary per modality and keep everything else the same. In Table 2, the last row only adds the training losses, some of which follow SimMLM [11], and it still gains +3.57 WT and +3.55 TC on BraTS 2018. So it is not clear how much of the gain comes from the architecture and how much from the training.

Table 2 names the module added at each step, but it does not describe the Baseline itself, how missing summaries are handled before EC is added, or which training objective the intermediate rows use. Without this, the gains of each step are hard to interpret.

Some ablations are missing. First, a missing slot is filled with a fixed mix of the mean of the observed summaries and the predicted summary, so it is the mean plus a down-weighted learned correction. Results with the mean alone are not reported, so the value of the completion module is unclear, and the mixing weights are not justified. Second, summaries are pooled with ground-truth masks in training but predicted masks at test time. Testing with ground-truth masks would show how much this costs. Third, the error of the predicted summaries is never reported, for example as a function of |A|.

The paper contrasts its approach with image synthesis, but the only synthesis-based methods (Hyper-GAE, MouseGAN++) appear in the appendix. A two-stage baseline, which synthesizes the missing sequences with a many-to-many model (e.g., a conditional diffusion model) and then segments the completed volume with a strong segmenter, would directly test whether completing compact summaries is better than completing images.

The low similarities in Table 5 are expected, since each summary is pooled over a different region with its own query. They do not show that per-task summaries help segmentation.

W2: Some per-subset values are hard to interpret under the subset definitions in Table S1. In Table S11 (BraTS 2020 ET), THEME obtains about 80 on subsets without T1ce (T1-only, T1+T2, FLAIR+T1+T2), close to its full-input score, but 57.0 to 62.7 on five of the eight subsets that include T1ce. Similar cases appear in BraTS 2018 TC subset 11 and BraTS 2020 WT FLAIR-only. Across baselines, the single-sequence subset with the highest ET also falls in different columns. If the rows follow different subset orderings, the per-subset comparisons and the analysis in Section 4.4 cannot be read as reported.

The paper does not state whether the baselines were re-trained or taken from earlier work. The BraTS 2018 per-subset values of the baselines shared with AdaMM [29] appear to match those reported in the AdaMM work, which used a different split and training budget. This would conflict with the statement in 4.1 that all methods share the same split and evaluation, and it is unclear how the five-seed standard deviations in Table 1 were obtained for these methods.

Eleven of the 18 baselines appear only in the appendix, and some score above main-table entries (e.g., SMUNet's ET of 58.07). TMFormer, SMML, UniMRSeg, D3Seg, SIUM, and Konwer et al. are cited but not evaluated on BraTS. DC-Seg and M3AE are evaluated only on Prostate158, and PASSION and LS3M are not discussed. A modality-dropout U-Net is also missing.

Only Dice is reported, with no paired test, and the TC gain is within seed variation. The data splits are not stated, and it is unclear whether the 128^3 center crop used for validation is also applied at test time.

Note: The evaluation relies on BraTS 2018 and 2020, which are small by current standards and partly overlap, since the 2020 release includes most 2018 cases. Recent releases provide substantially larger cohorts (e.g., over 1200 pre-treatment training cases in BraTS 2023/2024) and have adopted lesion-wise Dice and HD95 as standard metrics. In addition, gains on pre-treatment glioma segmentation with complete inputs have become small in recent challenges, and the community has moved toward settings such as post-treatment glioma, where segmentation is harder and clinically more relevant. Evaluating on a recent release with lesion-wise metrics, and ideally on a post-treatment cohort, would strengthen the relevance of the contribution.

Questions for authors
  1. Do all rows of the per-subset tables follow the subset order of Table S1? If not, please provide corrected tables.
  2. Which baselines were re-trained under your protocol, and which were taken from earlier work? For the latter, what do the standard deviations in Table 1 measure?
  3. For every method, the BraTS 2020 ET values differ from the BraTS 2018 values by a nearly constant offset, and the across-subset standard deviations are the same. How were the BraTS 2020 results produced?
  4. What does the Table 2 Baseline contain? Can you report the ablations listed in W1, in particular a single shared summary per modality?
  5. What is the training cost per epoch compared with the baselines?
Reviewer_V7K5OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a framework for brain tumor segmentation with missing modalities. Based on the definitions of three distinct tumor regions, the authors introduce the concept of task-specific evidence and use three types of features to disentangle the characteristics of these regions. By exploiting their task-specific differences, the framework combines missing-evidence completion with feature aggregation to progressively segment the three tumor regions under incomplete-modality settings. The method is reasonable and clearly motivated, and the experimental evaluation is generally comprehensive.

Critical strengths

1.The proposed framework is coherent and well designed. It effectively connects the missing-modality setting with the distinct characteristics of different tumor segmentation tasks, enabling the segmentation of three tumor regions.

2.The experiments are comprehensive and provide strong support for the proposed method.

Critical weaknesses

1a. Although the overall framework is coherent, several key components are built upon standard attention mechanisms, while region-specific modality aggregation has also been explored in prior work such as RFNet[1]. The paper would benefit from a clearer articulation of the methodological insight that distinguishes THEME from these related approaches.

1b. The experimental comparison omits several recent methods, such as Virtual Nodes GNN[2] and UniME[3]. Including at least one recently published 2026 method would make the empirical evaluation more convincing.

  1. The detailed modality ablations in the supplementary material contain several counterintuitive results that deserve further analysis. In Tables S9 and S11, using T1ce alone on BraTS 2020 produces relatively low TC and ET performance, particularly an ET Dice of only 0.608, despite T1ce generally being the most informative modality for enhancing tumor segmentation. This result is substantially worse than that obtained using T1 alone. Moreover, combining T1ce with T1 reduces the Dice score by approximately 0.17 compared with using T1 alone. A similar non-monotonic result appears for BraTS 2018 ET in Table 5, where combining T1ce with T2 performs considerably worse than using T1ce alone. These cases should be discussed in greater detail, including whether the degradation originates from evidence completion, modality routing, spatial gating, or the training strategy.
Questions for authors

Please refer to the weaknesses discussed above.

[1] Ding, Yuhang, Xin Yu, and Yi Yang. "RFNet: Region-aware fusion network for incomplete multi-modal brain tumor segmentation." ICCV 2021. [2] Tao, Sha, et al. "Virtual Nodes Guided Dynamic Graph Neural Network for Brain Tumor Segmentation with Missing Modalities." CVPR 2026. [3] Song, Peibo, et al. "Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities." CVPR 2026

Reviewer_CwrROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

THEME predicts region-specific summaries of missing MRI sequences and uses these alongside acquired spatial features for tumour segmentation. It reports gains across all 15 modality subsets on two BraTS releases, with additional prostate results and component ablations.

Critical strengths

Completing missing information separately for each tumour region makes sense. The WT/ET improvements and severe-missingness results suggest this is useful beyond simply adding another fusion block.

I appreciate the exhaustive modality-subset evaluation and five-run reporting. The prostate experiment adds some evidence beyond brain tumours.

Critical weaknesses

The pooling uses ground-truth region masks during training but predicted expert masks at inference. How those expert segmentation heads are trained is unclear in the stated objective. This is a central dependency, and the train–test mismatch needs checking.

The evaluation is underspecified: subject counts and train/validation/test assignments are missing. The cumulative ablations also do not isolate hierarchical completion from equally sized task-independent or unordered alternatives.

Questions for authors

Which loss supervises each expert’s mask predictions?

What happens when predicted masks also replace ground-truth pooling masks during training?

Please specify the subject splits, checkpoint-selection rule and whether every baseline was retrained under that protocol.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.