- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses missing modalities in multimodal MRI brain tumor segmentation. The authors train a Dirichlet-based uncertainty estimator using the difference between predictions under full and partial observations, and evaluate segmentation performance on BraTS2020 and BraTS2024. However, the uncertainty evaluation and some experimental details require further clarification.
Critical strengths
The paper observes that missing modalities can substantially change predictions even when model confidence remains high. Studying their effect on the current prediction is worthwhile. The theoretical analysis also motivates the training objective: the authors distinguish the prediction change for an individual sample from the expected change given a partial observation, and explain why the former can serve as a supervision signal for uncertainty learning under squared loss.
Critical weaknesses
- The segmentation setup and the stability of the results need further clarification.
- For BraTS2024, the paper reports only WT, TC, and ET results. It does not explain how the resection cavity (RC) label is handled or report segmentation results for RC.
- The main text does not report results across multiple random seeds, variances, or confidence intervals. For modality combinations with small Dice differences, it remains unclear whether the improvements are consistent.
- The current evidence does not fully establish the utility of the uncertainty estimates.
- The uncertainty evaluation primarily measures agreement with the training target. RMSE and UCE compare the estimated uncertainty with the difference between model predictions under full and partial observations. This shows that the model fits the proposed proxy target, but does not establish whether its uncertainty estimates identify actual segmentation errors or reflect clinical risk.
- Competitive uncertainty estimation baselines are missing. The learned concentration is compared only with several fixed concentrations, making it difficult to assess its advantage over predictive entropy, model ensembles, or other uncertainty estimation methods.
Questions for authors
I would appreciate clarification on the weaknesses above, particularly the following:
- How is the RC label handled in the BraTS2024 experiments?
- Can the authors compare their approach with existing uncertainty estimation methods and assess whether its uncertainty estimates predict segmentation errors or performance degradation caused by missing modalities?
If these concerns are adequately addressed, I would be willing to consider raising my rating during the rebuttal period.
2 comments
I thank the authors for their response to my review comments and for their detailed clarification of the relevant points. Their response has helped me better understand the paper. After considering it carefully, I believe that my current overall rating still reflects my assessment of the paper, so I am inclined to maintain my original rating.
Thank you for your thoughtful review. We address your concerns as follows.
Q1.How is the RC label handled in the BraTS2024 experiments?
A1. In our experiments, the RC label was mapped to the background and excluded from evaluation.
Q2.Can the authors compare their approach with existing uncertainty estimation methods and assess whether its uncertainty estimates predict segmentation errors or performance degradation caused by missing modalities?
A2. We compared our method with two uncertainty estimation approaches: predictive entropy and maximum softmax probability (MSP). We evaluated segmentation error detection and risk–coverage performance using AUROC, average precision (AP), and the area under the risk–coverage curve (AURC). Experiments covered four single-modality settings, with results averaged across the WT, TC, and ET regions. Segmentation errors were identified by comparing predicted and ground-truth region masks. The results are presented in the table below and have been added to the revised manuscript.
\hline \mathrm{Metric} & \mathrm{Entropy} & \mathrm{MSP} & \mathrm{Ours} \\\\ \hline \mathrm{AUROC}\uparrow & 0.5795 & 0.6084 & \mathbf{0.6163} \\\\ \hline \mathrm{AP}\uparrow & 0.6326 & 0.6645 & \mathbf{0.6753} \\\\ \hline \mathrm{AURC}\downarrow & 0.5651 & 0.5541 & \mathbf{0.5511} \\\\ \hline \end{array}$$ Q3.The main text does not report results across multiple random seeds, variances, or confidence intervals. For modality combinations with small Dice differences, it remains unclear whether the improvements are consistent. **A3.** Due to time constraints, we could not repeat all experiments. **We conducted three additional runs with different random seeds on BraTS2024 and compared their mean and standard deviation with the original results.** We report our results averaged over three random seeds alongside the original results of the competing methods, as summarized in the table below. $$\begin{array}{l|c|c|c|c} \hline \mathrm{Method} & \mathrm{WT} & \mathrm{TC} & \mathrm{ET} & \mathrm{Mean} \\\\ \hline \mathrm{RFNet} & 78.59 & 47.82 & 47.04 & 57.82 \\\\ \hline \mathrm{mmFormer} & 77.35 & 42.84 & 41.56 & 53.92 \\\\ \hline \mathrm{IMS}^{2}\mathrm{Trans} & 75.65 & 49.69 & 47.87 & 57.73 \\\\ \hline \mathrm{M}^{2}\mathrm{FTrans} & 74.48 & 47.49 & 46.79 & 56.25 \\\\ \hline \mathrm{MSTKD} & 80.52 & 48.84 & 48.02 & 59.13 \\\\ \hline \mathrm{Ours} & \mathbf{83.42\pm0.58} & \mathbf{51.18\pm0.87} & \mathbf{51.25\pm1.01} & \mathbf{61.95\pm0.48} \\\\ \hline \end{array}$$ **Overall, our method achieves competitive performance across all three tumor regions and the highest mean Dice score, demonstrating a favorable overall performance advantage over the compared methods.**