acceptodds
Desk rejected at ICLR 2027

OT-MIMIC: Learning Medical Image Segmentation with Missing Modalities via Structure-Aware Optimal Transport

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal vision-language models (VLMs) for medical image segmentation often assume fully paired image-text data, an assumption rarely satisfied in real-world clinical datasets. In practice, large-scale medical corpora contain abundant partially observed samples and only a small subset of fully paired records. We study medical image segmentation under this incomplete multimodal setting and propose OT-MIMIC, a structure-aware transport-based completion framework that learns from partially observed data rather than discarding it. OT-MIMIC maps incomplete records to a dynamically refined memory bank and reconstructs missing representations or supervision through barycentric aggregation. We instantiate the framework with an asymmetric Fused Gromov-Wasserstein (FGW) objective in a CLIP-aligned latent space, combining semantic correspondence with relational consistency between observed-side and complementary-side structure. We additionally derive a high-probability excess-risk bound that decomposes learning error into empirical risk minimization, transport completion, recoverability, and function-approximation terms. Experiments on QaTa-COV19 and MosMedData+ show strong performance under both missing-text and missing-image-and-mask settings as complete-pair availability decreases from 50% to 1%, with the largest gains under extreme scarcity. Controlled ablations further show that the FGW instantiation improves over nearest-anchor retrieval and feature-mean completion under an otherwise fixed training pipeline.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Representation Learning, Vision-Language Models, Missing Modalities, Optimal Transport, Incomplete Multimodal Data

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Cqa4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses medical image segmentation with incomplete image, text, and mask information during training. OT-MIMIC uses Fused Gromov–Wasserstein transport and a memory bank to connect incomplete samples with complete cases, then recovers missing representations or masks by barycentric aggregation. Experiments on QaTa-COV19 and MosMedData+ cover both missing-text and missing-image-and-mask settings, together with ablation and cross-domain results. The paper also presents an excess-risk analysis for a parametric transport formulation and discusses the difference between this theoretical setting and the POT-based implementation used in the experiments.

Critical strengths
  1. OT-MIMIC provides a unified way to use samples with missing image, text, or segmentation supervision. The same transport mechanism is used to recover both missing representations and missing masks, which makes the framework more general than methods built for a single missing-data pattern.
  2. The method makes direct use of incomplete records instead of simply discarding them or relying on a separate modality generator. The transport-based completion provides a clear way to recover missing information from the distribution of complete cases.
Critical weaknesses
  1. The theoretical guarantee is derived for a parametric completion map with independent sample splitting. The main POT implementation is non-parametric, updates the memory bank across epochs, and uses the same training records for completion and ERM, so the theorem does not directly apply to the reported system.
  2. The missing-data settings are created from fully paired datasets under MCAR. In the missing-image-and-mask setting, many samples become text-only records, which may be quite different from the missingness patterns seen in real clinical data.
  3. The uncertainty-based memory-bank update is presented as an important part of OT-MIMIC, but its effect is not isolated. The current ablations mainly vary the bank size and the completion rule.
Questions for authors
  1. How much does the uncertainty-based bank update contribute when the bank size and FGW completion rule are kept fixed? Is there a matched result without uncertainty ranking?
  2. How does OT-MIMIC behave when the missingness is not MCAR, especially when image or mask availability depends on the clinical case?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Cqa4 — Part 1/2

We thank the reviewer for the careful assessment of the theoretical assumptions and for the specific questions about memory-bank selection and missing-data mechanisms. We provide fixed-capacity memory-selection ablations, lesion-burden-dependent missingness experiments, and a clarification of the theorem's scope.

1. Uncertainty-based memory-bank selection (Q1, W3)

We agree that varying the memory-bank capacity does not independently assess the effect of uncertainty-based selection. To address this concern alongside the reviewer's question about non-MCAR missingness (Q2), we conduct a matched-capacity ablation using the same simulated lesion-burden-dependent MNAR setting examined in Section 2. This allows us to evaluate the contribution of memory-bank selection when paired-data availability depends on lesion burden.

We compare uncertainty-based selection with random selection and FIFO replacement, keeping the bank capacity and FGW completion settings fixed. For all methods, the bank capacity is set to 5% of the dataset size, with updates performed after every epoch. FIFO replaces the oldest 15% of bank entries, while Random uses random selection without uncertainty ranking. All methods use identical missingness splits and training protocols. We evaluate all five availability ratios on both QaTa-COV19 and Kvasir-SEG. The following tables report Dice/IoU under image-and-mask missingness.

QaTa-COV19

CompleteOursFIFORandom
50 %0.904 / 0.8250.903 / 0.8230.902 / 0.822
25 %0.894 / 0.8090.895 / 0.8100.895 / 0.810
10 %0.842 / 0.7270.841 / 0.7260.841 / 0.725
5 %0.825 / 0.7020.812 / 0.6840.812 / 0.684
1 %0.714 / 0.5560.663 / 0.4960.681 / 0.516

Kvasir-SEG

CompleteOursFIFORandom
50 %0.876 / 0.7800.868 / 0.7670.868 / 0.766
25 %0.814 / 0.6870.803 / 0.6710.815 / 0.688
10 %0.759 / 0.6110.744 / 0.5920.749 / 0.599
5 %0.791 / 0.6550.679 / 0.5140.685 / 0.521
1 %0.535 / 0.3650.322 / 0.1920.377 / 0.232

Uncertainty-based selection achieves the highest Dice and IoU in 8 of the 10 evaluated settings. In the remaining two settings, its Dice is only 0.001 below the best alternative. The gains are particularly pronounced at low availability. At 1% on QaTa-COV19, it achieves a Dice of 0.714, compared with 0.681 for Random and 0.663 for FIFO. On Kvasir-SEG, the corresponding scores are 0.535, 0.377, and 0.322. These controlled, fixed-capacity comparisons provide evidence that uncertainty-based memory updates can improve downstream segmentation under the simulated MNAR setting, without changing bank capacity or the FGW completion rule.

(Continued in Part 2/2.)

AuthorsOpenReview ↗

Response to Reviewer Cqa4 — Part 2/2 (continued)

2. Beyond MCAR: lesion-burden-dependent MNAR (Q2, W2)

We agree that MCAR does not capture clinically dependent missingness patterns. We therefore construct the lesion-burden-dependent MNAR setting used in both the memory-selection ablation above and the following comparison. Since the datasets do not provide clinical severity labels, we use lesion area fraction as a proxy for lesion burden. Specifically, lesion burden is defined as the fraction of pixels labeled as lesion in each ground-truth mask after resizing to 224 × 224 and binarization at a threshold of 0.5.

To simulate lesion-burden-dependent MNAR, we sample complete training cases without replacement using weights proportional to exp⁡(−zi)\exp(-z_i), where ziz_i is the standardized lesion burden (λ=1\lambda=1). Thus, cases with larger lesions are less likely to retain their images and masks, while their text remains available. All methods use identical missingness splits, and validation and test sets remain unchanged. Because the selection probability depends on lesion burden computed from ground-truth masks that are subsequently withheld, this setting simulates MNAR by making image-and-mask availability depend on lesion information that is unavailable in the resulting incomplete records. Lesion burden is used only to generate missingness and is never provided as a model input.

We compare OT-MIMIC with GuideSeg, the best-performing baseline in Table 2, under simulated lesion-burden-dependent MNAR.

Simulated lesion-burden-dependent MNAR · Dice

DatasetCompleteOT-MIMICGuideSeg
QaTa-COV1950%0.9040.888
QaTa-COV1925%0.8940.866
QaTa-COV1910%0.8420.854
QaTa-COV195%0.8250.818
QaTa-COV191%0.7140.675
Kvasir-SEG50%0.8760.818
Kvasir-SEG25%0.8140.775
Kvasir-SEG10%0.7590.733
Kvasir-SEG5%0.7910.682
Kvasir-SEG1%0.5350.423

Under simulated lesion-burden-dependent MNAR, OT-MIMIC achieves higher Dice than GuideSeg in 9 of the 10 evaluated settings, including all five ratios on Kvasir-SEG. At 1% availability, OT-MIMIC achieves Dice scores of 0.714 versus 0.675 on QaTa-COV19 and 0.535 versus 0.423 on Kvasir-SEG, corresponding to improvements of 0.039 and 0.112, respectively. These results indicate that OT-MIMIC retains a comparative performance advantage under the simulated MNAR mechanism. This controlled experiment evaluates one hypothesized direction of lesion-burden-dependent selection bias, rather than a clinically validated missingness process or all possible MNAR mechanisms.

3. Theoretical guarantee and neural FGW evaluation (W1)

We appreciate the reviewer's careful distinction between the theoretical formulation and the practical implementation. We agree that the excess-risk bound does not directly apply to our non-parametric, multi-epoch POT implementation, as discussed in Appendix B.6.

Appendix G.2 (Table 7) also evaluates a neural entropic FGW estimator based on a parametric completion map and a fixed memory bank. Under the sample-splitting and bounded-complexity assumptions of Theorem 2, this parametric formulation is theory-compatible.

Table 7 compares the neural estimator with POT. At 1% image-and-mask availability, POT achieves Dice scores of 0.800 versus 0.760 for neural FGW on QaTa-COV19 and 0.566 versus 0.365 on MosMedData+. These empirical results motivate our use of POT in the main experiments, but do not extend the theoretical guarantee to its non-parametric, multi-epoch implementation.

We hope the matched ablations, simulated MNAR results, and clarified theoretical scope address the central concerns and support a more favorable assessment.

Thank you for the response. The additional experiments help, but the theory still does not cover the main implementation, and evaluation relies on simulated missingness. I maintain my rating of 2.

We thank the reviewer for the follow-up and for recognizing the value of our additional experiments.

Regarding missingness, the original benchmarks do not provide naturally incomplete records with the corresponding ground truth needed for controlled evaluation. Simulating missingness allows us to preserve the original complete observations as ground truth and systematically evaluate different missingness rates and patterns. This is particularly important for assessing modality completion and ensuring reproducible, fair comparisons across methods. Beyond the original MCAR setting, our additional lesion-burden-dependent MNAR experiments further examine robustness under a different missingness mechanism. While these experiments do not replace validation on naturally incomplete clinical data, they provide controlled evidence of the framework's effectiveness under the evaluated settings.

Regarding theory, the main purpose of our analysis is to decompose the excess risk into distinct sources associated with empirical risk minimization, transport completion, recoverability, and function approximation. This helps us understand how different components of the framework contribute to the overall error, which terms can be controlled as the sample size increases, and which depend on completion quality or model capacity. While the current bound does not directly cover the deployed multi-epoch POT implementation, its scope and the corresponding theory–practice gap are now explicitly clarified in the revised manuscript.

We believe these controlled empirical findings and the theoretical insights within their stated scope provide meaningful contributions. We would appreciate the reviewer's reconsideration of the overall assessment of our work.

Reviewer_vzrdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OT-MIMIC, a framework for medical image segmentation with missing image or text data. It uses asymmetric Fused Gromov–Wasserstein (FGW) transport and an uncertainty-refined memory bank to reconstruct missing representations and segmentation supervision.

Critical strengths
  • Asymmetric FGW combines feature similarity and relational structure for meaningful missing-modality completion.
  • It performs well under severe missingness; completion-rule ablations, memory-bank analysis, and baseline comparisons support the design.
Critical weaknesses
  • The missing-image completion process needs clarification. It is unclear how text-only samples provide the spatial features required for pixel-level segmentation. Barycentric aggregation of masks from different patients also raises potential spatial-alignment concerns.
  • The theoretical guarantee applies to an idealized parametric completion procedure rather than the main multi-epoch POT implementation.
  • Evaluation uses only two chest infection datasets. Although Kvasir-SEG appears in the appendix, broader testing across anatomies, modalities, and realistic missingness is needed.
  • Comparisons cover few baselines. Controlled evaluation against recent methods would strengthen the study, and it is highly recommended.
  • Robustness to unreliable text remains underexplored. The weakened-text experiment does not test incorrect, contradictory, or misleading clinical descriptions.
  • Only Dice and mIoU are reported. Boundary metrics such as HD95 or ASSD would provide a more complete evaluation because boundary error is critical in medical image segmentation.
Questions for authors
  • How does performance change when clinical text is incorrect, contradictory, shorter, or more detailed?
  • How are spatial features and segmentation supervision derived from text-only training samples?
  • What happens when you compare FGW with feature-only Wasserstein and GW under the same downstream segmentation pipeline?
5 comments
AuthorsOpenReview ↗

Response to Reviewer vzrd — Part 1/5

We thank the reviewer for the careful reading and constructive, specific questions. We address these points with downstream OT comparisons, inference-time text perturbations, six additional baselines, boundary metrics, and an expanded Kvasir-SEG analysis, alongside implementation and theoretical clarifications.

1. Inference-time robustness to unreliable clinical text (Q1, W5)

The original manuscript already evaluates weakened (shorter) clinical descriptions in Table 10, retaining coarse infection status and laterality while removing lesion count and anatomical location. We extend this analysis with three additional inference-time perturbations in the image-and-mask-missing setting at 1% complete-pair availability on QaTa-COV19 and MosMedData+. Training remains unchanged, and no model is retrained. Test reports are replaced with Missing (the fixed placeholder [MISS]), Incorrect (a real report randomly sampled from another case in the same dataset), or Contradictory (a real report with opposite laterality and lesion extent). Perturbations are generated with fixed random seeds. We report test Dice/IoU under original and perturbed text.

Text perturbations · 1% complete-pair availability · Dice / IoU

DatasetOriginalMissingIncorrectContradictory
QaTa-COV190.800 / 0.6670.718 / 0.5610.684 / 0.5190.586 / 0.414
MosMedData+0.566 / 0.3950.561 / 0.3900.560 / 0.3890.556 / 0.385

On QaTa-COV19, Missing, Incorrect, and Contradictory reports reduce Dice by 0.082, 0.116, and 0.214, respectively, relative to the original 0.800. On MosMedData+, the corresponding decreases are 0.005, 0.006, and 0.010 relative to 0.566. Together with the shortened-text evaluation in Table 10, these experiments provide a broader quantitative assessment of robustness to incomplete and unreliable clinical descriptions, revealing distinct sensitivity patterns across datasets and perturbation types.

2. Text-only completion: spatial inputs and mask correspondence (Q2, W1)

For text-only records, the observed text embedding is the FGW query against complete memory-bank entries containing image and mask information. The same transport weights barycentrically aggregate stored images and masks into a corresponding pseudo-image and pseudo-mask. The pseudo-image is processed by the ConvNeXt spatial encoder, and the pseudo-mask supplies segmentation supervision. This couples the pseudo-image and pseudo-mask during completion, although no explicit anatomical registration is performed across patients. This pathway is now clarified in Section 3.2.

(Continued in Part 2/5.)

AuthorsOpenReview ↗

Response to Reviewer vzrd — Part 2/5 (continued)

3. FGW versus Wasserstein OT and GW: downstream comparison (Q3)

We thank the reviewer for requesting a downstream comparison of FGW with W-only and GW-only. As detailed in our response to Reviewer CX1u (Section 3), we conduct a controlled inference-only completion diagnostic across three datasets using the same frozen segmentation models and aligned retrieval features. We compare W-only (α=0\alpha=0), FGW (α=0.3\alpha=0.3), and GW-only (α=1\alpha=1) under progressively stronger rotations of query retrieval embeddings, while preserving within-domain structural relationships. FGW achieves the highest Dice in 6/9 dataset–shift settings, including all three datasets at moderate shift (t=0.3t=0.3). Under stronger shifts, GW-only becomes preferable on Kvasir-SEG and MosMedData+. These results illustrate the trade-off between feature-based and structural matching rather than consistent superiority of any single transport formulation.

4. Scope of the theoretical guarantee (W2)

We agree that Theorem 2 does not directly cover the non-parametric, multi-epoch POT implementation, as discussed in Appendix B.6. The theorem considers a bounded-complexity parametric completion map with a fixed memory bank and independent sample splitting. Appendix G.2 (Table 7) evaluates a neural entropic FGW estimator based on a parametric completion map and fixed bank. Under the stated sample-splitting and complexity assumptions, this formulation is compatible with the theoretical analysis. At 1% image-and-mask availability, POT achieves Dice 0.800 versus 0.760 for neural FGW on QaTa-COV19 and 0.566 versus 0.365 on MosMedData+. These comparisons motivate our empirical use of POT, but do not extend the theorem's guarantee to that implementation.

5. Additional evaluations: Kvasir-SEG and simulated MNAR (W3)

Beyond the original Kvasir-SEG Dice results, we add three-seed, recent-baseline, and boundary evaluations for endoscopic polyp segmentation. OT-MIMIC's three-seed mean Dice exceeds GuideSeg at all five availability ratios, including 0.704 ± 0.0055 versus 0.606 ± 0.0363 at 1% (Reviewer CX1u, Section 4). Against six additional recent baselines, OT-MIMIC leads at 50%, 10%, and 5%, and is within 0.005 of the best at 25% and 1% (this response, Section 6). At 1%, it also achieves the lowest ASSD of 17.73 px, compared with 22.84 px for the next-best method (this response, Section 7).

We additionally evaluate simulated lesion-burden-dependent MNAR on QaTa-COV19 and Kvasir-SEG, where the probability of retaining image-and-mask pairs decreases with lesion burden. OT-MIMIC outperforms GuideSeg in 9/10 settings, including Dice scores of 0.714 versus 0.675 on QaTa-COV19 and 0.535 versus 0.423 on Kvasir-SEG at 1% availability. The full MNAR protocol and results are provided in our response to Reviewer Cqa4, Section 2. These experiments broaden our evaluation across imaging contexts and missingness mechanisms, while validation under real-world clinical missingness remains an important direction for future work.

(Continued in Part 3/5.)

AuthorsOpenReview ↗

Response to Reviewer vzrd — Part 3/5 (continued)

6. Comparisons with recent segmentation baselines (W4)

We address the request for stronger comparisons by including six recent methods from NeurIPS 2026 and MICCAI 2026: RADAR [1], DD-CMD [2], RCA [3], MedTSCNet [4], TIRNet [5], and TMF-Seg [6]. All additional baselines follow the same experimental protocol described in the original manuscript, ensuring consistent data splits, missingness settings, and model evaluation. We report Dice/IoU for these six methods, MedCLIPSeg, and OT-MIMIC across all five availability ratios.

QaTa-COV19 · Dice / IoU

Method50%25%10%5%1%
RADAR0.878 / 0.7830.875 / 0.7770.840 / 0.7240.790 / 0.6530.617 / 0.447
DD-CMD0.899 / 0.8160.885 / 0.7940.864 / 0.7610.842 / 0.7280.767 / 0.622
RCA0.897 / 0.8140.883 / 0.7910.865 / 0.7620.836 / 0.7190.737 / 0.584
MedTSCNet0.895 / 0.8100.883 / 0.7900.868 / 0.7660.846 / 0.7330.716 / 0.557
TIRNet0.891 / 0.8030.877 / 0.7810.854 / 0.7450.812 / 0.6840.604 / 0.433
TMF-Seg0.866 / 0.7640.855 / 0.7460.816 / 0.6900.769 / 0.6250.061 / 0.032
MedCLIPSeg0.892 / 0.8040.885 / 0.7930.858 / 0.7510.842 / 0.7270.788 / 0.650
OT-MIMIC0.907 / 0.8300.903 / 0.8230.879 / 0.7850.860 / 0.7550.800 / 0.667

MosMedData+ · Dice / IoU

Method50%25%10%5%1%
RADAR0.709 / 0.5500.694 / 0.5320.401 / 0.2510.221 / 0.1240.067 / 0.035
DD-CMD0.757 / 0.6090.742 / 0.5900.709 / 0.5490.678 / 0.5130.504 / 0.337
RCA0.744 / 0.5920.736 / 0.5820.714 / 0.5550.660 / 0.4930.487 / 0.322
MedTSCNet0.733 / 0.5780.720 / 0.5620.698 / 0.5360.662 / 0.4950.393 / 0.244
TIRNet0.708 / 0.5480.697 / 0.5350.556 / 0.3850.422 / 0.2670.007 / 0.004
TMF-Seg0.758 / 0.6100.735 / 0.5820.607 / 0.4350.469 / 0.3070.000 / 0.000
MedCLIPSeg0.659 / 0.4910.626 / 0.4550.587 / 0.4150.519 / 0.3500.206 / 0.115
OT-MIMIC0.769 / 0.6240.752 / 0.6030.711 / 0.5520.684 / 0.5190.566 / 0.395

Kvasir-SEG · Dice / IoU

Method50%25%10%5%1%
RADAR0.897 / 0.8140.894 / 0.8090.717 / 0.5590.591 / 0.4190.564 / 0.393
DD-CMD0.886 / 0.7960.873 / 0.7740.810 / 0.6810.786 / 0.6470.704 / 0.543
RCA0.879 / 0.7840.850 / 0.7390.786 / 0.6480.763 / 0.6170.445 / 0.286
MedTSCNet0.888 / 0.7980.852 / 0.7420.815 / 0.6880.757 / 0.6090.664 / 0.497
TIRNet0.772 / 0.6280.654 / 0.4860.553 / 0.3820.494 / 0.3280.004 / 0.002
TMF-Seg0.817 / 0.6910.680 / 0.5150.500 / 0.3330.395 / 0.2460.114 / 0.061
MedCLIPSeg0.858 / 0.7510.828 / 0.7060.779 / 0.6380.750 / 0.6000.607 / 0.436
OT-MIMIC0.910 / 0.8360.889 / 0.7990.850 / 0.7390.805 / 0.6740.700 / 0.538

Among the methods included in the tables above, OT-MIMIC achieves the highest Dice in 12 of the 15 dataset–availability comparisons, including both chest datasets at 1% complete-pair availability. In the remaining three settings, its Dice is within 0.005 of the best method shown.

(Continued in Part 4/5.)

Reviewer_CX1uOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies medical image segmentation with incomplete image, text, and mask data during training. The authors propose OT-MIMIC, which uses a memory bank and Fused Gromov-Wasserstein (FGW) optimal transport to complete missing features or generate pseudo-masks. FGW considers both feature similarity and structural relationships between samples. The paper also provides a theoretical analysis of an idealized version of the completion framework. Experiments on QaTa-COV19 and MosMedData+ cover missing-text and missing-image-and-mask settings, with complete-pair availability ranging from 50% to 1%. The results show that OT-MIMIC generally outperforms the compared methods, especially when complete training pairs are scarce. Additional experiments examine different completion strategies, memory bank capacity, and performance on Kvasir-SEG.

Critical strengths

(1) The paper addresses a practical problem in medical image segmentation, where training samples may have missing modalities or labels. The results in Tables 1 and 2 show that OT-MIMIC outperforms the compared methods in most settings, especially when complete training pairs are scarce. (2) Table 6 also provides a useful controlled comparison. With the other training components kept unchanged, FGW outperforms nearest-anchor and feature-mean completion on QaTa-COV19.

Critical weaknesses

(1) Although Table 6 shows that FGW outperforms nearest-anchor and feature-mean completion, it does not clearly show the contribution of the structural term. Appendix C compares Wasserstein, GW, and FGW only at the representation level, without evaluating their downstream segmentation performance under the same settings. A direct comparison would provide stronger evidence for the use of FGW. The main results follow a fixed-seed protocol, without reported multi-seed means or standard deviations, so the variability of the gains, especially at 1% availability, remains unclear. (2) Some important implementation details need clarification. For text-only samples, Appendix E explains how completed CLIP embeddings condition the decoder, but the input to the spatial encoder and its correspondence with the aggregated pseudo-mask remain unclear. In the mask-only-missing setting, neither image nor text is missing, while masks are excluded from the FGW structure matrices. The paper does not clearly specify how the target-side structural matrix in Eq. (5) is defined for this case.

Questions for authors
  1. For text-only records, what input is supplied to the ConvNeXt U-Net spatial encoder, or is this branch bypassed? Please explain how the spatial input or features are constructed and paired with the aggregated pseudo-mask.
  2. In the mask-only-missing setting, the paper describes how transport weights are used to aggregate bank masks, but the target-side structural matrix in Eq. (5) remains unclear: both image and text are observed, while masks are excluded from the structure matrices.
  3. Could the authors provide a direct comparison between feature-only Wasserstein OT and FGW on downstream segmentation? The representation-level comparison in Appendix C is useful, but it does not clearly show whether the structural term improves segmentation performance.
3 comments
AuthorsOpenReview ↗

Response to Reviewer CX1u — Part 1/3

We sincerely thank the reviewer for the positive assessment and encouraging recommendation, and for recognizing the importance of learning from incomplete multimodal records. We address the specific questions through implementation clarifications, controlled downstream comparisons of transport variants, and three-seed evaluations.

1. Text-only records: spatial inputs and segmentation supervision (Q1, W2)

For text-only records, the observed text embedding serves as the query representation for FGW retrieval against memory-bank entries containing paired image and mask information. The same row-normalized transport weights barycentrically aggregate the stored images and masks into a corresponding pseudo-image and pseudo-mask. The pseudo-image is passed to the ConvNeXt spatial encoder, while the pseudo-mask provides segmentation supervision. Using the same memory-bank entries and transport weights constructs the pseudo-image and pseudo-mask from corresponding weighted pairs, without explicit anatomical registration across patients. This completion pathway is now clarified in Section 3.2.

2. Mask-only missingness: joint memory-bank structure (Q2, W2)

The mask is treated as another modality within the same FGW-based completion framework. The source representation can use one or more observed modalities, while the target structure consistently uses the joint image–text–mask representation stored in the memory bank across all missingness patterns. For mask-only missingness, both image and text are observed, and our implementation uses the image embedding as the source. FGW weights then barycentrically aggregate the stored masks to estimate the missing mask without accessing its ground truth. This construction is now clarified around Eq. (5).

(Continued in Part 2/3.)

AuthorsOpenReview ↗

Response to Reviewer CX1u — Part 2/3 (continued)

3. Contribution of FGW: unified Wasserstein/GW formulation (Q3, W1)

FGW combines direct feature matching (W-only, α=0\alpha=0) with relationships among samples (GW-only, α=1\alpha=1). In QaTa-COV19 ablations, FGW leads or ties in 7/10 Dice and 6/10 IoU settings, but differences are small. We test when each helps.

Setup. We freeze models trained with 10% complete pairs on QaTa-COV19, MosMedData+, and Kvasir-SEG. For every validation/test case, we hide its image and mask. From text alone, OT-MIMIC averages training-bank images using transport weights, predicts a mask, and scores it against the hidden true mask. No retraining is involved.

Controlled rotation. We aim to simulate a situation where direct feature matching becomes increasingly misleading, while the relationships among queries remain unchanged. First, we identify adversarial query-to-bank matches by finding a balanced assignment that minimizes the similarity between their ground-truth masks:

Padv=arg⁡min⁡P∈Π(a,b)∑i,kPikDice⁡(mi,mk)P_{\mathrm{adv}}=\arg\min_{P\in\Pi(a,b)}\sum_{i,k}P_{ik}\operatorname{Dice}(m_i,m_k),

where Π(a,b)\Pi(a,b) denotes transport plans with uniform marginals. We then construct an orthogonal rotation Radv∈SO(512)R_{\mathrm{adv}}\in SO(512) that brings query embeddings as close as possible to the bank feature directions associated with these misleading matches. Rather than applying the full rotation immediately, we gradually increase its strength:

qi(t)=Rtqi,Rt=exp⁡(tlog⁡Radv),t∈[0,1]q_i^{(t)}=R_tq_i,\quad R_t=\exp(t\log R_{\mathrm{adv}}),\quad t\in[0,1].

At t=0t=0, features remain unchanged. At t=1t=1, the full adversarial rotation is applied. Importantly, because RtR_t is orthogonal, we have ∣Rtqi−Rtqj∣22=∣qi−qj∣22|R_tq_i-R_tq_j|_2^2=|q_i-q_j|_2^2. Thus, the rotation changes cross-domain feature similarities used by W-only while preserving the within-domain distances used by GW-only. Only retrieval embeddings are rotated, and the segmentation model remains fixed. Ground-truth masks are used solely to construct this synthetic perturbation, not for modality completion.

Comparison. Query and memory bank features are aligned in a shared space. We compare fixed α=0\alpha=0 (Wasserstein-Only), 0.30.3 (FGW), and 11 (Gromov-Wasserstein), choose regularization on validation, and report pooled TEST Dice for completed test cases' segmentation prediction.

DatasetShift ttW-only (α=0\alpha=0)FGW (α=0.3\alpha=0.3)GW-only (α=1\alpha=1)
QaTa-COV190.00.4940.4980.456
0.30.4910.4970.456
1.00.4630.4640.456
MosMedData+0.00.1500.1720.112
0.30.1530.1740.112
1.00.0630.0470.112
Kvasir-SEG0.00.4600.4450.442
0.30.4470.4490.442
1.00.4080.4180.442

Interpretation. Three regimes emerge:

  1. Reliable cross-domain features: At t=0t=0, W-only or intermediate FGW works well because feature costs are informative.
  2. Intermediate feature degradation: At t=0.3t=0.3, fixed-α=0.3\alpha=0.3 FGW outperforms both endpoints on all three datasets , showing a benefit from combining both channels.
  3. Strongly misleading features: GW-only becomes preferable on Kvasir and MosMedData+ under stronger feature corruption, while QaTa does not reach a GW-favourable regime.

Implication. This experiment illustrate the trade-off between feature-based and structure-based matching. In real-world settings, both the degree of cross-domain feature mismatch and the informativeness of within-domain structural relationships can vary. By combining these two sources of information, FGW offers a flexible framework that can achieve a promising balance across different regimes along this trade-off, rather than relying exclusively on either feature or structural matching.

(Continued in Part 3/3.)

AuthorsOpenReview ↗

Response to Reviewer CX1u — Part 3/3 (continued)

4. Multi-seed evaluation across availability ratios (W1)

We appreciate the concern regarding the variability of results under severe missingness. To assess the stability of our method, we conducted three-seed experiments (42, 123, and 2026) and compared OT-MIMIC against GuideSeg, the best-performing baseline in Table 2. The per-seed Dice/IoU scores and three-seed Dice means and standard deviations are reported below.

QaTa-COV19 · Three-seed results (Dice / IoU)

CompleteMethodSeed 42Seed 123Seed 2026Dice mean ± std
50%GuideSeg0.897 / 0.8130.898 / 0.8150.893 / 0.8060.896 ± 0.0030
50%OT-MIMIC0.907 / 0.8300.903 / 0.8230.905 / 0.8260.905 ± 0.0021
25%GuideSeg0.888 / 0.7980.884 / 0.7920.887 / 0.7970.886 ± 0.0021
25%OT-MIMIC0.903 / 0.8230.902 / 0.8210.897 / 0.8140.901 ± 0.0030
10%GuideSeg0.879 / 0.7840.868 / 0.7660.868 / 0.7670.872 ± 0.0063
10%OT-MIMIC0.879 / 0.7850.875 / 0.7780.877 / 0.7810.877 ± 0.0021
5%GuideSeg0.862 / 0.7570.840 / 0.7240.845 / 0.7310.849 ± 0.0116
5%OT-MIMIC0.860 / 0.7550.853 / 0.7440.855 / 0.7470.856 ± 0.0036
1%GuideSeg0.732 / 0.5780.724 / 0.5670.759 / 0.6110.738 ± 0.0183
1%OT-MIMIC0.800 / 0.6670.789 / 0.6510.797 / 0.6620.795 ± 0.0058

MosMedData+ · Three-seed results (Dice / IoU)

CompleteMethodSeed 42Seed 123Seed 2026Dice mean ± std
50%GuideSeg0.748 / 0.5970.744 / 0.5920.747 / 0.5960.746 ± 0.0022
50%OT-MIMIC0.769 / 0.6240.760 / 0.6130.765 / 0.6200.765 ± 0.0044
25%GuideSeg0.748 / 0.5980.737 / 0.5840.742 / 0.5900.743 ± 0.0055
25%OT-MIMIC0.752 / 0.6030.737 / 0.5840.751 / 0.6010.747 ± 0.0082
10%GuideSeg0.729 / 0.5730.721 / 0.5630.709 / 0.5490.719 ± 0.0098
10%OT-MIMIC0.711 / 0.5520.707 / 0.5470.700 / 0.5390.706 ± 0.0056
5%GuideSeg0.679 / 0.5140.690 / 0.5260.647 / 0.4790.672 ± 0.0219
5%OT-MIMIC0.684 / 0.5190.706 / 0.5460.637 / 0.4670.676 ± 0.0355
1%GuideSeg0.512 / 0.3440.590 / 0.4190.243 / 0.1380.448 ± 0.1823
1%OT-MIMIC0.566 / 0.3950.614 / 0.4430.280 / 0.1630.487 ± 0.1810

Kvasir-SEG · Three-seed results (Dice / IoU)

CompleteMethodSeed 42 (D/I)Seed 123 (D/I)Seed 2026 (D/I)Dice mean ± std
50%GuideSeg0.899 / 0.8170.871 / 0.7720.877 / 0.7810.882 ± 0.0147
50%OT-MIMIC0.910 / 0.8360.899 / 0.8170.904 / 0.8240.904 ± 0.0055
25%GuideSeg0.859 / 0.7530.853 / 0.7430.848 / 0.7370.853 ± 0.0055
25%OT-MIMIC0.889 / 0.7990.884 / 0.7930.874 / 0.7760.882 ± 0.0076
10%GuideSeg0.794 / 0.6590.807 / 0.6760.828 / 0.7060.810 ± 0.0172
10%OT-MIMIC0.850 / 0.7390.822 / 0.6980.871 / 0.7710.848 ± 0.0246
5%GuideSeg0.772 / 0.6290.739 / 0.5860.783 / 0.6440.765 ± 0.0229
5%OT-MIMIC0.805 / 0.6740.798 / 0.6630.846 / 0.7330.816 ± 0.0259
1%GuideSeg0.565 / 0.3940.619 / 0.4480.634 / 0.4640.606 ± 0.0363
1%OT-MIMIC0.700 / 0.5380.701 / 0.5400.710 / 0.5500.704 ± 0.0055

Per-seed scores are displayed to three decimal places, while means and standard deviations are calculated at higher precision.

OT-MIMIC achieves higher mean Dice than GuideSeg in 14 of the 15 dataset–availability settings. Notably, at 1% complete-pair availability, OT-MIMIC outperforms GuideSeg in all nine seed-wise comparisons across QaTa-COV19, MosMedData+, and Kvasir-SEG. These results demonstrate that the comparative advantage under severe missingness is not limited to a single random seed.

We appreciate the reviewer's encouraging assessment. We hope the added evidence and clarifications address the remaining concerns and support an even more favorable evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.