- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes DECENT for multimodal learning with missing modalities. The key motivation is that modality-level graph construction may cause false grouping of heterogeneous signals within the same modality and false separation of related signals across different modalities. DECENT decomposes each modality into index-aligned proto-tokens, learns a signal-level partition using cross-modal co-variation, and then performs prediction with view-specific graph experts and availability-aware fusion.
Critical strengths
-
The motivation of false grouping and false separation is intuitive and clearly explains why modality-level graph construction can be suboptimal under missing modalities.
-
The partition ablation is relatively convincing. With the tokenizer and graph encoder kept fixed, the learned partition consistently outperforms modality-based, balanced-random, and single-view partitions in all six missing-modality settings. This helps isolate the benefit of the learned signal grouping rather than simply adding more graph experts. The two-stage training design is supported by a useful ablation. Freezing the learned partition after Phase 1 performs better than task-only training in all six settings and better than Phase-2 fine-tuning in five of six settings, showing that directly optimizing the partition with task labels is not necessarily beneficial.
-
The paper identifies a concrete limitation of modality-level graph construction, namely false grouping within a modality and false separation across modalities. More importantly, the experiments are consistent with this motivation: compared with MUSE, the advantage of DECENT becomes clearer as the missing rate increases, and its AUC drops less from to on both ADNI and MIMIC-IV.
Critical weaknesses
W1 - The core assumption that cross-modal co-variation corresponds to a meaningful shared latent process is relatively strong. Cross-modal correlation may also arise from disease severity or other shared confounding factors, so the semantic interpretation of the learned views is not fully justified.
W2 - The evaluation is limited to two main clinical datasets, and most controlled missingness experiments rely on random modality masking. This may not sufficiently reflect realistic clinical missingness patterns.
W3 - The semantic validation is mainly performed on ADNI using post-hoc disease-axis annotations. More direct evidence is needed to show that the learned partition consistently captures meaningful latent structure across datasets.
Questions for authors
See Weakness
5 comments
Response to Reviewer YnfC (2/3)
W3. Meaningful latent structure across datasets
More direct evidence is needed to show that the learned partition consistently captures meaningful latent structure across datasets.
We appreciate the reviewer’s request for evidence beyond the ADNI disease-axis analysis. ADNI’s ATN(V) annotations provide a feature-level reference for interpreting individual views. On both ADNI and MIMIC-IV, we can also ask whether the learned grouping captures relationships that persist on held-out patients and whether those relationships are useful for prediction.
- Held-out co-variation. In submitted Figure 3, ΔC measures how much more strongly cross-modal signals co-vary within views than across views. Across the partitions evaluated in that analysis, ΔC on held-out patients was positively associated with test macro-F1 (Spearman 0.81 on ADNI and 0.64 on MIMIC-IV).
- Patient-correspondence control. As discussed in W1, shuffling patient correspondence before computing CKA reduced macro-F1 at ρ = 0.5 by 4.1 pp on ADNI and 5.1 pp on MIMIC-IV. On both datasets, the reduction occurred in all five paired seeds and the paired 95% confidence interval excluded zero (revised §3.3 and Appendix D.3). Table R3 shows that the shuffled target also yielded macro-AUC close to balanced random.
- Cross-patient evidence. At ρ = 0.5, blocking the flow of other patients’ evidence on modalities missing for the target lowered macro-F1 among affected patients by 0.14 pp on ADNI and 1.12 pp on MIMIC-IV. On both datasets, the reduction occurred in all five paired seeds and the 95% confidence interval excluded zero (revised Appendix C.6, “At inference”). The same intervention left predictions unchanged under modality-based grouping.
- Cross-seed stability. Fixing the tokenizer and retraining the routing components raised cross-seed partition ARI from 0.72 to 0.83 on ADNI and from 0.55 to 0.61 on MIMIC-IV. Held-out ΔC remained similar (0.183 to 0.181 and 0.149 to 0.141), and mean macro-F1 changed by 0.1 pp on each dataset (revised Appendix D.5, Table 15).
Table R3. Patient-correspondence shuffle at ρ = 0.5. Five-seed mean macro-F1 and macro-AUC (%). Column bests are in bold.
| Partition | ADNI Macro-F1 | ADNI Macro-AUC | MIMIC-IV Macro-F1 | MIMIC-IV Macro-AUC |
|---|---|---|---|---|
| Learned, fixed (ours) | 57.9 | 77.8 | 57.4 | 71.1 |
| Shuffled target | 53.8 | 73.9 | 52.3 | 67.4 |
| Balanced random | 53.7 | 73.8 | 52.1 | 67.3 |
Together, the held-out analysis and the two interventions support the same conclusion: on both clinical benchmarks, learned grouping captures cross-modal relationships that persist beyond the training patients and are useful for prediction. The stability analysis further suggests that prediction can remain similar when the exact assignments vary across runs. We use the ADNI annotations to interpret individual views and clarify the scope of that analysis in revised Appendix F.1.
We have incorporated these responses and additional results into the revised paper. Please refer to the uploaded PDF, where the changes are highlighted in blue.
We hope that our clarifications and additional experiments have addressed your concerns, and we would be grateful if you could consider reflecting this in your evaluation.
Response to Reviewer YnfC (3/3)
Concern about Figure 4: Description embedded in the figure file
Figure 4 is captioned as a five-seed mean ± SD, but its embedded file description reads “Deterministic synthetic ARI trajectories; not measured training results”.
Thank you for bringing this discrepancy to our attention. We sincerely apologize for the incorrect description embedded in the Figure 4 PDF and understand why it raised concerns about the provenance of the plotted results.
Origin of the incorrect description. During early manuscript preparation, we used synthetic trajectories in a plotting template to verify figure dimensions, styling, and PDF export formatting before the multi-seed experiments were finalized. When we subsequently generated Figure 4 from our measured training results, we inadvertently retained the template’s outdated PDF metadata instead of overwriting those fields. The embedded description therefore referred to the earlier layout template rather than the data plotted in the submitted figure.
Verification against original logs and cross-referenced paper results. To directly verify the provenance of the plotted values, we audited the original five-seed ARI logs and the corresponding reference-label metrics (metrics.json) used to generate Figure 4. At epoch 100 ( on ADNI), the five-seed mean ± SD ARIs computed from the original logs are:
| Training strategy | Disease-axis ARI (, Mean ± SD) | Source-modality ARI (, Mean ± SD) |
|---|---|---|
| DECENT | 0.617 ± 0.101 | 0.206 ± 0.075 |
| Task-only training | 0.121 ± 0.071 | 0.549 ± 0.123 |
Both the logged ARI trajectories and the underlying reference-label metrics match multiple cross-referenced entries reported across the paper:
- Figure 4 & Section 3.4: The per-epoch trajectories reproduce the plotted curves, the reported epoch-100 endpoints ( and for DECENT; and for task-only training), and the annotated final gap ().
- Table 13 (Appendix D.3): The epoch-100 DECENT alignment values (
0.62and0.21) match the corresponding ADNI label-alignment entries in Table 13. - Table 18 & Appendix E.3: The seed-specific reference label file (
metrics.json), which records the post hoc semantic annotations and retained token subsets used to compute ARI (excluding groups dominated byOtherornullfeatures), contains 13–14 retained groups out of 16 per seed (mean 13.4 ± 0.55; Table 18a), as well as the exact per-modality retained counts across the five seeds (18/20 Biospecimen, 19/20 Genetic, 20/20 Image, and 10/20 Clinical; Table 18b).
Verification materials. To make the provenance of Figure 4 fully transparent and verifiable, we have uploaded the original per-epoch ARI logs, the corresponding reference-label metrics used to compute them, the plotting script, and the training code to our anonymous repository (https://anonymous.4open.science/r/DECENT-7202/figure4/README.md):
- Original logs and ARI computation metrics (
figure4/figure4_logs.zip): Contains the seed-specific reference labels and retained group subsets (metrics.json) used for ARI evaluation, along with the original per-epoch ARI trajectories across epochs 0–100 and seeds 0–4 underseed_logs/{decent,task_only}/from which Figure 4 was generated. These files allow reviewers and readers to inspect the raw per-seed trajectories, verify consistency with the tables above, and recompute the five-seed mean curves and standard-deviation bands without requiring credentialed ADNI access. - Figure verification script (
figure4/generate_partition_alignment.py): Runningpython -m figure4.generate_partition_alignmentfrom the repository root generates the ARI trajectory plot and numerical summaries directly from the uploaded original logs and metrics, confirming an exact match with Figure 4. The resulting PDF is also provided in the same directory. - Training code (
figure4/train.py): Provides the training pipeline used to generate the original runs (executing two-stage DECENT or joint task-only training). When run with--alignment_reference, it records the adjusted Rand index (ARI) of the learned partition against both the disease-axis labels (Appendix A.2) and the source-modality labels at initialization (epoch 0) and after each Phase-1 epoch for DECENT or each task-training epoch for the task-only baseline. Both ARIs are computed over the exact same retained token subset per seed, excluding groups labeledOtherornull.
We have corrected the outdated embedded description in the revised Figure 4 PDF. We sincerely apologize for this oversight and appreciate the opportunity to clarify the metadata discrepancy and provide verifiable records of the original experiments.
Response to Reviewer YnfC (1/3)
We sincerely thank the reviewer for the thoughtful comments, and for finding the motivation intuitive and the partition ablation helpful. We address each point below.
First of all, the incorrect description embedded in Figure 4 was entirely due to a mistake in updating the template used to generate the figure PDF. It does not reflect any issue with research ethics or the integrity of the reported results. Please see our detailed responses below.
W1. Cross-modal co-variation and a shared latent process
We agree that disease severity, site, and care intensity can induce cross-modal co-variation. DECENT uses co-variation to decide which signals share a graph operator. The partition controls in submitted Table 2(b) support that design choice at fixed model capacity. To test the role of patient correspondence directly, we shuffled one side of each modality pair before computing CKA, keeping the co-observed patients, modality-wise token distributions, and model configuration fixed.
Matching modalities by patient improved macro-F1 by 3.4–5.1 pp across the six induced-missingness settings. At ρ = 0.5, the paired 95% confidence intervals exclude zero on both datasets, with higher macro-F1 in all five paired seeds. The shuffled target stayed within 0.9 pp of balanced random. This control establishes the predictive value of patient-level correspondence within the co-observed set. Shared clinical and acquisition factors remain possible sources of that correspondence. We added the control to revised Table 2(b) and Appendix D.3, and discuss the interpretation in Appendix F.1.
Table R1. Patient-correspondence shuffle control. Five-seed mean macro-F1 (%) at the indicated masking rates ρ. Column bests are in bold.
W2. Evaluation breadth and realistic clinical missingness patterns
We agree that random masking captures only one form of missingness. At ρ = 0, ADNI preserves natural missingness and DECENT is within 0.7 pp of MUSE on both metrics. The evaluated MIMIC-IV cohort is complete at ρ = 0. As masking increases to ρ = 0.7, macro-AUC declines by 6.3 and 6.7 pp for DECENT on ADNI and MIMIC-IV, compared with 8.9 and 10.1 pp for MUSE (Table 1).
In the clinical cohorts, we observe which modalities are missing, while the mechanisms behind those patterns remain unknown. To test what happens when missingness depends on observed characteristics or latent severity, we use a synthetic benchmark with controlled missingness mechanisms and known ground-truth views (revised Appendix E.5, Table 19). DECENT recovered planted structure under MCAR, MAR, and MNAR, with the weakest recovery under MNAR. False grouping and false separation were below both reference partitions in all three settings. Under MNAR, macro-F1 was highest despite weaker recovery. The higher mask-only F1 under MNAR shows that availability itself carried predictive information in this setting. This distinction matters when using prediction to interpret a learned partition.
Table R2. Synthetic view recovery at ρ = 0.5 and λ = 1.0. Five-seed means × 100. Soft AUC measures recovery. Bold marks the best displayed recovery and prediction scores.
Beyond ENRICO in the submission, we added 30-day readmission on the same MIMIC-IV cohort and audio–video–text sentiment classification on CMU-MOSI. These extend the outcome and modality coverage. See revised Appendices C.2–C.3 for the new tasks and Appendix F.1 for the scope of the clinical evaluation.