acceptodds
Under review as a conference paper at ICLR 2027

Decoupled Graph Experts for Robust Multimodal Learning with Missing Modalities

Abstract

Multimodal clinical data routinely arrive with *missing* modalities. Bipartite graph formulations address this missingness by encoding patients and modality-derived units as a bipartite graph, but modality-level graph construction forces heterogeneous sub-signals within one modality to share an operator (*false grouping*) and separates related sub-signals across modalities (*false separation*). Both restrictions are costly for patients with missing modalities, who rely on evidence propagated from other patients. False separation places related evidence from a missing modality in a view the patient cannot reach, and false grouping mixes the evidence the patient does receive with unrelated sub-signals. Which signals share an operator should therefore be learned from data: signals that co-vary across modalities through a shared process should share a view, while co-variation within one modality, which acquisition effects confound, should not decide the grouping. We propose DECENT, which encodes each observed modality into index-aligned proto-tokens and learns the signal-node partition by self-supervised sparse routing, trained to route signals that co-vary across modalities to the same view. With this partition fixed, DECENT runs message passing on the resulting view-specific sample–signal graphs and fuses the views observed for each patient. On ADNI and MIMIC-IV, DECENT achieves the highest average macro-F1 and macro-AUC among the evaluated methods across four missingness settings. Partition controls with the tokenizer and graph encoder fixed attribute these gains to the learned partition. Post hoc analyses on ADNI further show that the learned views align more closely with disease-informed axes than with source modalities. Code is available at [https://anonymous.4open.science/r/DECENT-7202/](https://anonymous.4open.science/r/DECENT-7202/README.md).

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal learning, Missing modalities, Sparse routing, Learned graph partitioning, Graph-based fusion, Clinical prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YnfCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DECENT for multimodal learning with missing modalities. The key motivation is that modality-level graph construction may cause false grouping of heterogeneous signals within the same modality and false separation of related signals across different modalities. DECENT decomposes each modality into index-aligned proto-tokens, learns a signal-level partition using cross-modal co-variation, and then performs prediction with view-specific graph experts and availability-aware fusion.

Critical strengths
  • The motivation of false grouping and false separation is intuitive and clearly explains why modality-level graph construction can be suboptimal under missing modalities.

  • The partition ablation is relatively convincing. With the tokenizer and graph encoder kept fixed, the learned partition consistently outperforms modality-based, balanced-random, and single-view partitions in all six missing-modality settings. This helps isolate the benefit of the learned signal grouping rather than simply adding more graph experts. The two-stage training design is supported by a useful ablation. Freezing the learned partition after Phase 1 performs better than task-only training in all six settings and better than Phase-2 fine-tuning in five of six settings, showing that directly optimizing the partition with task labels is not necessarily beneficial.

  • The paper identifies a concrete limitation of modality-level graph construction, namely false grouping within a modality and false separation across modalities. More importantly, the experiments are consistent with this motivation: compared with MUSE, the advantage of DECENT becomes clearer as the missing rate increases, and its AUC drops less from ρ=0\rho=0 to ρ=0.7\rho=0.7 on both ADNI and MIMIC-IV.

Critical weaknesses

W1 - The core assumption that cross-modal co-variation corresponds to a meaningful shared latent process is relatively strong. Cross-modal correlation may also arise from disease severity or other shared confounding factors, so the semantic interpretation of the learned views is not fully justified.

W2 - The evaluation is limited to two main clinical datasets, and most controlled missingness experiments rely on random modality masking. This may not sufficiently reflect realistic clinical missingness patterns.

W3 - The semantic validation is mainly performed on ADNI using post-hoc disease-axis annotations. More direct evidence is needed to show that the learned partition consistently captures meaningful latent structure across datasets.

Questions for authors

See Weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer YnfC (1/3)

We sincerely thank the reviewer for the thoughtful comments, and for finding the motivation intuitive and the partition ablation helpful. We address each point below.

First of all, the incorrect description embedded in Figure 4 was entirely due to a mistake in updating the template used to generate the figure PDF. It does not reflect any issue with research ethics or the integrity of the reported results. Please see our detailed responses below.

W1. Cross-modal co-variation and a shared latent process

Cross-modal correlation may also arise from disease severity or other shared confounding factors, so the semantic interpretation of the learned views is not fully justified.

We agree that disease severity, site, and care intensity can induce cross-modal co-variation. DECENT uses co-variation to decide which signals share a graph operator. The partition controls in submitted Table 2(b) support that design choice at fixed model capacity. To test the role of patient correspondence directly, we shuffled one side of each modality pair before computing CKA, keeping the co-observed patients, modality-wise token distributions, and model configuration fixed.

Matching modalities by patient improved macro-F1 by 3.4–5.1 pp across the six induced-missingness settings. At ρ = 0.5, the paired 95% confidence intervals exclude zero on both datasets, with higher macro-F1 in all five paired seeds. The shuffled target stayed within 0.9 pp of balanced random. This control establishes the predictive value of patient-level correspondence within the co-observed set. Shared clinical and acquisition factors remain possible sources of that correspondence. We added the control to revised Table 2(b) and Appendix D.3, and discuss the interpretation in Appendix F.1.

Table R1. Patient-correspondence shuffle control. Five-seed mean macro-F1 (%) at the indicated masking rates ρ. Column bests are in bold.

PartitionADNI 0.2ADNI 0.5ADNI 0.7MIMIC-IV 0.2MIMIC-IV 0.5MIMIC-IV 0.7
Learned, fixed (ours)61.957.955.858.757.456.5
Shuffled target57.353.851.255.352.351.4
Balanced random57.053.750.354.752.151.4

W2. Evaluation breadth and realistic clinical missingness patterns

The evaluation is limited to two main clinical datasets, and random modality masking may not sufficiently reflect realistic clinical missingness patterns.

We agree that random masking captures only one form of missingness. At ρ = 0, ADNI preserves natural missingness and DECENT is within 0.7 pp of MUSE on both metrics. The evaluated MIMIC-IV cohort is complete at ρ = 0. As masking increases to ρ = 0.7, macro-AUC declines by 6.3 and 6.7 pp for DECENT on ADNI and MIMIC-IV, compared with 8.9 and 10.1 pp for MUSE (Table 1).

In the clinical cohorts, we observe which modalities are missing, while the mechanisms behind those patterns remain unknown. To test what happens when missingness depends on observed characteristics or latent severity, we use a synthetic benchmark with controlled missingness mechanisms and known ground-truth views (revised Appendix E.5, Table 19). DECENT recovered planted structure under MCAR, MAR, and MNAR, with the weakest recovery under MNAR. False grouping and false separation were below both reference partitions in all three settings. Under MNAR, macro-F1 was highest despite weaker recovery. The higher mask-only F1 under MNAR shows that availability itself carried predictive information in this setting. This distinction matters when using prediction to interpret a learned partition.

Table R2. Synthetic view recovery at ρ = 0.5 and λ = 1.0. Five-seed means × 100. Soft AUC measures recovery. Bold marks the best displayed recovery and prediction scores.

Mechanism or referenceSoft AUC ↑Hard ARI ↑False grouping ↓False separation ↓Macro-F1 ↑Mask-only F1
MCAR (ours)86.766.59.428.963.034.4
MAR (ours)87.867.18.026.862.333.8
MNAR (ours)83.862.712.631.166.542.4
Balanced random (reference)——20.080.0——
Modality-based (reference)——100.0100.0——

Beyond ENRICO in the submission, we added 30-day readmission on the same MIMIC-IV cohort and audio–video–text sentiment classification on CMU-MOSI. These extend the outcome and modality coverage. See revised Appendices C.2–C.3 for the new tasks and Appendix F.1 for the scope of the clinical evaluation.

AuthorsOpenReview ↗

Response to Reviewer YnfC (2/3)

W3. Meaningful latent structure across datasets

More direct evidence is needed to show that the learned partition consistently captures meaningful latent structure across datasets.

We appreciate the reviewer’s request for evidence beyond the ADNI disease-axis analysis. ADNI’s ATN(V) annotations provide a feature-level reference for interpreting individual views. On both ADNI and MIMIC-IV, we can also ask whether the learned grouping captures relationships that persist on held-out patients and whether those relationships are useful for prediction.

  • Held-out co-variation. In submitted Figure 3, ΔC measures how much more strongly cross-modal signals co-vary within views than across views. Across the partitions evaluated in that analysis, ΔC on held-out patients was positively associated with test macro-F1 (Spearman 0.81 on ADNI and 0.64 on MIMIC-IV).
  • Patient-correspondence control. As discussed in W1, shuffling patient correspondence before computing CKA reduced macro-F1 at ρ = 0.5 by 4.1 pp on ADNI and 5.1 pp on MIMIC-IV. On both datasets, the reduction occurred in all five paired seeds and the paired 95% confidence interval excluded zero (revised §3.3 and Appendix D.3). Table R3 shows that the shuffled target also yielded macro-AUC close to balanced random.
  • Cross-patient evidence. At ρ = 0.5, blocking the flow of other patients’ evidence on modalities missing for the target lowered macro-F1 among affected patients by 0.14 pp on ADNI and 1.12 pp on MIMIC-IV. On both datasets, the reduction occurred in all five paired seeds and the 95% confidence interval excluded zero (revised Appendix C.6, “At inference”). The same intervention left predictions unchanged under modality-based grouping.
  • Cross-seed stability. Fixing the tokenizer and retraining the routing components raised cross-seed partition ARI from 0.72 to 0.83 on ADNI and from 0.55 to 0.61 on MIMIC-IV. Held-out ΔC remained similar (0.183 to 0.181 and 0.149 to 0.141), and mean macro-F1 changed by 0.1 pp on each dataset (revised Appendix D.5, Table 15).

Table R3. Patient-correspondence shuffle at ρ = 0.5. Five-seed mean macro-F1 and macro-AUC (%). Column bests are in bold.

PartitionADNI Macro-F1ADNI Macro-AUCMIMIC-IV Macro-F1MIMIC-IV Macro-AUC
Learned, fixed (ours)57.977.857.471.1
Shuffled target53.873.952.367.4
Balanced random53.773.852.167.3

Together, the held-out analysis and the two interventions support the same conclusion: on both clinical benchmarks, learned grouping captures cross-modal relationships that persist beyond the training patients and are useful for prediction. The stability analysis further suggests that prediction can remain similar when the exact assignments vary across runs. We use the ADNI annotations to interpret individual views and clarify the scope of that analysis in revised Appendix F.1.


We have incorporated these responses and additional results into the revised paper. Please refer to the uploaded PDF, where the changes are highlighted in blue.

We hope that our clarifications and additional experiments have addressed your concerns, and we would be grateful if you could consider reflecting this in your evaluation.

AuthorsOpenReview ↗

Response to Reviewer YnfC (3/3)

Concern about Figure 4: Description embedded in the figure file

Figure 4 is captioned as a five-seed mean ± SD, but its embedded file description reads “Deterministic synthetic ARI trajectories; not measured training results”.

Thank you for bringing this discrepancy to our attention. We sincerely apologize for the incorrect description embedded in the Figure 4 PDF and understand why it raised concerns about the provenance of the plotted results.

Origin of the incorrect description. During early manuscript preparation, we used synthetic trajectories in a plotting template to verify figure dimensions, styling, and PDF export formatting before the multi-seed experiments were finalized. When we subsequently generated Figure 4 from our measured training results, we inadvertently retained the template’s outdated PDF metadata instead of overwriting those fields. The embedded description therefore referred to the earlier layout template rather than the data plotted in the submitted figure.

Verification against original logs and cross-referenced paper results. To directly verify the provenance of the plotted values, we audited the original five-seed ARI logs and the corresponding reference-label metrics (metrics.json) used to generate Figure 4. At epoch 100 (ρ=0.5\rho = 0.5 on ADNI), the five-seed mean ± SD ARIs computed from the original logs are:

Training strategyDisease-axis ARI (AaxisA_{\text{axis}}, Mean ± SD)Source-modality ARI (AmodA_{\text{mod}}, Mean ± SD)
DECENT0.617 ± 0.1010.206 ± 0.075
Task-only training0.121 ± 0.0710.549 ± 0.123

Both the logged ARI trajectories and the underlying reference-label metrics match multiple cross-referenced entries reported across the paper:

  • Figure 4 & Section 3.4: The per-epoch trajectories reproduce the plotted curves, the reported epoch-100 endpoints (Aaxis=0.62A_{\text{axis}} = 0.62 and Amod=0.21A_{\text{mod}} = 0.21 for DECENT; Amod=0.55A_{\text{mod}} = 0.55 and Aaxis=0.12A_{\text{axis}} = 0.12 for task-only training), and the annotated final gap ΔA=Aaxis−Amod=0.41\Delta A = A_{\text{axis}} - A_{\text{mod}} = 0.41 (0.617−0.206=0.4110.617 - 0.206 = 0.411).
  • Table 13 (Appendix D.3): The epoch-100 DECENT alignment values (0.62 and 0.21) match the corresponding ADNI label-alignment entries in Table 13.
  • Table 18 & Appendix E.3: The seed-specific reference label file (metrics.json), which records the post hoc semantic annotations and retained token subsets used to compute ARI (excluding groups dominated by Other or null features), contains 13–14 retained groups out of 16 per seed (mean 13.4 ± 0.55; Table 18a), as well as the exact per-modality retained counts across the five seeds (18/20 Biospecimen, 19/20 Genetic, 20/20 Image, and 10/20 Clinical; Table 18b).

Verification materials. To make the provenance of Figure 4 fully transparent and verifiable, we have uploaded the original per-epoch ARI logs, the corresponding reference-label metrics used to compute them, the plotting script, and the training code to our anonymous repository (https://anonymous.4open.science/r/DECENT-7202/figure4/README.md):

  • Original logs and ARI computation metrics (figure4/figure4_logs.zip): Contains the seed-specific reference labels and retained group subsets (metrics.json) used for ARI evaluation, along with the original per-epoch ARI trajectories across epochs 0–100 and seeds 0–4 under seed_logs/{decent,task_only}/ from which Figure 4 was generated. These files allow reviewers and readers to inspect the raw per-seed trajectories, verify consistency with the tables above, and recompute the five-seed mean curves and standard-deviation bands without requiring credentialed ADNI access.
  • Figure verification script (figure4/generate_partition_alignment.py): Running python -m figure4.generate_partition_alignment from the repository root generates the ARI trajectory plot and numerical summaries directly from the uploaded original logs and metrics, confirming an exact match with Figure 4. The resulting PDF is also provided in the same directory.
  • Training code (figure4/train.py): Provides the training pipeline used to generate the original runs (executing two-stage DECENT or joint task-only training). When run with --alignment_reference, it records the adjusted Rand index (ARI) of the learned partition against both the disease-axis labels (Appendix A.2) and the source-modality labels at initialization (epoch 0) and after each Phase-1 epoch for DECENT or each task-training epoch for the task-only baseline. Both ARIs are computed over the exact same retained token subset per seed, excluding groups labeled Other or null.

We have corrected the outdated embedded description in the revised Figure 4 PDF. We sincerely apologize for this oversight and appreciate the opportunity to clarify the metadata discrepancy and provide verifiable records of the original experiments.

Reviewer_5bdqOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
2(−1)
Soundness
3(+1)
Read review
Paper summary

This paper proposes DECENT, a graph-based method for multimodal prediction when some modalities are missing. The authors argue that grouping information by its acquisition modality can put unrelated signals together and separate related signals that come from different modalities. To address this, they learn how to group signals according to their co-variation across patients, then keep these groups fixed while training graph experts for prediction. Their main contribution is to allow signals within a modality to be processed separately and related signals across modalities to be processed together, using a grouping learned without task labels. They evaluate the method on ADNI and MIMIC-IV under several missingness settings and report improvements over modality-based grouping, supported by controlled comparisons of the grouping rule, training strategy, and graph operators. They also provide a non-medical evaluation and synthetic experiments to examine the scope of the approach.

Critical strengths
  1. I find the proposed way of grouping signals well motivated and useful. The authors allow a modality to contribute to several graph views and allow related signals from different modalities to share an operator. Although learned graph decomposition has precedents in DisenGCN and FactorGCN, learning a fixed signal partition from cross-modal co-variation makes a meaningful architectural contribution to the missing-modality problem. This is the main idea I would take from the paper, and I think its value is supported by the experiments.

  2. The ablation study does a good job of testing the design choices. In particular, Table 2(b) keeps the tokenizer and downstream architecture fixed while changing the grouping rule, and the learned partition improves over modality-based and random assignments across the induced-missingness settings. The comparisons of tokenization, training strategy, and operator sharing, together with the routing-target and spectral-clustering comparisons, make it easier to understand the source of the gains. I also appreciate the reporting of cases where PCA/FA tokenizers or independent GNNs perform better. Overall, this is a sufficiently complete experimental study to support acceptance.

Critical weaknesses
  1. The explanation of missing-modality evidence propagation needs some clarification. With constant patient-node initialization and the synchronous updates in Eq. (15), I am not fully clear how another patient’s evidence from a modality absent in the focal patient reaches that patient in the two-layer MIMIC-IV configuration. A clearer account of the implemented update schedule would help me understand this argument. This is a concern about the explanation; the controlled results still support the value of the learned grouping.

  2. The training-strategy comparison supports the choice to freeze the learned structure, but the reason for the improvement is less certain. The tokenizer, projections, and router can change together in the alternatives, so the degradation cannot be attributed solely to changes in the partition. I would keep the conclusion about task supervision pushing groups toward modality boundaries specific to the tested setup. The disease-axis analysis is also best read as an informative diagnostic, consistent with the limitations the authors already acknowledge.

Questions for authors
  1. Could you clarify whether one implemented GNN layer corresponds to one synchronous update in Eq. (15), or to sequential patient and signal updates? A short explanation of the information path in the two-layer MIMIC-IV model would resolve my main uncertainty about the missing-modality propagation argument.

  2. How should the task-only and fine-tuning comparisons be interpreted given that token representations and routing can change together? In particular, please clarify which parts of the routing decision receive task gradients and whether the observed modality-alignment trend can be attributed specifically to partition updates. I would be satisfied with a clearer account of what the current comparisons establish.

  3. Suggestion for additional experiments: the clinical evidence currently rests on two cohorts, ADNI (2,380 subjects, three-class CN/MCI/AD diagnosis) and MIMIC-IV (9,003 subjects, binary one-year mortality), each evaluated on a single fixed patient split. The ADNI test split contains 361 subjects across three classes, i.e., about 120 per class on average, and the class distributions of the splits are not reported. Since the central claim is that the learned partition benefits patients with missing modalities in clinical data, it would strengthen the paper to add one or two public multimodal clinical datasets with binary prediction tasks and at least roughly 200 subjects per class, and to report the class distribution of each split. Such results would make it clearer whether the gains over modality-level partitioning generalize beyond these two cohorts.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5bdq (1/2)

We thank the reviewer for recognizing the motivation for grouping signals across modalities and the value of the controlled ablations. We address the update schedule and training-strategy comparison below.

W1 & Q1. Implemented update schedule and the information path

Does one implemented GNN layer correspond to one synchronous update in Eq. (15), or to sequential patient and signal updates?

Thank you for examining the update schedule. One GNN layer implements one synchronous update of Eq. (15). Two layers can convey evidence on a shared observed signal, and evidence from a modality missing in the focal patient requires at least three. If the focal patient observes A and another patient observes A and B, that patient’s B evidence enters their own state in layer 1, reaches the shared A signal in layer 2, and reaches the focal patient in layer 3 when A and B share a view. The reviewer is correct that this path requires three layers. The reported MIMIC-IV experiments used three GNN layers (L = 3). The L = 2 entries in the original §3.1 and Table 4 were typographical errors. We have corrected both in the revision.

At ρ = 0.5, blocking this path at inference for test patients with a missing modality reduced their macro-F1 by 0.14 pp on ADNI and 1.12 pp on MIMIC-IV. The reduction occurred in all five paired seeds on both datasets, and both paired 95% confidence intervals exclude zero. Modality-based views have no such path because each view contains signals from one modality. We have clarified the update schedule in Appendix A.1 and the information path in Appendix B.2 (Proposition 2), and added the path-removal analysis to Appendix C.6 of the revised paper.

W2 & Q2. What the training-strategy comparison establishes

How should the task-only and fine-tuning comparisons be interpreted given that token representations and routing can change together?

We agree that token representations and routing can change together. The comparison supports freezing the learned structure as part of the two-phase training strategy. It measures the joint effect of allowing the tokenizer, projections, and router to adapt. Both alternatives use hard graph membership and one-hot message-basis selection in the forward pass. In the backward pass, a straight-through estimator passes task gradients to the router through the soft assignment weights in the message-basis combination. The hard graph-membership decision is non-differentiable.

Table R1. Training-strategy comparison from submitted Table 2(c). Values are five-seed mean macro-F1 (%) at each induced masking rate ρ. The best value in each column is in bold.

Training strategyADNI 0.2ADNI 0.5ADNI 0.7MIMIC-IV 0.2MIMIC-IV 0.5MIMIC-IV 0.7
Frozen two-phase (ours)61.957.955.858.757.456.5
Phase-2 fine-tuning59.855.556.357.156.255.6
Task-only training59.255.053.156.856.255.7

The frozen strategy outperformed task-only training in all six settings and Phase-2 fine-tuning in five of six. At ρ = 0.5, its gains over task-only training were 2.9 pp on ADNI and 1.2 pp on MIMIC-IV. The separate partition control in submitted Table 2(b) holds the tokenizer and downstream architecture fixed and changes the grouping rule, providing direct evidence for the predictive value of learned grouping.

We agree that the modality-alignment trend is specific to the tested training setup, where the tokenizer and router can adapt jointly. We interpret the disease-axis comparison as a post hoc diagnostic based on the feature annotations. The revised Appendix A.2 describes the task-gradient path, and §3.3–3.4 and Appendix F.1 discuss the scope of these comparisons.

AuthorsOpenReview ↗

Response to Reviewer 5bdq (2/2)

Q3. Class distributions and additional clinical cohorts

Please add public multimodal clinical datasets with binary prediction tasks and at least roughly 200 subjects per class, and report the class distribution of each split.

Thank you for this constructive suggestion. We agree that class counts by split and evaluation on additional clinical cohorts matter. The submission included ENRICO (Appendix C.2). The revised evaluation covers five tasks across four datasets, including a second binary clinical outcome with at least 376 patients per class in each split and an audio–video–text benchmark. All methods share fixed splits, and five seeds capture training variability.

Table R2. Dataset and split summary. Counts follow class order. ENRICO shows class-stratified split totals.

Dataset (task)ModalitiesClassesTrainValTest
ADNI (diagnosis)MRI, Gen, Cl, BioCN / MCI / AD730 / 595 / 342162 / 122 / 68164 / 127 / 70
MIMIC-IV (1-yr mortality)Lab, Note, CodeSurvived / died5,757 / 5461,243 / 1081,243 / 106
MIMIC-IV (30-d readmission)Lab, Note, CodeNo / yes4,351 / 1,952975 / 376945 / 404
ENRICO (UI design)Screen, Wireframe, UI hierarchy, Metadata20 classes1,020219219
CMU-MOSI (sentiment)Audio, Video, TextNeg. / pos.606 / 67791 / 123409 / 277

MIMIC-IV readmission. We kept the mortality-trained Phase-1 tokenizer and partition fixed and trained Phase 2 on 30-day readmission. We retrained all baselines on the same splits and masks without retuning. DECENT had the highest average macro-F1, and SimMLM had the highest average macro-AUC. As ρ increased from 0 to 0.7, DECENT’s macro-AUC dropped by 4.0 pp, compared with 5.3 pp for SimMLM (Table R3). This shows that the mortality partition can be reused for readmission in the same cohort while retaining competitive performance as more modalities are masked.

Table R3. MIMIC-IV readmission. Five-seed mean macro-F1 and macro-AUC (%) by masking rate ρ. Column bests are in bold.

MethodF1 (0)F1 (0.2)F1 (0.5)F1 (0.7)AUC (0)AUC (0.2)AUC (0.5)AUC (0.7)
GRAPE54.753.653.952.557.456.855.953.8
GTP-4o53.852.051.650.554.653.153.852.3
HGMF53.051.049.549.554.152.650.249.4
M3Care54.050.749.950.654.852.051.351.3
MUSE53.352.852.851.455.854.654.653.4
AMC50.647.147.845.854.051.852.651.1
Flex-MoE55.253.352.251.257.656.153.852.5
MoE-Retriever53.852.552.050.755.753.152.250.8
SimMLM55.553.954.253.059.457.655.954.1
DECENT (ours)56.255.054.053.058.656.856.354.6

CMU-MOSI. All methods used the same frozen audio, video, and text embeddings and MultiBench splits. DECENT and GRAPE started within 0.1 pp in macro-F1 at ρ = 0. At ρ = 0.7, DECENT scored 55.0% and GRAPE scored 51.8%. DECENT led macro-F1 at ρ = 0.5 and 0.7 and macro-AUC at every rate (Table R4). The advantage under heavier masking thus extends to audio–video–text inputs.

Table R4. CMU-MOSI. Five-seed mean macro-F1 and macro-AUC (%) by masking rate ρ. Column bests are in bold.

MethodF1 (0)F1 (0.2)F1 (0.5)F1 (0.7)AUC (0)AUC (0.2)AUC (0.5)AUC (0.7)
GRAPE69.066.858.151.875.674.262.955.3
GTP-4o55.049.650.646.658.053.252.751.6
HGMF51.150.833.145.553.654.055.949.3
M3Care66.660.756.652.872.466.559.757.6
MUSE55.253.250.346.659.457.852.351.4
AMC50.049.948.345.452.151.950.650.2
Flex-MoE55.655.752.250.462.259.354.355.2
MoE-Retriever61.558.156.048.666.662.659.951.3
SimMLM59.557.555.054.763.864.159.758.1
DECENT (ours)68.965.660.055.077.874.964.460.4

We report clinical class counts in revised Table 3 and the additional evaluations in Appendices C.2–C.3 (Tables 8–9).


We have incorporated these responses and additional results into the revised paper. Please refer to the uploaded PDF, where the changes are highlighted in blue.

We hope that our clarifications and additional experiments address your concerns. If so, we would be grateful if you could take them into account in your evaluation. Please let us know if any questions remain.

Thank you for the detailed rebuttal and additional experiments. The clarification of the three-layer message-passing mechanism and the path-removal experiment provide more direct evidence for the proposed cross-modal information propagation. The additional datasets and task-transfer results also strengthen the empirical support for the method. I appreciate the clarification regarding the scope of the training-strategy comparison. These responses have addressed several of my concerns, and I have raised my Soundness score from 2 to 3.

AuthorsOpenReview ↗

Response to the Follow-Up

Thank you for taking the time to review our rebuttal and for updating your Soundness score. Your questions helped us clarify how cross-modal evidence reaches a patient in the three-layer model and test that pathway directly. We also appreciate your feedback on the additional evaluations and our clarification of the training-strategy comparison.

If you feel that any concerns remain insufficiently addressed to warrant a change in your overall rating, we would greatly appreciate your feedback. We would be happy to provide more detailed explanations or clarifications as needed. Thank you again for your thoughtful and constructive review.

Reviewer_LLYxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DECENT, a graph-based framework for prediction from multimodal data with missing modalities. The authors identify two limitations of modality-based graph partitioning: false grouping, where heterogeneous signals within one modality share an operator, and false separation, where related signals across modalities are assigned to different views. DECENT encodes each observed modality into indexed proto-tokens and trains a sparse router to group token types according to their cross-modal co-variation, measured using debiased CKA. The tokenizer and partition are then frozen while view-specific graph experts, fusion, and a classifier are trained using outcome labels. Experiments evaluate three-class diagnosis classification on ADNI and one-year mortality prediction on MIMIC-IV. DECENT achieves the highest average macro-F1 and macro-AUC across the evaluated missingness settings, with its clearest advantages under induced modality removal. Partition controls, component ablations, synthetic experiments, and post hoc analyses provide additional validation.

Critical strengths

The paper raises a worthwhile structural question: should signals share a graph operator because they originate from the same acquisition modality, or because their information is related? The proposed distinction between false grouping and false separation is intuitively meaningful. Learning partitions below the modality level provides a potentially useful way to organize heterogeneous evidence when inputs are incomplete. The specific contribution is the learned signal partition and its use in graph encoding, rather than graph-based multimodal learning with missing modalities in general.

The partition controls provide meaningful empirical support. With the tokenizer and downstream architecture retained, the learned partition outperforms modality-based, balanced random, and single-view alternatives across the induced-missingness settings. For example, at rho = 0.5, learned versus random grouping achieves macro-F1 of 57.9 versus 53.7 on ADNI and 57.4 versus 52.1 on MIMIC-IV. Comparisons against spectral clustering, alternative routing targets, and removal of the CKA-matching loss further investigate the proposed design. These controls make the predictive benefit more convincing than a comparison against external baselines alone.

Critical weaknesses
  1. The connection between the learned statistical grouping and the proposed mechanism remains insufficiently established. The VAE is optimized for reconstruction, while the routing objective cannot update the tokenizer. A fixed token index therefore establishes computational correspondence across patients, but does not guarantee that each token isolates a coherent underlying process. Cross-modal CKA may capture shared nuisance, overall severity, or selection into co-observation, and high association does not necessarily establish that two signals should share a graph operator. The reconstruction, non-collapse, semantic-alignment, and synthetic analyses are useful, but do not fully distinguish clinically meaningful grouping from a beneficial statistical partition or regularization effect. I recognize that the authors acknowledge the limits of process identification. My concern is whether the evidence adequately supports the explanation used to motivate the method.

  2. The comparative and practical value of the additional complexity remains unclear.

  • The strongest gains occur under induced masking, while DECENT does not consistently outperform the strongest baselines under natural missingness. This does not invalidate the method, but limits the evidence for its advantage under the original clinical observation patterns. In addition, baseline reproduction gaps require explanation, and several baselines need adaptations to different inputs and tasks. I hope the authors could specify how they are adopting each baseline, together with the hyperparameters they chose for the baseline and the optimizer updates, and the training epochs for the baselines.

  • The multistage pipeline combines VAEs, routing, multiple graph operators, and attention-based fusion, yet the parameter and computational costs are not sufficiently characterized. Given the 1,667 training subjects in ADNI, capacity and overfitting deserve closer examination. I hope the authors could provide a table showing the total number of parameters, FLOPS, and end-to-end training and inference time (in seconds) for DECENT compared with all baselines.

  1. Limited evaluation breadth. The clinical evaluation covers only ADNI diagnosis classification and MIMIC-IV one-year mortality prediction. Although the ENRICO experiment in Appendix C.2 provides useful evidence beyond healthcare, the overall task diversity remains limited for a general multimodal learning framework. Related work such as MoE-Retriever (https://openreview.net/forum?id=j9DbobO0mY) evaluates additional settings, including audiovisual-text sentiment analysis on CMU-MOSI. Evaluating DECENT on additional clinical outcomes, such as readmission or length-of-stay and more diverse modality combinations would help establish whether its learned partitioning principle generalizes beyond the selected benchmarks. This is particularly relevant because the proposed grouping is learned without outcome labels, yet its predictive benefits may still depend on the downstream task.
Questions for authors
  1. Why is the tokenizer trained only through reconstruction and VAE regularization, with routing gradients stopped? What evidence supports this separation? A Phase-1 comparison with and without this gradient boundary, retaining the same objectives and training budget, would directly address whether the separation is necessary. Task-only training and Phase-2 fine-tuning change additional aspects of the procedure.

  2. How can we distinguish grouping of clinically related signals from grouping driven by shared nuisance or observation selection? The existing semantic analyses are suggestive, but attribution-derived dominant labels can obscure mixed tokens. Please report the number of retained groups, their attribution distributions, and evidence about token purity. A control that shuffles patient correspondence across modalities before calculating CKA would also help test the specific relationship target.

  3. How sensitive are the relationship matrix and final partition to tokenizer versus router initialization? Table 9 reports cross-seed partition agreement after token matching, which is helpful. Can you separate these sources of variation by fixing one tokenizer and rerunning the router, then varying the tokenizer? This would clarify whether similar predictive performance depends on reproducible signal structure.

  4. Please clarify the information flow under Equation 15. With constant initial patient-node features and synchronous updates, another patient’s modality-B evidence appears to require three layers to reach a target patient who observes only modality A. MIMIC-IV uses two layers. Does the implementation use a different update order, or is its benefit primarily due to shared operators and propagation of observed-modality evidence?

  5. Please reconcile the baseline results with the corresponding original protocols. For example, AMC reports F1 of 54.82 on ADNI and 59.75 on MIMIC-IV, while the current paper reports macro-F1 of 49.5 and 55.2 at rho = 0. Please clarify metric averaging, input encoders, cohort construction, label prevalence, thresholds, and tuning budgets. Conversely, MUSE’s original 90-day post-discharge mortality results are not directly comparable to this paper’s one-year task. A baseline-specific adaptation table would help establish fair comparisons.

  6. Could you compare against retrieval-based missing-modality methods, including MoE-Retriever from the AAAI 2025 GenAI4Health Workshop (https://openreview.net/forum?id=j9DbobO0mY)? Its use of cross-patient supporting examples and observed modalities makes it relevant to DECENT’s motivation.

  7. What are the total and trainable parameter counts, training time for each phase, inference time, and peak memory, compared with the strongest baselines? Can a model trained with the same tokens and partition but without cross-patient message passing achieve comparable performance? This is particularly relevant because Table 7 shows only a 0.20-point ADNI macro-F1 difference between solo and operating-batch inference.

  8. Please specify the clinical label and observation protocol more fully: the origin of the one-year mortality window, feature-availability windows, cohort inclusion criteria, class distributions, and original modality-availability patterns. Please also report realized missingness after the retain-one rule. These details are important for interpreting both the absolute performance and the baseline reproduction differences.

8 comments
AuthorsOpenReview ↗

Response to Reviewer LLYx (1/6)

We thank the reviewer for highlighting the question of which signals should share a graph operator and for recognizing the value of the partition controls.

W1 (Q2). Connection between the learned statistical grouping and the proposed mechanism

How can we distinguish grouping of clinically related signals from grouping driven by shared nuisance or observation selection?

Preserving patient correspondence when computing cross-modal CKA improved the learned partition’s predictive performance on both datasets, with the co-observed patient set held fixed. In the submission, the learned partition outperformed modality-based and balanced random grouping using the same tokenizer, number of views, and graph encoder architecture (Table 2(b)). Across the tested partitions, held-out co-variation was associated with test macro-F1 (Spearman correlations of 0.81 on ADNI and 0.64 on MIMIC-IV, Fig. 3).

Following the reviewer’s suggestion, we recomputed debiased CKA after shuffling the tokens on one side of each cross-modal pair within the same co-observed training patients. This breaks patient correspondence while preserving the patients included in each pair and the token distribution on each side. We held the tokenizer, router training procedure, graph encoder architecture, and training budget fixed.

Table R1. Patient-correspondence control at ρ = 0.5. Values are five-seed mean macro-F1 and macro-AUC (%). The best value in each column is in bold.

PartitionADNI F1ADNI AUCMIMIC-IV F1MIMIC-IV AUC
Learned from cross-modal CKA (ours)57.977.857.471.1
Learned from shuffled-correspondence CKA53.873.952.367.4
Modality-based55.976.656.270.3
Balanced random53.773.852.167.3

At ρ = 0.5, shuffling lowered macro-F1 by 4.1 pp on ADNI and 5.1 pp on MIMIC-IV. The drop occurred in all five paired seeds on each dataset, and both paired 95% confidence intervals exclude zero. Across all six induced-missingness settings, the shuffled partition stayed within 0.9 pp of balanced random (revised Table 2(b) and Appendix D.3). Proposition 1 gives the rationale for using cross-modal co-variation under independent modality-specific acquisition effects. The attribution diagnostic below examines how the resulting proto-tokens relate to the clinical annotations.

Q2. Retained groups, attribution distributions, and token purity

Please report the number of retained groups, their attribution distributions, and evidence about token purity.

The dominant-axis attribution exceeded the permutation reference in all five seeds, and 11 of 57 retained non-clinical groups had mixed attributions. On ADNI at ρ = 0.5, we retained 13–14 of 16 groups per seed whose highest-attribution feature had an ATN(V) annotation. Among retained non-clinical groups, the median share of annotated attribution assigned to the dominant axis ranged from 0.78 to 0.81 across seeds. It exceeded the corresponding 95th percentile of the within-modality annotation-permutation null (0.73–0.75) in every seed.

Table R2. ADNI proto-token attributions for DECENT (ours) at ρ = 0.5. Counts are pooled across five seeds. Annotated share is the median fraction of total attribution on annotated features. A group is counted as mixed when its dominant axis accounts for less than 60% of its annotated attribution. The highest retention count and annotated share are in bold.

ModalityRetained (of 20)Annotated shareMixed / retained
Biospecimen180.748 / 18
Genetic190.682 / 19
Image200.831 / 20
Clinical100.56—

Clinical had only one annotated axis in this analysis, so we excluded its groups from the dominant-axis and mixing statistics. Eight of the eleven mixed groups came from biospecimen. Revised Appendix E.3 (Table 18) reports these distributions as a post hoc diagnostic of attribution concentration under the ATN(V) annotations.

AuthorsOpenReview ↗

Response to Reviewer LLYx (2/6)

W2. Advantage under natural missingness

The strongest gains occur under induced masking, while DECENT does not consistently outperform the strongest baselines under natural missingness.

We agree that DECENT is close to MUSE on ADNI under natural missingness (within 0.7 pp on both metrics). DECENT is designed to retain predictive performance as fewer modalities are available. Table 1 tests this by increasing the severity of induced masking from the original observation patterns (ρ = 0, naturally incomplete on ADNI and complete on MIMIC-IV). From ρ = 0 to 0.7, DECENT loses 6.3 pp in macro-AUC on ADNI and 6.7 pp on MIMIC-IV, compared with 8.9 and 10.1 pp for MUSE. DECENT leads macro-AUC in all six induced-missingness settings and macro-F1 in five of six. The submitted partition controls also show that learned grouping improves macro-F1 over modality-based grouping in all six settings with the same tokenizer, number of views, and graph encoder (Table 2(b)).

W2 (Q5). Adoption of each baseline

Please specify how each baseline is adopted, together with its hyperparameters, optimizer updates, and training epochs.

We appreciate the reviewer’s request for more detail on the baseline adaptations. We started from the authors’ code where available and used their published or repository settings where applicable. When tuning was needed, we included published settings among the candidates where available and selected configurations by validation macro-F1 at ρ = 0.5, with confirmation across seeds. Every method, including DECENT (ours), was evaluated on the same input features, patient splits, outcome labels, and missingness masks. Each method used one selected configuration per dataset across all missing rates and evaluation seeds. DECENT used the same validation criterion (24 configurations screened, 34 fits). We evaluated all test patients with the same five seeds and selected binary decision thresholds on validation data.

Table R3. Baseline adaptations and tuning. The two-stage baseline search varies learning rate and model width, followed by two method-specific factors.

MethodImplementation and input adaptationConfiguration source
GRAPEOfficial code; transductive graph over vector entriesTwo-stage search (20 fits)
M3Care, MUSEOfficial code; shared vectors passed to method-specific encodersTwo-stage search (23 fits each)
SimMLMOfficial code; modality experts adapted to vector inputsTwo-stage search (23 fits)
GTP-4o, HGMFRe-implemented from the papers for vector inputsTwo-stage search (23 fits each); early-stopping patience of 25 epochs
AMC, Flex-MoEOfficial code; vector patch embeddingsPublished settings on ADNI; two-stage search on MIMIC-IV
MoE-RetrieverOfficial code; vector patch embeddings and training-patient retrieval poolRepository settings on ADNI; two-stage search on MIMIC-IV

We have added the selected hyperparameters, epochs, and optimizer updates for every baseline and dataset in revised Appendix A.2 (Table 5).

Q5. Baseline results and original protocols

Please reconcile the baseline results with the corresponding original protocols.

Thank you for pointing out the difference between AMC’s published macro-F1 (54.82 on ADNI and 59.75 on MIMIC-IV) and our rerun at ρ = 0 (49.5 and 55.2). We reran AMC from its official implementation, matched four of its published training settings, and evaluated it under the same benchmark protocol as every other method in Table 1. We used AMC’s published ADNI configuration. On MIMIC-IV, validation selected the same learning rate, hidden width, batch size, and epoch count as in AMC’s paper.

Table R4. AMC’s published protocol and its adaptation to our shared benchmark.

ItemAMC paper / official codeShared benchmark in this paper
Metric averagingMacro-F1 in the official codeMacro-F1
Cohort sizes2,380 ADNI; 9,003 MIMIC-IVSame totals; fixed patient splits and availability masks (Table 3)
ADNI MRI input3D MRI volumes encoded by a 3D CNNSame preprocessed MRI feature vectors for all methods
MIMIC-IV outcomeMortality within one year of admissionSame first-admission one-year mortality labels for all methods (106/1,349 test patients positive, 7.9%)
MIMIC-IV decision ruleArgmax in the official codeSame validation procedure for all methods to select their own thresholds
ADNI trainingLR 1e-4, width 64, batch 32, 30 epochsSame values; 1,590 optimizer updates
MIMIC-IV trainingLR 1e-3, width 64, batch 64, 100 epochsSame values; 9,900 optimizer updates

Sharing the MRI features and mortality labels, and applying the same threshold-selection procedure, puts all methods on equal footing within each benchmark. The differences in MRI representation and evaluation protocol relative to AMC’s original setup (Table R4) may contribute to the gap between its published and rerun scores.

AuthorsOpenReview ↗

Response to Reviewer LLYx (3/6)

Q8. Clinical label and observation protocol

Please specify the clinical label and observation protocol more fully, and report realized missingness after the retain-one rule.

We agree that the observation windows and availability patterns are important for interpreting the benchmark. On ADNI, 35.6% of subjects are missing at least one modality at ρ = 0, while the evaluated MIMIC-IV cohort has all three modalities at ρ = 0.

  • ADNI. We use preprocessed clinical, biospecimen, MRI-derived, and genomic features following Flex-MoE. We exclude subjects whose diagnosis in ADNI’s DXSUM table is missing or changes across visits. The outcome is the stable diagnosis (CN, MCI, or AD). This is a retrospective classification task using the supplied features.
  • MIMIC-IV. We retain patients with more than one admission and use lab/vital, note, and ICD-9 features from the first admission. The one-year mortality window starts at the first admission. For the secondary outcome in Appendix C.2, inputs are from the first admission through discharge, and readmission is positive if the second admission starts within 30 days of that discharge.

Table R5. Cohort sizes, class counts, and availability. Restoration counts at ρ = 0.2 / 0.5 / 0.7 are from the fixed masks.

Dataset and quantityTrainValidationTestTotal
ADNI subjects1,6673523612,380
ADNI CN / MCI / AD730 / 595 / 342162 / 122 / 68164 / 127 / 701,056 / 844 / 480
ADNI missing ≥1 modality at ρ = 0594130124848 (35.6%)
ADNI restored by retain-one9 / 185 / 5131 / 31 / 1036 / 45 / 9916 / 261 / 715
MIMIC-IV patients6,3031,3511,3499,003
MIMIC-IV survived / died within one year5,757 / 5461,243 / 1081,243 / 1068,243 / 760
MIMIC-IV not readmitted / readmitted within 30 days4,351 / 1,952975 / 376945 / 4046,271 / 2,732
MIMIC-IV missing ≥1 modality at ρ = 00000
MIMIC-IV restored by retain-one47 / 777 / 2,1087 / 182 / 4577 / 157 / 45461 / 1,116 / 3,019

At each nominal masking rate ρ, we reuse the same random scores and restore the highest-scored modality if all observed modalities are masked. At ρ = 0.2 / 0.5 / 0.7, the expected post-restoration fractions of all patient–modality entries that are missing, including natural missingness, are 28.9% / 53.2% / 66.1% on ADNI and 19.7% / 45.8% / 58.6% on MIMIC-IV. The realized fractions require counting the missing entries in the final masks. Revised Appendix A.2 and Table 3 give the full original availability patterns and the observed restoration counts.

W2 (Q7). Parameter and computational costs

Please provide the total number of parameters, FLOPs, and end-to-end training and inference time of DECENT compared with all baselines.

We appreciate the reviewer’s request to quantify the cost of the two-stage pipeline. DECENT updates 5.42M of its 24.06M parameters in the supervised stage and has the third-shortest inference time among the ten methods profiled on ADNI.

Table R6. ADNI cost at ρ = 0.5 (seed 0, one RTX 3090). Stage 1 is DECENT’s Phase 1 or pretraining for HGMF and SimMLM. Task params are updated in the supervised stage. GFLOPs are per test patient (multiply–add = 2 FLOPs). Training time includes validation. Inference covers all 361 test patients and is averaged over five repeats. Memory is peak allocated CUDA memory. Column minima are in bold.

MethodTotal params (M)Task params (M)GFLOPsStage 1 (s)Task (s)Total train (s)Inference (s)Train GiBInfer GiB
GRAPE4.894.891.357—5,2115,2261.1634.4013.512
GTP-4o18.2918.290.239—1121230.2380.8900.344
HGMF330.52329.720.310185872860.3078.4113.689
M3Care39.4139.410.089—3473550.2610.7600.213
MUSE18.2118.210.037—1611690.1850.8200.271
AMC0.820.820.228—2502600.3802.0380.333
Flex-MoE2.372.370.059—6376450.6850.0640.028
MoE-Retriever3.713.710.248—2913011.4640.1750.053
SimMLM35.9935.990.0722052965100.2080.7130.243
DECENT (ours)24.065.420.1162181904160.2280.4430.295

On ADNI (1,667 training subjects), we select the task checkpoint by validation macro-F1. In submitted Table 2(b), learned grouping improves macro-F1 over modality-based and balanced random grouping in all six induced-missingness settings with the same tokenizer, number of views, and parameter count. This comparison isolates the partition rule at fixed capacity. Revised Appendix C.4 (Table 10) reports the profiling setup and results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.