- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper deals with irregular sampling and missing modalities in longitudinal multimodal data. It identifies and proves a representational limitation of linear state-space models whose transition operators are independent of modality availability. The paper then proposes CAMOS, which makes the transition operator a function of the modality availability pattern. The method is evaluated on two public longitudinal clinical datasets, ADNI and OASIS-3, using Alzheimer’s disease-related prediction tasks.
Critical strengths
- The paper identifies and proves the modality availability-additivity limitation of linear state-space layers whose transition operators do not depend on the modality availability pattern: their latent states cannot directly represent higher-order interactions among availability indicators.
- multi-facet experiments: CAMOS is evaluated on three complementary tasks, same-visit staging, landmark prediction, and longitudinal forecasting on ADNI dataset. And it demostrates stronger zero-shot transfer to the OASIS-3 dataset.
Critical weaknesses
- The motivation for incorporating availability into the transition operator is not fully convincing. As the paper pointed in (line 255-260) Theorem 5 establishes availability-additivity for an individual linear state-space layer; it does not show that deep networks with nonlinear readouts cannot model availability interactions, nor does it distinguish CAMOS from representation-level fusion methods such as MedFuse or DrFuse, whose encoders and fusion modules are nonlinear with respect to availability. If such interactions can already be modeled by adding nonlinear layers or fusion modules, why is it necessary to encode them in the transition operator, particularly given the additional structural constraints imposed to obtain the stability guarantees?
- Regarding the research gap, the paper claims that ''Multimodal fusion methods fuse after the temporal modelling has been applied, so nothing lets one modality shape another’s trajectory across a long gap.'' (line 053-056) This is too strong a claim without sufficient references or empirical evidence to support it. Why cannot the trajectory of one modality influence another in existing longitudinal multimodal fusion methods?
- The presentation could be improved for clarity. Several notations are undefined. In particular, in Figure 1, , , , and are not explicitly defined. Theorem 7 depends on the undefined boundary quantities and . Moreover, , , and are each overloaded for several unrelated objects. The framework figure does not clearly specify the inputs and outputs of each module, making the data flow less clear.
- About the experiments, several commonly used longitudinal multimodal baselines are missing, including FlexMoE[1], FuseMoE[2], UMSE[3], and Zhang et al., 2023's work[4]. Baselines such as MedFuse and DrFuse were not originally designed for longitudinal multimodal modeling; in their original settings, they use a single or last available CXR as input rather than modeling longitudinal observations for every modality. Therefore, they do not include dedicated temporal modeling for each modality. Further justification and implementation details for ''replacing their modality encoders with per-modality gated recurrent encoders over the visit sequence'' would help establish a clearer and more informative comparison.
References:
[1] Yun, Sukwon, et al. "Flex-MoE: Modeling arbitrary modality combination via the flexible mixture-of-experts."Advances in Neural Information Processing Systems 37 (2024): 98782-98805.
[2] Han, Xing, et al. "FuseMoE: Mixture-of-experts transformers for fleximodal fusion." Advances in Neural Information Processing Systems 37 (2024): 67850-67900.
[3] Lee, Kwanhyung, et al. "Learning missing modal electronic health records with unified multi-modal data embedding and modality-aware attention." Machine learning for healthcare conference. PMLR, 2023.
[4] Zhang, Xinlu, et al. "Improving medical predictions by irregular multimodal electronic health records modeling." International conference on machine learning. PMLR, 2023.
Questions for authors
- Do all baselines use the same task heads and jointly optimize the same multi-task losses (Equation 10) as CAMOS?
- Can the authors report performance under controlled missing rates and modality combinations, and analyze results by availability pattern to show when transition gating helps?
- MedFuse and DrFuse were originally evaluated on MIMIC-based datasets[5], and the MIMIC ecosystem provides longitudinal multimodal data, including CXRs, clinical notes, ECGs, and electronic health records. Could the authors explain why MIMIC was not included in the experiments? Evaluation on this dataset would help assess CAMOS in a broader longitudinal multimodal setting.
Reference: [1] Johnson, Alistair EW, et al. "MIMIC-IV, a freely accessible electronic health record dataset." Scientific data 10.1 (2023): 1.
6 comments
Critical Weaknesses:
W1: Remark 8 states that Theorem 5 concerns a single linear layer and does not separate CAMOS from MedFuse or DrFuse. We do not claim that transition gating is necessary. It is a sufficient mechanism with properties that readout-level reconstruction does not provide:
| Aspect | Readout-level or fusion-level reconstruction | Transition gating (CAMOS) |
|---|---|---|
| Isolation of an unobserved modality | Learned from the availability patterns seen in training | Exact for all patterns, including unseen ones (Proposition 3) |
| Where the interaction is formed | By a nonlinear readout acting on an additive summary of the history | Inside the recurrence, at the visit where it occurs |
| Stability under changing availability | No statement | Uniform in pattern and gap (Lemma 2, Proposition 6, Theorem 7) |
| Cost | Additional layers or fusion modules | 0.14% of the parameters per layer; constraints hold by construction (Eq. 5); parallel scan preserved (Lemma 14) |
The empirical evidence is consistent with this: CAMOS improves on MedFuse and DrFuse on all three tasks; on OASIS-3, where CSF is never observed, DrFuse falls to the single-class solution and CAMOS does not; ungated coupling is worse than no coupling on landmark prediction and forecasting (Table 3); and the effect of the gate grows with missingness (Q2).
W2: The statement holds for late-fusion designs such as MedFuse and DrFuse, where each modality has its own temporal encoder and the temporal state of one modality is not an input to the update of another. It does not hold for methods that interleave cross-modal attention with temporal modelling, such as Zhang et al. (2023), UMSE and Coupled Mamba.
| Family | Cross-modal influence over time | Treatment of the gap | Guarantee under a missing modality |
|---|---|---|---|
| Late fusion (MedFuse, DrFuse) | None inside the temporal encoders | Per-modality encoder input | Learned representation of the missing modality |
| Interleaved attention or coupled states (Zhang et al., 2023; UMSE; Coupled Mamba) | Learned, in a discrete token or state space | Embedding or feature | None stated |
| CAMOS | Coupling inside the differential equation | Integration step | Exact isolation and stability bounds |
W3:
| Symbol | Definition |
|---|---|
| , (Figure 1) | The projected input and input matrix of Eqs. 1 and 6, with the modality index |
| , (Theorem 7) | Convention , hence , so the first visit is not counted as a transition; is the energy of the initial state from the static encoder |
| The -th smallest of , matched to the -th smallest eigenvalue in Weyl’s inequality | |
| Projection width; also the forecast horizon index and the modality token | |
| Static covariates; also in Proposition 11 and the Laplace variable in Proposition 12 | |
| Coupling strength ; also the horizon discount |
For , and the meaning is fixed by context in each case. The data flow in Figure 1 is:
| Module | Input | Output |
|---|---|---|
| Encoders | , , , | , |
| GCO stack ( blocks) | , , , | , split into modality tokens |
| Within-visit attention | , | |
| Task heads | , , |
W4: Appendix I compares against twelve baselines from four families, and CAMOS exceeds all twelve on every ADNI staging and landmark metric:
| Family | Baselines |
|---|---|
| Uncoupled oscillatory SSMs | LinOSS-IM, LinOSS-IMEX, D-LinOSS |
| General and coupled SSMs | S5, Mamba, Coupled Mamba |
| Irregular-time and missing-value models | GRU-D, mTAN, Raindrop, ODE-RNN |
| Clinical fusion models | MedFuse, DrFuse |
mTAN is the temporal module underlying the irregularity-aware embeddings of FuseMoE and Zhang et al. (2023), so that family is represented. Flex-MoE is formulated for a single observation per subject and needs the same longitudinal adaptation as MedFuse and DrFuse.
In their original form, MedFuse and DrFuse use a single or last image and would be compared on unequal information. We therefore replace each modality encoder by a gated recurrent encoder over that modality’s visit sequence and keep the published fusion module of each method, so both receive more temporal information than in their original form, with the same heads and losses as CAMOS.
Thank you for the clarifications and the experiments.
- Regarding Q2, in the experiment result table, compared to full model, the "CSF removed" "CSF + imaging removed" "CSF/imaging dropped 75%" variants actually has two changes: the input data, and the transition gating, which makes the drop of the performance hard to be attributed. Further control of the experiment would be helpful.
- Regarding Q3, a follow up question is why CAMOS can not fit MIMIC dataset (with the same irregularity and missingness problem)?
We thank the reviewer for the valuable feedback. We have addressed all comments and updated the paper accordingly wherever possible.
Questions For Authors:
Q1: Yes.
MedFuse and DrFuse use per-modality gated recurrent encoders over the visit sequence, so they see the same temporal information as the state-space models; the oscillatory baselines receive the availability vector and the gaps as extra channels (Appendix G, baseline adaptation). The ℓ1 penalty on the couplings in Eq. 10 applies only to CAMOS, since no baseline has a coupling matrix.
Q2: The only systematic case in the original submission is OASIS-3, where CSF is never observed. We now add a controlled analysis on the trained ADNI checkpoints (full table in our response to Reviewer yCSE, Question 6). With CSF and imaging both removed at test time, CAMOS keeps a macro-F1 of 0.765 (all observed: 0.779) and a landmark AUROC of 0.949 (0.961), whereas the strongest baseline, DrFuse, falls from 0.735 to 0.690 in macro-F1 and from 0.939 to 0.903 in AUROC. The advantage of CAMOS therefore increases as more biomarkers are missing.
The ablation shows that the availability gate contributes to this. The table gives the ungated variant minus the full model, with 95% paired bootstrap intervals over test subjects:
With everything observed, the ungated variant is already worse than the full model in Table 3 (landmark AUPRC 0.7916 vs. 0.8687, forecasting MAE about 11% higher). Its macro-F1 and AUROC gaps (0.010 and 0.013) are small and, with 162 test subjects, not individually distinguishable from zero, whereas the AUPRC gap is. The gaps grow when several biomarker modalities are missing together: with CSF and imaging both removed, or three quarters of their blocks dropped, the ungated variant loses 0.021 to 0.025 macro-F1, 0.027 to 0.031 AUROC and 0.118 to 0.132 AUPRC, all with intervals excluding zero, roughly 1.5 to 2.5 times the all-observed gaps. This is consistent with the explanation given with Table 3: without the gate, unobserved or unreliable states propagate across modalities, whereas with the gate an unobserved oscillator is isolated from the others (Proposition 3).
Q3:
Our target regime is continuous-time dynamics over multi-year gaps with procedure-level missingness in slowly progressing disease, and MIMIC’s regime lies outside it. It would be a worthwhile test of generality, and we list it as future work.