acceptodds
Under review as a conference paper at ICLR 2027

LongMoE: Longitudinal Multimodal Learning via Trajectory-Aware Mixture-of-Experts

Abstract

Multimodal clinical learning is increasingly important for integrating diverse patient data, including imaging, text, and personalised health records. However, it faces two fundamental challenges: i) modality missingness, where arbitrary subsets of modalities are unavailable at a given patient visit, ii) longitudinal dynamics, where the diagnostic significance of an observation depends on the patient's evolving disease trajectory over time. Existing methods address these challenges in isolation: missing-modality frameworks treat each visit as an independent static snapshot and discard temporal context, while longitudinal models often assume complete modality availability and degrade under systematic modality incompleteness. We propose LongMoE (Longitudinal Mixture-of-Experts), the unified framework to jointly address both challenges. LongMoE combines a context-aware imputation module with an attentional tokenization module that captures frequency-domain temporal patterns across irregular visit sequences, a trajectory-aware encoder for modeling disease progression, and context-conditioned Sparse MoE routing for patient-specific expert selection. Experiments on ADNI, OASIS-3, and MIMIC-IV show that LongMoE improves robustness under missing or weak contemporaneous modalities and remains competitive in full-modality settings, establishing a strong foundation for longitudinally-aware multimodal clinical learning.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Mixture-of-Experts, Multimodal Learning, Longitudinal Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_X36GOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes LongMoE for clinical prediction when visits are irregular and some modalities are missing at each visit. It has four parts:

  1. Per-modality encoders.
  2. An imputation step. A missing modality is filled with its last observed embedding, or, if there is none, with an MLP applied to the mean of the observed embeddings at that visit.
  3. A tokenizer. It applies self-attention across modalities and adds a multi-frequency sinusoidal encoding of the visit time.
  4. A causal 2-layer Transformer over the visits, followed by a sparse MoE with hard top-1 routing and “shared” experts. Predictions are combined across visits, weighted by router confidence.

Experiments cover ADNI and OASIS-3 (CN/MCI/AD staging) and MIMIC-IV (one-year mortality), across all 2^L − 1 modality combinations. Large gains are reported when modalities are missing, especially in settings without clinical scores.

Critical strengths

1. The full-trajectory masking control (Appendix A.9.1, Table 14).

This is the paper's strongest evidence. When the excluded modality is withheld at every visit, so the model cannot carry earlier observations of it forward, LongMoE still keeps large margins on ADNI (B: 0.862 vs. 0.642; G: 0.852 vs. 0.639; I: 0.875 vs. 0.714). It also leads on three of four sparse single-modality settings on MIMIC-IV. The authors report that the advantage disappears on OASIS-3 rather than leaving that out.

Why this matters: my main concern is that the headline gains come from LongMoE seeing history the baselines do not. Table 14 is the only result that partly addresses that concern, and it suggests the longitudinal design helps even with the information gap reduced. It is the main reason I see a viable paper here. However, its comparison column reuses the same baselines whose numbers look broken (e.g., Lee-MMGRU at 0.5000 ± 0.0000), so it cannot yet support acceptance by itself.

2. A relevant problem with a broad evaluation.

The paper tackles missing modalities and irregular visits together, a combination that is common in clinical cohorts. It evaluates on three datasets covering two task types (neurodegenerative staging and ICU mortality), every one of the 2^L - 1 modality combinations, five seeds and three metrics, and lists every model's hyperparameters (Appendix A.12).

Critical weaknesses

1. Baseline protocol. Section 5 (l. ~298–302) says Flex-MoE and FuseMoE were augmented with visit history, but l. 320–321 says visit-independent baselines lack history and the Table 14 caption says static baselines receive one row per patient. Which is correct? Could all baselines be re-run with the same current-visit masking protocol LongMoE uses (prior visits intact)?

2. History-only controls. Could you report (a) LongMoE with the current visit fully masked, and (b) simple longitudinal baselines on the same history (e.g., last observation carried forward + logistic regression, or predicting the previous diagnosis), in particular for the ADNI B-only and G-only rows where the gains are largest?

3. Clinical variables and leakage. Which ADNI clinical variables are retained after "label-leaking columns" are excluded (l. ~276)? Are prior diagnoses, CDR-SB, or diagnosis-derived fields in the history? How are the OASIS-3 CN/MCI/AD labels defined, and is CDR an input at the evaluated visit? For MIMIC-IV, is the prediction made at admission or discharge of the final visit, are in-hospital deaths included as positives, and could final-visit notes/ICD codes carry end-of-life information (hospice discharge, palliative care codes)? Were ICD-10 admissions dropped or mapped to ICD-9?

4. Baseline and metric anomalies. How do you explain:

  • Lee-MMGRU scoring 0.5000 ± 0.0000 AUC on ADNI B, G and B,G, and identical values for C / B,C / C,G / B,C,G (all 0.9704) and I / I,B / I,G / I,B,G (all 0.5771)?
  • MulT, Flex-MoE and FuseMoE at ~0.50 AUC and 0.3333 accuracy on OASIS-3 from CDR/MMSE alone, while Lee-MMGRU reaches 0.88?
  • Macro-F1 of 0.0000 in Table 11? On ADNI, Lee-MMGRU's 0.2047 is exactly the always-MCI macro-F1, so the metric is computed correctly there; on OASIS-3 no constant predictor can score below ≈0.085, which also rules out the 0.0131 and 0.0240 entries.
  • Cross-table conflicts: w/o LE ΔAUC −0.0335 (Table 5) vs. −0.001 (Table 15); LongMoE ADNI F1 0.8064 (Table 5) vs. 0.8843 (Table 9); OASIS-3 F1 0.6537 (Table 16) vs. 0.4886 (Table 11); identical w/o LE and w/o Imp-ZP rows in Tables 15–17?
Questions for authors
  1. Baseline protocol. Section 5 (l. 296-299) says Flex-MoE and FuseMoE were augmented with longitudinal history (a mean-pooled summary of preceding visits and a GRU over visits, respectively). The Table 14 caption says the static baselines receive one row per patient. Which is correct? Could you re-run all baselines with the same visit history that LongMoE receives?

  2. History-only control. Could you report (a) LongMoE given only prior visits, with the current visit fully masked, and (b) simple longitudinal baselines using the same history, such as carrying the last observation forward plus logistic regression, or predicting the previous diagnosis? These would show how much of the gain comes from the architecture rather than from access to history.

  3. Clinical variables and leakage. Which clinical variables are used for ADNI and OASIS-3 (l. 277 says only that "label-leaking columns [were] excluded")? Are prior-visit diagnoses, CDR-SB, or any fields derived from the diagnosis available as history? For MIMIC-IV, when is the prediction made relative to the one-year mortality window? Could notes or ICD codes at the final visit encode end-of-life information, such as hospice discharge or palliative care codes?

  4. Baseline and metric anomalies. How do you explain:

    • Lee-MMGRU scoring exactly 0.5000 ± 0.0000 AUC on every ADNI combination without C or I, and giving identical results for C, B,C, C,G and B,C,G?
    • MulT, Flex-MoE and FuseMoE scoring about 0.50 AUC on OASIS-3 from CDR/MMSE alone, while Lee-MMGRU reaches 0.88 on the same input?
    • The macro-F1 values of 0.0000 in Table 11, which are impossible for a 3-class task?
4 comments

We thank the reviewer for the valuable feedback. We have addressed all comments and updated the paper accordingly wherever possible.

Questions For Authors:

Q1: Table 14 is correct for the submitted results: TF, MulT, Flex-MoE, and FuseMoE received one cross-sectional row per patient. For a fair history-matched control, we temporalized Flex-MoE, the closest missing-modality MoE baseline, using its own encoders, missing-modality bank, experts, and routing at each visit, followed by a temporal Transformer. It receives exactly the same patients, visit histories, masks, splits, and five seeds as LongMoE, and reuses no LongMoE components. Results are five-seed AUROC on ADNI, mean ±\pm standard deviation. II, BB, CC, and GG denote imaging, biospecimen, clinical, and genetic modalities.

Current-visit masking Only the listed modalities are available at the prediction visit; every model receives the same naturally observed preceding history.

Current modalitiesLongMoETemporal Flex-MoECapacity-matched Temporal Flex-MoE
B0.9476±0.00800.9476 \pm 0.00800.9212±0.00950.9212 \pm 0.00950.9386±0.00940.9386 \pm 0.0094
C0.9723±0.00540.9723 \pm 0.00540.9496±0.00920.9496 \pm 0.00920.9613±0.00510.9613 \pm 0.0051
G0.9490±0.00820.9490 \pm 0.00820.9203±0.00860.9203 \pm 0.00860.9396±0.00480.9396 \pm 0.0048
I0.9479±0.00650.9479 \pm 0.00650.9199±0.00980.9199 \pm 0.00980.9356±0.00750.9356 \pm 0.0075
B,C0.9719±0.00560.9719 \pm 0.00560.9498±0.00920.9498 \pm 0.00920.9617±0.00520.9617 \pm 0.0052
B,G0.9485±0.00810.9485 \pm 0.00810.9199±0.00810.9199 \pm 0.00810.9403±0.00490.9403 \pm 0.0049
C,G0.9729±0.00510.9729 \pm 0.00510.9510±0.00900.9510 \pm 0.00900.9655±0.00330.9655 \pm 0.0033
I,B0.9480±0.00620.9480 \pm 0.00620.9199±0.01000.9199 \pm 0.01000.9362±0.00740.9362 \pm 0.0074
I,C0.9715±0.00530.9715 \pm 0.00530.9452±0.00970.9452 \pm 0.00970.9593±0.00610.9593 \pm 0.0061
I,G0.9488±0.00610.9488 \pm 0.00610.9187±0.00820.9187 \pm 0.00820.9377±0.00280.9377 \pm 0.0028
B,C,G0.9726±0.00530.9726 \pm 0.00530.9513±0.00930.9513 \pm 0.00930.9658±0.00330.9658 \pm 0.0033
I,B,C0.9714±0.00530.9714 \pm 0.00530.9454±0.01000.9454 \pm 0.01000.9595±0.00610.9595 \pm 0.0061
I,B,G0.9486±0.00620.9486 \pm 0.00620.9185±0.00860.9185 \pm 0.00860.9384±0.00220.9384 \pm 0.0022
I,C,G0.9723±0.00440.9723 \pm 0.00440.9474±0.00890.9474 \pm 0.00890.9636±0.00290.9636 \pm 0.0029
I,B,C,G0.9721±0.00450.9721 \pm 0.00450.9481±0.00850.9481 \pm 0.00850.9638±0.00260.9638 \pm 0.0026

Full-trajectory masking Only the listed modalities are retained throughout the record, giving every model the same restricted longitudinal information.

Modalities retained throughoutLongMoETemporal Flex-MoECapacity-matched Temporal Flex-MoE
B0.8543±0.01070.8543 \pm 0.01070.8246±0.01020.8246 \pm 0.01020.8385±0.00780.8385 \pm 0.0078
C0.9734±0.00420.9734 \pm 0.00420.9558±0.01000.9558 \pm 0.01000.9664±0.00350.9664 \pm 0.0035
G0.8509±0.01160.8509 \pm 0.01160.8099±0.01350.8099 \pm 0.01350.8181±0.02050.8181 \pm 0.0205
I0.8690±0.00700.8690 \pm 0.00700.8328±0.00990.8328 \pm 0.00990.8357±0.01000.8357 \pm 0.0100
B,C0.9727±0.00440.9727 \pm 0.00440.9530±0.01080.9530 \pm 0.01080.9636±0.00600.9636 \pm 0.0060
B,G0.8600±0.00640.8600 \pm 0.00640.8305±0.01300.8305 \pm 0.01300.8400±0.01290.8400 \pm 0.0129
C,G0.9723±0.00520.9723 \pm 0.00520.9554±0.00970.9554 \pm 0.00970.9673±0.00260.9673 \pm 0.0026
I,B0.8737±0.00560.8737 \pm 0.00560.8434±0.00710.8434 \pm 0.00710.8444±0.01110.8444 \pm 0.0111
I,C0.9714±0.00580.9714 \pm 0.00580.9476±0.00890.9476 \pm 0.00890.9625±0.00490.9625 \pm 0.0049
I,G0.8736±0.00620.8736 \pm 0.00620.8415±0.00680.8415 \pm 0.00680.8406±0.01780.8406 \pm 0.0178
B,C,G0.9721±0.00500.9721 \pm 0.00500.9540±0.00990.9540 \pm 0.00990.9669±0.00300.9669 \pm 0.0030
I,B,C0.9714±0.00530.9714 \pm 0.00530.9454±0.01000.9454 \pm 0.01000.9595±0.00610.9595 \pm 0.0061
I,B,G0.8780±0.00410.8780 \pm 0.00410.8474±0.00580.8474 \pm 0.00580.8526±0.01680.8526 \pm 0.0168
I,C,G0.9723±0.00480.9723 \pm 0.00480.9505±0.00780.9505 \pm 0.00780.9650±0.00290.9650 \pm 0.0029
I,B,C,G0.9721±0.00450.9721 \pm 0.00450.9481±0.00850.9481 \pm 0.00850.9638±0.00260.9638 \pm 0.0026

LongMoE is best for all 15/15 combinations under both protocols. Its mean advantage over the capacity-matched Temporal Flex-MoE is +0.0099 AUROC under current-visit masking and +0.0168 AUROC under full-trajectory masking.

These results show i) the temporal baseline reaches 0.92 to 0.94 on B-only and G-only under current-visit masking, against 0.49 to 0.64 for the static baselines in Table 1. Most of the headline gap in Table 1 was therefore due to access to history; ii) a consistent but much smaller advantage remains under identical information. We also trained GRU and Transformer controls that share LongMoE's imputer (Reviewer VRho, Q1). These are comparable to LongMoE, which locates the remaining advantage in the history-based imputation and not in the MoE head. Because the remaining methods are intrinsically cross-sectional, longitudinalizing them requires adding a non-unique temporal architecture. They will be labelled as static references and will not be described as history-matched.

Q2: (a) Current visit fully masked On fixed cohorts containing at least five visits, we removed all time-varying modalities from the final kk visits; k=1k = 1 is the requested history-only setting.

DatasetNatural AUROCHistory only (k=1k=1)Δ\DeltaAUROC, k=1k=1k=2k=2k=3k=3k=4k=4
OASIS-30.8083±0.01670.8083 \pm 0.01670.7648−0.0435-0.0435−0.0861-0.0861−0.1049-0.1049−0.1356-0.1356
MIMIC-IV0.8795±0.00170.8795 \pm 0.00170.8517−0.0278-0.0278−0.0515-0.0515−0.0768-0.0768−0.1007-0.1007

History alone therefore carries most of the signal on these tasks.

(b) Simple longitudinal baselines We have not run last-observation-carried-forward with logistic regression. The closest existing evidence is the matched GRU and Transformer controls, which receive the same history and imputer as LongMoE and perform comparably (Reviewer VRho, Q1). Together with (a), this indicates that access to history, and not the LongMoE architecture, accounts for most of the gain in the B-only and G-only rows.

Q3:

DatasetClinical inputsLabelDiagnosis-derived inputs
ADNIMMSE, ADAS-13, CDR-SB, FAQFirst valid diagnosis (see below)Diagnosis columns excluded; CDR-SB retained
OASIS-3MMSE and age at visitDerived from global CDR at the last visitCDR global, CDR sum of boxes, CDR sub-scores, and prior labels excluded
MIMIC-IVLaboratory means, discharge ICD categories, discharge summary, age and sexDeath within 365 days after the final dischargeEnd-of-life codes can be present

For ADNI, diagnosis columns are not model inputs. The preprocessing uses the first valid diagnosis as the label while retaining later visits, and 77.8% of patients have post-label observations. The submitted ADNI task is therefore retrospective classification of the initial diagnosis from the full record, not staging at the final visit as Section 5 states. This also explains the nearly flat current-visit masking curve on ADNI, since the visit to which the label refers remained in the history.

For OASIS-3, the clinical modality contains only MMSE and age at visit. The description “CDR/MMSE assessments” in Section 5 has been corrected in the uploaded revised paper. CDR is the source of the label and is not an input at any visit.

For MIMIC-IV, prediction in the submitted pipeline is made after the final discharge, and the label is death within the following 365 days. The diagnosis input includes ICD-10 coded admissions, so the description “ICD-9 codes” in Section 5 has been corrected. End-of-life information can be present:

EvidenceOne-year deathsSurvivors
Z66 (do not resuscitate) at the final admission20.2%2.1%
Z51 (includes palliative care) at the final admission12.6%0.8%

Final-admission ICD codes alone reach 0.880 AUROC.

Q4:

Lee-MMGRU: its stage-2 L1 logistic regression was fitted once on the full-modality representation and only masked at evaluation. Across all three ADNI seeds, it selected zero coefficients for every biospecimen and genetics feature; imaging retained 11, 8, and 0 coefficients. Consequently, B/G-only predictions are constant (AUROC 0.5), and adding B or G to C or to I cannot change the prediction, producing the identical rows (0.9704 and 0.5771). We will refit stage 2 separately for each modality combination and add modality dropout.

OASIS-3 baselines: the stored mask order is [MRI, clinical, genetics], whereas MulT, Flex-MoE, and FuseMoE read it alphabetically as [clinical, genetics, MRI]. Under C-only evaluation, the clinical embedding is therefore multiplied by the MRI flag and erased. Lee-MMGRU uses a separate correctly indexed loader, explaining its 0.88 AUROC. The affected static-baseline results are invalid and will be rerun using name-indexed masks, with unit tests for mask alignment in every loader.

Table 11 F1: these values are not macro-F1. They are AD-vs-rest positive-class F1 at a validation-selected 99%-specificity threshold. A value of zero occurs when no positive test sample exceeds that threshold, and the 0.0131 and 0.0240 entries arise in the same way. We will label this metric explicitly and separately report true three-class macro-F1 from argmax predictions.

Thank you for the extensive rebuttal. However, after re-reading the manuscript and your response, I would like to maintain my score.

Reviewer_Bo2nOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

LongMoE studies two orthognal problems of longitudal temporal sequences and missing modalities by proposing history-based imputation and a causal transformer with MoEs. Mostly validated on clinical data.

Critical strengths
  • Addresses a practical problem.
  • Results seem strong on ADNI dataset.
Critical weaknesses
  • Related-work positioning needs clarification. ShaSpec uses sample-dependent feature completion, making its characterization as population-based imputation inaccurate. Also other recent methods like FlexMoE,M3Care and MAESTRO are missing from references.

  • Evaluation does not explicitly characterize consecutive missing visits, and is a bit limiting. Overall I find it hard to understand what specifically requires the proposed longitudinal architecture beyond a temporal extension of existing missing-modality methods. Can the authors do a matched comparison?

Questions for authors

(Please see weakness)

  • Additionally, I am curious how would LongMoE handle modalities missing for consecutive visits or arriving asynchronously, as considered through sequential fusion in MultiModN? I understand this may be a bit out of scope, but a discussion of these practical settings and limitations would be helpful in clinical longitudinal settings context.
3 comments

We thank the reviewer for the valuable feedback. We have addressed all comments and updated the paper accordingly wherever possible.

Critical Weaknesses:

W1: ShaSpec performs sample-dependent completion from the shared features of the same sample, so describing it as population-based imputation was inaccurate, and the sentence will be corrected. Flex-MoE is cited in the submission as Yun et al. (2024). We have added M3Care, MAESTRO, and MultiModN in the uploaded revised paper.

The distinction we draw is the source of the completing information. Existing methods complete or bypass a missing modality using the current sample, learned patient-independent embeddings (Flex-MoE), or other patients, whereas LongMoE retrieves the modality from the same patient's earlier visits.

W2:

Matched comparison

We constructed Temporal Flex-MoE: Flex-MoE's own encoders, missing-modality bank, experts, and routing applied at every visit, followed by a temporal Transformer. It receives the same patients, visit histories, masks, splits, and seeds as LongMoE, in original and capacity-matched sizes, and reuses no LongMoE components.

ADNI protocolCombinations where LongMoE is highestMean AUROC gain over capacity-matched Temporal Flex-MoE
Current-visit masking15/15+0.0099
Full-trajectory masking15/15+0.0168

The complete tables are given in our response to Reviewer X36G.

Consecutive missing visits

We explicitly evaluated consecutive missing visits by removing each dynamic modality from the final k∈{1,2,3,4}k \in \{1,2,3,4\} visits while preserving all other naturally observed inputs. This is stricter than independent random masking because the model cannot recover the modality from an immediately adjacent visit.

Consecutive missing visitsLongMoETemporal Flex-MoE
10.96450.9559
20.96230.9519
30.95920.9465
40.95380.9403

The margin grows from approximately 0.009 to 0.014 AUROC as the missing interval lengthens. This is the specific behaviour that a temporal extension of a per-visit missing-modality method does not provide. Such an extension first completes each visit using learned, patient-independent embeddings and then compresses the completed visits into a generic temporal state. LongMoE instead performs observation-conditioned, modality-specific retrieval from the patient’s available history, marks the retrieved value as imputed, and can therefore use an informative observation several visits earlier when the modality is absent consecutively.

Questions For Authors:

Q1: LongMoE does not require every modality to be observed synchronously at every visit. Its observation mask is defined independently for each modality and visit. When modality mm is absent at visit tt, the strictly causal longitudinal imputer uses only that modality’s observations at visits s<ts < t. Absolute time and time since the previous visit are included in retrieval, while the tokenizer explicitly identifies whether the current value is observed or reconstructed from history. Thus, several consecutive missing visits do not break the trajectory: LongMoE can retrieve an older observation and does not require the immediately preceding visit to contain that modality. If the modality has never been observed, it uses its learned fallback; it does not manufacture unavailable patient-specific information.

We evaluated this directly on fixed cohorts containing at least five visits by removing all time-varying modalities from the final kk visits. We use OASIS-3 and MIMIC-IV here because their outcomes are associated with the final visit.

DatasetNatural AUROCΔ\DeltaAUROC, k=1k=1k=2k=2k=3k=3k=4k=4
OASIS-30.8083±0.01670.8083 \pm 0.0167−0.0435-0.0435−0.0861-0.0861−0.1049-0.1049−0.1356-0.1356
MIMIC-IV0.8795±0.00170.8795 \pm 0.0017−0.0278-0.0278−0.0515-0.0515−0.0768-0.0768−0.1007-0.1007

These results show gradual, monotone degradation as the last available information becomes older, with no abrupt failure after the first missing visit.They also expose the appropriate limitation: historical retrieval can mitigate consecutive absence, but cannot replace information that has never been recorded, and performance decreases when all dynamic modalities remain unavailable for increasingly long periods

On the same cohorts, LongMoE loses less AUROC than the matched GRU, which receives the same history and imputer (paired difference in Δ\DeltaAUROC, LongMoE minus GRU, 95% CI over five seeds; positive values mean LongMoE degrades less):

Datasetk=1k=1k=2k=2k=3k=3k=4k=4
MIMIC-IV+0.0075 [+0.0043, +0.0110]+0.0136 [+0.0095, +0.0180]+0.0179 [+0.0130, +0.0229]+0.0175 [+0.0118, +0.0231]
OASIS-3+0.0084 [−0.0148, +0.0346]+0.0329 [+0.0045, +0.0648]+0.0411 [+0.0082, +0.0756]+0.0451 [+0.0065, +0.0829]

On MIMIC-IV the difference is positive in all five seeds at every kk; on OASIS-3 it is resolved for k≥2k \ge 2.

For asynchronous arrival, there are two distinct settings:

  • Across-visit or delayed modality arrival: this is already supported. A modality is marked missing until it becomes available, and its actual observation time and irregular inter-visit interval are encoded. Inference can be updated when the result arrives.

  • Within-visit event ordering: the present implementation aggregates observations into visit-level modality slots. It therefore does not preserve whether, for example, a laboratory result arrived before or after an imaging result within the same visit. MultiModN’s sequential fusion directly models this finer event-level ordering.

Accordingly, LongMoE handles asynchronous availability at the longitudinal visit level, but is not currently an event-stream model. Extending it to within-visit asynchronous data would replace each visit-level slot with timestamped modality-arrival tokens while retaining the same causal history retrieval and routing mechanism; this is untested. A direct comparison with MultiModN would require event-level preprocessing and prediction cutoffs defined before each arrival.

I thank the authors for their response, but the results in W2 confirm my concern of matched comparison with exisitng backbones. The gain is marginal. In the current form I think the paper's contributions and performance are thin and hence, I retain my rating.

Reviewer_BgsQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LongMoE, a framework for learning from longitudinal multimodal clinical data with missing modalities. Missing modality embeddings are imputed by carrying forward a previous observation or applying a learned MLP to the mean-pooled embeddings of available modalities at the current visit. Modality embeddings are combined through self-attention and augmented with a multi-frequency sinusoidal timestamp encoding. A causal Transformer models dependencies across visits, followed by a mixture-of-experts module and router-score-based aggregation of visit-level predictions. Experiments evaluate Alzheimer's disease stage classification on ADNI and OASIS-3 and one-year mortality prediction on MIMIC-IV.

Critical strengths
  1. Clinically relevant motivation. Missing modalities and irregular longitudinal observations are common in clinical data. Addressing these challenges together is a meaningful objective for multimodal clinical prediction.
  2. Evaluation across different clinical settings. The experiments include two Alzheimer’s disease cohorts, ADNI and OASIS-3, and a hospital-based mortality prediction task on MIMIC-IV. This provides an opportunity to assess the framework across different modality combinations, patient populations, and prediction tasks.
Critical weaknesses
  1. The statements that "no existing model jointly addresses modality missingness and longitudinal dynamics” (Introduction) and that Transformer-based models have not been adapted to the joint challenges of irregular sampling, arbitrary modality combinations, and population heterogeneity appear not sufficiently supported by the discussion of prior work. Specifically, FuseMoE already considers missing and irregularly sampled modalities. However, the authors do not clearly explain its limitations in modeling longitudinal multimodal data or how LongMoE overcomes them. The discussion should also include relevant methods such as UTDE, UMSE, and TAMME, clarify their limitations in jointly handling modality missingness and longitudinal dynamics, and explain how LongMoE addresses these limitations.

  2. The method novelty and justification of several design choices remain insufficiently established. (1)Carry-forward imputation uses historical information, but it does not explicitly model how a missing modality evolves with elapsed time or the patient’s trajectory. The fallback strategy combines mean pooling with an MLP conditioned on currently observed modalities. These are reasonable design choices, but their methodological contribution appears limited. (2)Temporal encoding and fusion. The viscode module applies sinusoidal features to continuous timestamps and projects them through an MLP. Although this differs from the original Transformer’s encoding of discrete positions, it builds on established temporal representation techniques. Similarly, modality self-attention, additive fusion with temporal embeddings, and a causal Transformer encoder are familiar components. The paper should more clearly articulate what is novel in their integration. (3)Aggregation across visits. Equation 4 aggregates visit-level logits using weights derived from the selected experts’ routing scores. The interpretation of larger routing scores as indicating “more informative or reliable observations” (lines 260–261) requires further justification. A routing score reflects expert selection and is not necessarily a calibrated measure of observation reliability or relevance to the prediction target. Moreover, although each visit representation incorporates its preceding history, an earlier representation does not include subsequent changes in the patient’s condition. It is therefore unclear when aggregating earlier visit-level logits is preferable to predicting directly from the final contextualized representation, especially when clinical status changes substantially between visits.

  3. The baseline coverage could be strengthened. Beyond FuseMoE and Flex-MoE, the comparisons rely on relatively early approaches, including TF, MulT, and Lee-MMGRU. More recent methods targeting missing modalities or multimodal temporal modeling could be considered. Potentially relevant works include UTDE, UMSE, TAMME, DrFuse and DiPro. The authors should discuss their applicability and include those most closely aligned with the experimental setting, or explain why a direct comparison would be inappropriate.

[References]

  1. [UTDE] Zhang, Xinlu, et al. "Improving medical predictions by irregular multimodal electronic health records modeling." International conference on machine learning. PMLR, 2023.
  2. [UMSE] Lee, Kwanhyung, et al. "Learning missing modal electronic health records with unified multi-modal data embedding and modality-aware attention." Machine learning for healthcare conference. PMLR, 2023.
  3. [TAMME] Susetzky, Tobias, et al. "A Holistic Time-Aware Classification Model for Multimodal Longitudinal Patient Data." International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2025.
  4. [DrFuse] Yao, Wenfang, et al. "DrFuse: Learning disentangled representation for clinical multi-modal fusion with missing modality and modal inconsistency." Proceedings of the AAAI conference on artificial intelligence. Vol. 38. No. 15. 2024.
  5. [DiPro] Liu, Chen, et al. "Multimodal Disease Progression Modeling via Spatiotemporal Disentanglement and Multiscale Alignment." Advances in Neural Information Processing Systems 38 (2026): 132539-132566.
Questions for authors
  1. MIMIC-IV preprocessing and temporal resolution. The paper states that “lab values and vitals are aggregated over the last visit window.” Could the authors clarify the duration of this window and the aggregation method? Vital signs in ICU settings are often recorded at intervals of minutes and can capture rapid changes in a patient’s condition. Aggregating these measurements into a single visit-level representation may obscure clinically relevant fluctuations or episodes of acute deterioration. The rationale for this preprocessing choice is therefore less clear in an acute-care setting than in slowly progressing conditions such as Alzheimer’s disease. Could the authors justify this aggregation strategy and evaluate whether preserving within-visit temporal dynamics improves performance?
  2. Visit aggregation and routing scores. What evidence supports interpreting the routing scores in Equation (4) as measures of observation informativeness or reliability? How does the aggregation handle earlier visits that are confidently routed but less relevant to the patient’s current condition?
3 comments

We thank the reviewer for the valuable feedback.

Questions For Authors:

Q1: The description has been updated in the uploaded revised paper. A MIMIC-IV visit corresponds to a hospital admission, not an ICU stay. The “visit window” is therefore the interval between admission and discharge, not a fixed number of hours.

ModalityAdmission-level representationAggregation/dimension
Laboratory measurementsSelected structured lab events within the admissionMean for each of the 50 most frequent laboratory variables
Clinical notesDischarge-summary text256-dim TF-IDF representation
DiagnosesAdmission ICD categories200-dim multi-hot representation
DemographicsAge & sex2 variables
Minute-level charted vital signsNot included in the pipelineNot applicable

This temporal resolution is aligned with the prediction target, one-year mortality after discharge, and with LongMoE’s outer temporal axis, which models sequences of hospital admissions. Admission-level aggregation also prevents frequently sampled variables from dominating solely because of their measurement density. The model is not intended as a minute-level ICU deterioration predictor. Within-admission deterioration and cross-admission disease history are distinct temporal problems, and LongMoE addresses the latter.

The admission-level comparison is:

MIMIC-IV full-modality modelAUC
Lee-MMGRU0.9143
Submitted LongMoE0.9150
Differentiable LongMoE0.9169
Matched Transformer0.9179
Matched GRU0.9203

We agree that mean aggregation can hide clinically relevant within-admission dynamics, and we do not claim that it is optimal. A hierarchical variant is the natural extension, with a within-admission encoder feeding LongMoE across admissions.

Q2: Routing confidence and temporal relevance are separate quantities. The router conditions on:

  • the contextual visit state
  • observation availability
  • supplied confidence values
  • whether a modality was imputed from history
  • absolute time encoding

Its score is therefore a conditional expert-routing preference & not used as the visit's temporal weight.

QuantityFunctionConfidently routed but older visit
Router score rt,kr_{t,k}Selects or weights the expert processing visit ttCan still be routed confidently to an appropriate expert
Temporal score q⊤htq^\top h_tEstimates relevance of visit tt to the final predictionMay be small even when routing confidence is high
αt=softmax⁡t ⁣(q⊤ht)\alpha_t = \operatorname{softmax}_t\!\left(q^\top h_t\right)Controls the contribution of visit tt to final logitsDownweights earlier or less relevant visits
ℓfinal=∑tαtℓt\ell_{\mathrm{final}} = \sum_t \alpha_t \ell_tProduces the subject-level predictionPrevents expert certainty from forcing a large temporal contribution

Thus, an earlier visit can be routed confidently because the router knows which expert should process its observation pattern, while receiving a small αt\alpha_t because it is less relevant to the patient’s final condition.

The expert interventions provide functional evidence:

EvidenceResult
Maximum prediction change after zeroing all active-MoE logits0.6454
ADNI prediction change after zeroing differentiable shared experts0.0417
OASIS-3 prediction change after zeroing differentiable shared experts0.1070
MIMIC-IV prediction change after zeroing differentiable shared experts0.0349

These results show that expert routing affects the prediction. They do not establish that router probabilities are calibrated clinical-reliability estimates.

Critical Weaknesses:

W1: We agree that both statements were too strong and has been updated in the uploaded revised paper. FuseMoE handles irregularly sampled and missing modalities within an encounter. What it does not do is complete a missing modality from the same patient's observations of that modality at earlier visits. The same distinction applies to the other methods. UTDE and UMSE model irregularly sampled data within a single stay, and TAMME addresses time-aware classification of longitudinal multimodal patient data and is the closest of the three to our setting. We will cite and discuss all three, together with DrFuse and DiPro.

W2: We agree that the individual components are established. New matched controls show where the benefit lies:

ControlWhat is matched to LongMoEFinding
Temporal Flex-MoE (original and capacity-matched)Patients, histories, masks, splits, seeds; its own missing-modality bank and routingLongMoE higher in 15/15 ADNI combinations under both protocols (+0.0099 and +0.0168 mean AUROC)
Matched GRU and TransformerAll of the above plus LongMoE's encoders, history-retrieval imputer, time inputs, and masksComparable to LongMoE; GRU highest on ADNI and MIMIC-IV, differentiable LongMoE highest on OASIS-3

(1) Imputation. The corrected Proposition A.4 quantifies this: carry-forward error is 2σ2(1−ρ)2\sigma^2(1-\rho), which grows as autocorrelation decays with elapsed time. For this reason the imputer receives the elapsed time, and the tokenizer receives an explicit imputed-from-history flag, so that downstream layers can discount stale evidence. The contribution is empirical and specific: with identical history, patient-specific completion outperforms a learned missing-modality bank in every ADNI combination.

(2) Temporal encoding and fusion. We agree and present these as standard components. Tables 15 and 17 show that the effect of the temporal encoding is small and dataset dependent (−0.015-0.015 on ADNI and +0.004+0.004 on OASIS-3 under full modalities).

(3) Aggregation across visits. The implemented temporal weight is αt=softmax⁡t ⁣(q⊤ht)\alpha_t = \operatorname{softmax}_t\!\left(q^\top h_t\right), computed from the contextual visit state and independent of the router score. The sentence interpreting routing scores as informativeness or reliability has been removed and Equation (4) has been updated in the uploaded revised paper (see Question 2). Regarding prediction from the final contextualised representation: the attention readout contains that predictor as the special case αT→1\alpha_T \to 1, so it can recover it when earlier visits are not useful. We have not run a last-visit readout ablation, so we make no claim that aggregation is superior to prediction from the final representation, and we acknowledge this as an open point.

W3: The listed methods fall into two groups. UTDE, UMSE, and DrFuse model a single encounter with fine-grained within-stay time series (and chest X-rays for UMSE and DrFuse). Our datasets are visit-level sequences without those inputs, so applying them requires the same non-unique temporal extension as for Flex-MoE. TAMME and DiPro are longitudinal and closer to our setting. In place of several loosely adapted baselines, we prioritised controls that answer the underlying question directly: Temporal Flex-MoE and matched GRU and Transformer models with identical history and imputation.

Thank you for your response and clarifications. While I appreciate the additional details, my main concerns about the incremental novelty (as the individual components are well-established) and the absence of recent longitudinal multimodal baselines have not been fully resolved. As such, I will maintain my original score.

Reviewer_VRhoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes LongMoE to handle missing modalities at the current visit and irregular visits in clinical multimodal data. The method combines imputation from historical observations, multi-frequency temporal encoding, a longitudinal Transformer, and context-conditioned MoE routing. Experiments on ADNI, OASIS-3, and MIMIC-IV evaluate disease staging and mortality prediction under current-visit and full-trajectory masking.

Critical strengths
  1. A practically useful framework for jointly handling longitudinal history and missing modalities. LongMoE combines imputation from historical observations, temporal encoding, and context-conditioned routing to compensate for missing current observations. This addresses common characteristics of clinical data, including incomplete examinations and irregular visit intervals.

  2. Evaluation across multiple clinical tasks and missing-modality settings. The experiments cover ADNI, OASIS-3, and MIMIC-IV, including disease staging and mortality prediction. Results across modality combinations and comparisons between current-visit and full-trajectory masking provide useful evidence for examining the role of historical information.

Critical weaknesses
  1. The experimental results are inconsistent between the main text and appendix. Under full modalities on ADNI, removing the longitudinal encoder changes AUC by −0.0335 in Table 5 but −0.001 in Table 15; replacing MoE with a single expert yields −0.0241 and +0.001, respectively. On OASIS-3, removing temporal encoding under full modalities yields −0.0256 in Table 16 but +0.004 in Table 17. These discrepancies affect the assessment of component necessity. Differences in settings should be specified, or the results corrected.

  2. The theoretical analysis does not support the claimed guarantees. Proposition A.4 conflates the optimal linear estimator ρ(1)et−1\rho(1)e_{t-1} with directly copying et−1e_{t-1} and therefore does not establish the claimed optimality of carry-forward imputation. Proposition A.11 claims that the minimum load-balancing loss is 1, but K=2K=2, f=(0.75,0.25)f=(0.75,0.25), and P=(0.4,0.6)P=(0.4,0.6) provide a feasible counterexample with a loss of 0.9. Both theoretical statements and their proofs require correction.

Questions for authors
  1. Do all methods have access to the same historical information? The longitudinally augmented baselines in Section 5 are inconsistent with the static-input description in Table 14. When LongMoE uses other modalities from past visits, can the baselines access them as well? Please specify the information available under both masking protocols and provide a GRU or Transformer control using the same history and imputation strategy to separate gains from additional information from those attributable to the architecture.

  2. How are shared experts and load balancing implemented? Please clarify how shared-expert outputs contribute to the final prediction in Equation (4). Also, please clarify whether the load-balancing term uses raw routing scores or softmax probabilities, aligning the main-text definitions with the appendix assumptions.

4 comments

We thank the reviewer for the valuable feedback. We have addressed all comments and updated the paper accordingly wherever possible.

Critical Weaknesses:

W1: The two sets of tables do not follow one evaluation protocol. Tables 5 and 16 match our logged results; Tables 15 and 17 do not. We have successfully re-run the tests using our checkpoints and updated the correct values in the uploaded revised paper. The matched controls reported under Q1 give independent evidence on component necessity, and they agree with the smaller effects in Tables 15 and 17: a Transformer with the same imputer and history but no MoE reaches 0.9741 on ADNI, against 0.9692 for hard-routed LongMoE.

W2:

Proposition A.4

The proof derived the error of the optimal single-lag linear estimator ρ(k) et−k\rho(k)\,e_{t-k} and then attributed it to the copy estimator et−1e_{t-1}. The corrected statement concerns the copy estimator directly. For a zero-mean stationary process with variance σ2\sigma^2 and autocorrelation ρ(⋅)\rho(\cdot),

E ∥et−et−k∥2=2σ2(1−ρ(k)).\mathbb{E}\,\lVert e_t - e_{t-k}\rVert^2 = 2\sigma^2\bigl(1-\rho(k)\bigr).

ConsequenceStatement
Most recent observationAmong copy estimators, lag 1 is optimal whenever ρ(1)≥ρ(k)\rho(1) \ge \rho(k) for all k>1k > 1
Comparison with mean or zero imputationCarry-forward has lower error than σ2\sigma^2 if and only if ρ(k)>1/2\rho(k) > 1/2 at the lag used
Gap to the optimal linear estimatorThe excess error over ρ(k) et−k\rho(k)\,e_{t-k} is exactly σ2(1−ρ(k))2\sigma^2\bigl(1-\rho(k)\bigr)^2

The proposition has been restated accordingly and no longer claim MMSE optimality. The second consequence also makes explicit that carry-forward becomes less reliable as elapsed time grows, which is why the imputer receives the elapsed time and an imputed-from-history indicator.

Proposition A.11

The counterexample is valid: 2 (0.75⋅0.4+0.25⋅0.6)=0.9<12\,(0.75 \cdot 0.4 + 0.25 \cdot 0.6) = 0.9 < 1. When ff and PP are treated as independent points on the simplex, the infimum is in fact 0. The AM-GM argument was invalid because it used upper bounds on ∏kfk\prod_k f_k and ∏kPk\prod_k P_k as if they were lower bounds. The bound of 1 holds only for f=Pf = P, where K∑kPk2≥1K\sum_k P_k^2 \ge 1 by the Cauchy-Schwarz inequality. We have replaced the proposition with a statement about the loss that is actually implemented (Q2):

Lbal=1K∑k=1K(pˉk−1K)2≥0,L_{\mathrm{bal}} = \frac{1}{K}\sum_{k=1}^{K}\Bigl(\bar p_k - \frac{1}{K}\Bigr)^2 \ge 0,

with equality if and only if pˉ\bar p is uniform, and with value (K−1)/K2(K-1)/K^2 under complete collapse onto one expert (≈0.109\approx 0.109 for K=8K = 8).

Questions For Authors:

Q1: Table 14 is correct: the static baselines received no history. Information available to each method:

MethodEarlier visitsCurrent-visit maskingFull-trajectory masking
LongMoEComplete naturally observed historyExcluded modalities removed only at the evaluation visit; static covariates removed globallyExcluded modalities removed at every visit
Static TF/MulT/Flex-MoE/FuseMoENone (cross-sectional extract)Requested modalities retained in that extractIdentical to current-visit
Lee-MMGRURetained-modality sequenceRetained modalities available longitudinallyExcluded modalities removed throughout the sequence
Matched GRU/Transformer controlsExactly the same history and imputer as LongMoESame masking function as LongMoESame masking function as LongMoE

As requested, we trained GRU and Transformer controls with LongMoE's subjects, splits, visit histories, modality encoders, history-retrieval imputer, time inputs, observation masks, training-time masking, optimizer, checkpoint rule and five seeds (mean AUROC). LongMoE was trained in the same control pipeline, so its values differ slightly from Table 1.

Dataset/endpointGRUTransformerHard LongMoEDifferentiable LongMoE
ADNI, natural input0.97650.97410.96920.9735
OASIS-3, natural input0.86450.85580.85450.8671
MIMIC-IV, natural input0.92030.91790.91600.9169
ADNI, current-visit single-modality mean0.96160.95760.95720.9570
OASIS-3, current-visit single-modality mean0.81900.81530.80890.8247
MIMIC-IV, current-visit single-modality mean0.88850.88620.88540.8833
ADNI, full-trajectory single-modality mean0.89800.89150.88970.8907
OASIS-3, full-trajectory single-modality mean0.74180.71600.72520.7282
MIMIC-IV, full-trajectory single-modality mean0.84480.84200.84130.8417

ADNI:

ModelNatural AUCNatural Macro-F1Current-visit AUCCurrent-visit Macro-F1Full-trajectory AUCFull-trajectory Macro-F1
LongMoE0.9746 ± 0.00370.8923 ± 0.00720.9577 ± 0.00300.8413 ± 0.00700.8864 ± 0.00820.7310 ± 0.0158
GRU + matched imputer0.9726 ± 0.00310.8725 ± 0.01040.9517 ± 0.00660.8122 ± 0.01240.8781 ± 0.00830.7058 ± 0.0066

MIMIC-IV:

ModelNatural AUCNatural Macro-F1Current-visit AUCCurrent-visit Macro-F1Full-trajectory AUCFull-trajectory Macro-F1
LongMoE0.9154 ± 0.00090.7948 ± 0.01480.8808 ± 0.00320.7285 ± 0.02480.8408 ± 0.00270.6839 ± 0.0101
GRU + matched imputer0.9205 ± 0.00080.7960 ± 0.00670.8879 ± 0.00150.7177 ± 0.01220.8452 ± 0.00110.6830 ± 0.0142

OASIS-3:

ModelNatural AUCNatural Macro-F1Current-visit AUCCurrent-visit Macro-F1Full-trajectory AUCFull-trajectory Macro-F1
LongMoE0.8664 ± 0.00330.6025 ± 0.05150.8224 ± 0.02940.5631 ± 0.04290.7242 ± 0.01340.4452 ± 0.0184
GRU + matched imputer0.8789 ± 0.01160.6177 ± 0.02920.8400 ± 0.01610.5950 ± 0.01730.7447 ± 0.00780.4703 ± 0.0181

These results separate information access from architecture. With history and imputation matched, the four models are within about 0.01 AUROC of each other in most settings, and the five-seed paired intervals overlap zero. The matched GRU has the highest mean in seven of nine rows; differentiable LongMoE has the highest mean on OASIS-3 natural input and current-visit masking. The large margins in Tables 1 to 3 therefore come mainly from longitudinal information and the history-based imputer, not from MoE routing.

A complementary control isolates the imputer: Temporal Flex-MoE receives the same history as LongMoE but uses its own learned missing-modality bank:

ADNI protocolCombinations where LongMoE is highestMean AUROC gain over capacity-matched Temporal Flex-MoE
Current-visit masking15/15+0.0099
Full-trajectory masking15/15+0.0168

Robustness therefore comes mainly from patient-specific retrieval of the missing modality's earlier observations, which outperforms a learned missing-modality bank under identical history. The added contribution of MoE routing over a matched recurrent or Transformer readout is dataset dependent.

Q2: Equation (4) summarised two separate operations, expert routing and temporal aggregation, as one.

ComponentComputationFunction
RouterScores rt,kr_{t,k} from the visit state, observation mask, confidence, history indicators, and time encodingChooses or weights experts
Hard expert pathkt=arg⁡max⁡krt,kk_t = \arg\max_k r_{t,k}Selects one shared or modality-associated expert
Per-visit predictionShared classifier applied to the selected/mixed expert outputProduces per-visit logits ℓt\ell_t
Temporal aggregationαt=softmax⁡t ⁣(q⊤ht)\alpha_t = \operatorname{softmax}_t\!\left(q^\top h_t\right)Determines visit relevance
Final predictionℓfinal=∑tαtℓt\ell_{\mathrm{final}} = \sum_t \alpha_t \ell_tAggregates expert-derived logits across valid visits

“Shared” means that the expert is eligible for every modality pattern, not that it is added unconditionally. In the implementation,

\qquad \alpha_t = \operatorname{softmax}_t\!\left(q^\top h_t\right).$$ Expert selection thus depends on router scores, and visit aggregation on the contextual visit representation. Zeroing all MoE logits in an active checkpoint changes the final probabilities by up to 0.6454 and produces a uniform prediction, confirming that the active expert path controls the output. **Load balancing** The implemented balance loss uses softmax probabilities, not raw scores. Training additionally uses a KL routing prior conditioned on modality availability and confidence: | Loss | Coefficient | | :-------------------- | :-------------------------------- | | Routing-prior KL loss | $\lambda_{\mathrm{prior}} = 0.05$ | | Load-balancing loss | $\lambda_{\mathrm{bal}} = 0.01$ | Section 4.4 of the uploaded revised paper uses this definition, and the corrected Proposition A.11 refers to it.

Differentiable shared-expert verification

The question exposed a limitation we state openly: hard argmax blocks classification-loss gradients from reaching the router, which is then trained only by the two auxiliary losses. We evaluated one controlled repair, softmax-weighted expert mixing, keeping the backbone, experts, imputer, temporal aggregation, auxiliary losses, training protocol and seeds fixed.

DatasetHard router task-gradientSoft router task-gradientSoft mass on shared expertsPrediction change after zeroing shared expertsHard$\to$soft natural AUC
ADNI0.00000.09210.34350.04170.9692→0.97350.9692 \to 0.9735
OASIS-30.00000.10910.51800.10700.8545→0.86710.8545 \to 0.8671
MIMIC-IV0.00000.02720.27910.03490.9160→0.91690.9160 \to 0.9169

Under differentiable routing the shared experts contribute functionally, which explains the favorable OASIS-3 result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.