acceptodds
Under review as a conference paper at ICLR 2027

TriHAR: Hierarchical Adaptive Routing of Representations, Modalities, and Cross-modal Interactions for Multimodal Clinical Prediction

Abstract

Conventional multimodal fusion methods typically rely on fixed encoder outputs, overlooking that informative representation depths may vary across modalities and patients. They also inadequately capture patient-specific variations in direct modality and cross-modal interaction contributions. We propose TriHAR, a hierarchical adaptive routing framework that integrates EHR, ECG, and CXR data through three routing processes. TriHAR selects an informative encoder depth within each modality and independently routes the resulting modality-specific predictions and EHR-centered cross-modal interaction predictions. Experiments on ICU length-of-stay prediction across 27 encoder combinations show that TriHAR achieves the best average rank for Accuracy, AUROC, and Precision and remains effective when either ECG or CXR is unavailable. Further analyses reveal that preferred representation depths vary across modalities and encoder architectures, while modality-expert and interaction-expert routing exhibit distinct yet complementary prediction characteristics. These results demonstrate the effectiveness of hierarchical adaptive routing across heterogeneous multimodal clinical representations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal learning, Mixture-of-experts, Representation learning, Clinical prediction, MIMIC-IV

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hmbmOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work introduces TriHAR, a multi-modal routing model for binary ICU length-of-stay prediction using EHR, ECG and CXR data. The framework designs three adaptive routing modules: R1 selects multi-layer intermediate representations from each modality encoder, R2 performs unimodal expert weighting, and R3 models EHR-centric cross-modal interactions. Predictions are fused from two complementary branches, one for unimodal evidence and one for cross-modal interactions. The main contribution is to dynamically pick appropriate encoder depths per patient-modality pair and separate unimodal and cross-modal reasoning with dual-branch MoE routing. The authors validate TriHAR on a filtered MIMIC subset with complete tri-modal inputs. They test over 27 combinations of modality encoders, compare against strong multimodal MoE baselines, and conduct mask-based missing-modality simulations. Additional analysis demonstrates complementary behavior between the two prediction branches.

Critical strengths
  1. The paper targets a practically meaningful problem in multimodal clinical prediction: existing models fix the encoder’s final-layer representation for fusion, ignoring patient- and modality-specific optimal representation depth and heterogeneous patient-level modal importance. The proposed TriHAR framework modularly separates direct modal evidence and cross-modal interaction evidence via three routing modules, which offers a conceptually clean design.
  2. The empirical evaluation is comprehensive: the authors test across all 27 combinations of 3 EHR / 3 ECG / 3 CXR encoders, alongside simulated missing-modality (bimodal) settings. They further analyze routing behaviors and demonstrate complementary predictive behaviors between the two prediction branches, providing interpretability insights beyond simple performance tables.
Critical weaknesses

1.Incomplete ablation fails to isolate the source of performance gains. The paper does not include three critical ablation controls: (1) R2+R3 without R1 (fixed top-layer encoder features); (2) concatenation of low/mid/high features as an alternative to R1’s Gumbel-Softmax selection; (3) R3 without EHR-centric cross-modal attention. Without these comparisons, we cannot verify whether the R1 adaptive layer-selection component contributes meaningfully, or whether all gains come purely from the dual MoE branches. This undermines the core technical claim of the work. 2. Limited dataset and evaluation design weaken generalization and conclusion robustness. All experiments rely on a single-site MIMIC subset filtered to only samples with complete EHR+ECG+CXR. Bimodal tests use mask-based simulated missingness rather than real clinically missing data, which has different data distribution. The binary label uses an arbitrary 3-day ICU stay cutoff with no sensitivity analysis. Moreover, evaluation heavily relies on average ranking instead of statistically verified absolute improvements, omits PR-AUC/F1, and does not compare against strong non-deep-learning baselines such as XGBoost. These issues limit confidence in the claimed advantages of TriHAR.

Questions for authors

1.Please provide ablation results for the three missing control experiments: R2+R3 without R1, multi-layer feature concatenation replacing R1, and non-EHR-centric cross-modal attention for R3. Please clearly answer: How much performance changes for each ablation case across all your 27 encoder combinations? Quantify the contribution of each component. 2.The bimodal experiments adopt mask-simulated missing-modal data on fully observed tri-modal patients. Please clearly answer: What distributional gaps exist between your mask-simulated missing data and real-world clinical missingness? Can you additionally run bimodal experiments using naturally-occurring real missing-modality patient samples from MIMIC and report the performance? If yes, how does the model performance differ compared with mask-simulated missing setting? 3. The paper uses average ranking as primary evidence. Please clearly answer: Can you report PR-AUC and F1 metrics and add comparisons against XGBoost/LightGBM baselines? 4.The routing analysis only provides aggregate statistics of expert weight and layer selection. Please clearly answer: Can you provide patient-level case examples to explain why the model selects a particular representation layer or expert for individual patients, instead of only reporting aggregated routing statistics? 5. The LOS prediction task uses a manually defined 3-day threshold for binary labels. Please clearly answer: How do your model’s quantitative results change if you alter this 3-day ICU stay cutoff? 6. The paper claims that TriHAR differs from prior multimodal MoE works such as I²MoE and FuseMoE. Please clearly answer: What is the core algorithmic distinction between your dual-branch routing design and the gating mechanism adopted by I²MoE and FuseMoE? Why cannot the gains observed in your experiments be reproduced by simply applying I²MoE/FuseMoE over multi-scale concatenated features?

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer hmbm (1/3)

Thank you for the thorough review and for the insightful comments. Your requests for additional experiments were especially important and valuable to us. We carried out all of the requested experiments and incorporated the main results into the revised paper, which strengthened its contributions and made the evaluation more rigorous. We answer the weaknesses first and then each question; our response is split into three comments (W1 and W2-1; W2-2 to W2-4; Questions).


W1-1, W1-2. R2+R3 without R1, and multi-level concatenation.

We evaluated Fixed-High (R1 removed, top-level encoder features) and Level Concat (R1 replaced by the concatenation of low, mid, and high features) on all 27 combinations:

MethodACCAUROCPRECPR-AUCRecallF1Avg. rank (ACC / AUROC / PREC)
Fixed-High0.6230.6500.6120.5310.6200.5201.87 / 2.00 / 2.07
Level Concat0.6180.6490.6020.5290.6120.5182.33 / 2.11 / 2.07
TriHAR0.6350.6560.6150.5310.6500.5381.80 / 1.89 / 1.85

(Mean over 27 combinations. TriHAR: ACC, AUROC, PREC from Table 1. Recall and F1 at t = 0.4.)

Removing R1 lowers ACC / AUROC / PREC by 0.012 / 0.006 / 0.003, and replacing it with concatenation lowers them by 0.017 / 0.007 / 0.013 while adding 0.07M parameters. The margins are modest, but R1 is the entry point of TriHAR: both R2 and R3 operate on the representations it selects. Without R1, the same R2 and R3 routing performs at the level of the strongest baselines (0.623 / 0.650 / 0.612 vs. 0.629 / 0.653 / 0.609), whereas with R1 TriHAR is the best method in all three metrics. We revised Sec. 4.2.4 to state this role of R1.

W1-3. Non-EHR-centric interaction.

We replaced the EHR anchor of R3 with a CXR anchor (CXR↔EHR, CXR↔ECG, CXR enhancement) and an ECG anchor (ECG↔EHR, ECG↔CXR, ECG enhancement). Results on the combination with the highest TriHAR performance in Table 1 (TabTrans / WaveNet / VGGNet, three runs):

R3 anchorACCAUROCPRECPR-AUCRecallF1
EHR (TriHAR)0.6500.6830.6330.5570.6610.532
CXR0.6410.6690.6320.5420.6130.546
ECG0.6220.6640.6220.5330.6950.550

The interaction branch also works with other anchors, while the EHR anchor is better in most metrics on this combination. We keep EHR as the anchor because (i) EHR is the modality most likely to be available for every ICU patient and is shared by all configurations we consider (trimodal, EHR+ECG, EHR+CXR), (ii) it matches clinical practice, where ECG and CXR findings are interpreted in the context of the patient's state, and (iii) it gives the best results on the highest-performing combination.

W2-1. Mask-based bimodal experiments.

We agree that our bimodal setting does not model real-world missingness. It is a controlled evaluation: both bimodal settings use exactly the same 7,504 trimodal patients and splits as the main experiments, and only the modality configuration changes. The excluded modality is masked and never used in training or inference, and the bimodal baselines are native two-input models. Regarding the distributional gap, patient groups defined by naturally available modality pairs differ from our trimodal cohort only moderately (first ICU stay per patient):

GroupICU LoS > 3 daysAge, mean
Naturally EHR+ECG34.9%65.0
Naturally EHR+CXR33.8%63.4
Trimodal (ours)38.9%64.5

Our cohort has a somewhat higher long-stay rate (38.9% vs. 34.9% and 33.8%), with a similar mean age. The gap is larger for patients who lack the third modality (long stay 33.5% for EHR+ECG only and 20.1% for EHR+CXR only), which is the case-mix shift that a naturally bimodal cohort would introduce. Because the trimodal cohort is close to the overall EHR+ECG and EHR+CXR populations, the controlled setting is a reasonable test of TriHAR under different modality configurations, and it isolates the effect of the input modalities from changes in case mix.

AuthorsOpenReview ↗

Author Response to Reviewer hmbm (2/3)

W2-2. PR-AUC, F1, and tree-based models.

We now report PR-AUC, recall, and F1 for all methods. Recall and F1 use the threshold maximizing validation F1 (t = 0.4; Rajkomar et al., 2018).

MethodACCAUROCPRECPR-AUCRecallF1
I²MoE0.6290.6510.6090.5300.6200.545
FuseMoE0.6260.6440.6030.5090.6250.546
DDMF0.6220.6420.6020.5210.6160.540
MulT0.6270.6530.6080.5340.6060.542
TriHAR0.6350.6560.6150.5310.6500.538

For tree-based models, we trained XGBoost and LightGBM on the same EHR variables and data splits, and compare them with TriHAR on the highest combination of Table 1 and with the average of the two models' predicted probabilities:

MethodInputACCAUROCPRECPR-AUCRecallF1
XGBoostEHR0.6570.6680.6390.5550.5540.541
LightGBMEHR0.6580.6730.6380.5650.5790.554
TriHAREHR+ECG+CXR0.6500.6830.6330.5570.6610.532
TriHAR + LightGBM (average)EHR+ECG+CXR0.6700.7060.6680.6000.6590.579

Thank you for this suggestion, which gave us a useful insight. Gradient-boosted trees are known to be strong on tabular data (Shwartz-Ziv & Armon, 2022), and they perform well on EHR alone here. However, they cannot take raw ECG waveforms or CXR images, so this is not a like-for-like comparison of fusion designs; the gap mainly reflects the tabular encoder, and TriHAR improves over its own EHR encoder on the highest, median, and lowest combinations (AUROC +0.017 to +0.114). More importantly, the two models are complementary: their predicted probabilities are only weakly correlated (r = 0.26 to 0.37), and averaging them improves over LightGBM in all metrics (last row; AUROC 0.689 to 0.706 vs. 0.673 across the three combinations), consistent with Shwartz-Ziv & Armon (2022).

References: Rajkomar et al. (2018). Scalable and accurate deep learning with electronic health records. npj Digital Medicine, 1, 18. Shwartz-Ziv & Armon (2022). Tabular data: Deep learning is not all you need. Information Fusion, 81, 84–90.

W2-3. Sensitivity to the 3-day cutoff.

The class balance changes sharply with the cutoff. We retrained TriHAR with 2- and 5-day cutoffs on the highest, median, and lowest combinations of Table 1 (three runs each):

CutoffLong stay (test set)Short : longAUROC (high / med / low)Mean AUROC
2 days56.7%0.76 : 10.646 / 0.642 / 0.6250.638
3 days38.8%1.57 : 10.683 / 0.655 / 0.6290.656
5 days22.0%3.55 : 10.672 / 0.656 / 0.6220.650

Sensitivity is moderate: AUROC stays within 0.62 to 0.68, mean AUROC changes by at most 0.018, and the ordering of combinations is unchanged, so our conclusions do not depend on the cutoff. We use 3 days not because it performs best, but because it yields a meaningful prediction target. LoS > 3 days is a standard MIMIC task (Wang et al., 2020; Susetzky et al., 2025), and the alternatives degenerate in opposite directions. At 2 days, close to the median ICU stay, long stay becomes the majority class and the model labels most patients as long stay at the default threshold (recall 0.72 to 0.85). At 5 days, it labels almost all patients as short stay (recall 0.04 to 0.39), and its accuracy equals that of always predicting short stay; higher cutoffs make this imbalance more extreme. The 3-day cutoff keeps long stay a clinically meaningful minority class with enough positives for stable evaluation.

References: Wang et al. (2020). MIMIC-Extract: A data extraction, preprocessing, and representation pipeline for MIMIC-III. ACM CHIL. Susetzky et al. (2025). A holistic time-aware classification model for multimodal longitudinal patient data. MICCAI.

W2-4. Single-site cohort and average ranks.

We agree that our study is limited to a single MIMIC cohort and a single LoS task; to our knowledge, no EHR–ECG–CXR resource independent of these MIMIC databases currently exists, as stated in our conclusion. The 27 encoder combinations therefore evaluate robustness to the encoder choice, not generalization across datasets. Beyond average ranks, we report the absolute mean of all six metrics for every method in W2-2, where TriHAR has the highest ACC, AUROC, PREC, and recall.

AuthorsOpenReview ↗

Author Response to Reviewer hmbm (3/3)

Q1. Three ablations.

We addressed this in detail in W1-1 to W1-3, with results on all 27 combinations for Fixed-High and Level Concat and on the highest-performing combination for the CXR and ECG anchors.

Q2. Mask-based vs. natural missingness.

We addressed this in detail in W2-1.

Q3. PR-AUC, F1, and XGBoost/LightGBM.

We addressed this in detail in W2-2.

Q4. Patient-level routing cases.

Each test patient follows an R1 path, defined by the levels selected for EHR, ECG, and CXR. Across all 81 models (27 combinations × 3 runs), paths with higher long-stay rates consistently show lower bicarbonate (mean r = −0.55; same direction in 93% of models), higher WBC (+0.53; 91%), higher anion gap (+0.46; 88%), lower systolic BP (−0.43; 88%), higher respiratory rate (+0.39; 82%), and higher BUN (+0.32; 78%), while sex shows no association (0.00). In a representative model (FTTrans / 1D-ResNet / ResNet18), grouping patients by the selected EHR level gives clinically distinct subgroups:

EHR levelnLong stayBicarbonateBUNCreatinineWBCSystolic BP
High64134.9%23.721.01.209.7128
Low33537.3%24.322.81.1111.5116
Mid52544.8%21.138.91.9915.8111

Comparing two patients from this model makes the pattern concrete. They were chosen by a predefined rule: the correctly classified patient closest to the median clinical profile of the lowest-risk and the highest-risk path.

Patient APatient B
OutcomeShort stayLong stay
R1 path (EHR / ECG / CXR)High / Mid / MidMid / Low / High
Long-stay rate of the path27.0%48.6%
R2 / R3 / final probability0.26 / 0.38 / 0.320.73 / 0.41 / 0.57
Age50s50s
Resp. rate, anion gap, BUNNormalElevated

The two patients are of similar age, but Patient B shows tachypnea, an elevated anion gap, and elevated BUN, the same acuity markers that separate high- and low-risk paths across models. R1 routes the two patients through different depths in all three modalities, and their paths belong to subgroups with different long-stay rates (27.0% vs. 48.6%). Patient B also shows how the branches complement each other: R3 underestimates the risk (0.41), R2 assigns a high probability (0.73), and the fused prediction (0.57) is correct. Which level corresponds to higher acuity is learned per model, so we interpret these findings descriptively rather than causally.

Q5. Changing the 3-day cutoff.

We addressed this in detail in W2-3.

Q6. Difference from I²MoE and FuseMoE.

Both apply a single stage of expert gating on a fixed encoder representation. TriHAR (i) first selects a representation depth per modality and patient (R1), and (ii) separates modality-specific evidence (R2) from EHR-anchored interaction evidence (R3) in parallel branches. As shown in W1-2, Level Concat, which feeds multi-scale concatenated features into the same routing, does not improve over TriHAR, so the gains are not explained by richer multi-scale features alone.

References: Xin et al. (2025). I²MoE: Interpretable multimodal interaction-aware mixture-of-experts. ICML. Han et al. (2024). FuseMoE: Mixture-of-experts transformers for fleximodal fusion. NeurIPS.


We sincerely thank the reviewer for the thorough and constructive review. The requested controls, metrics, and analyses substantially strengthened the paper, and we hope our responses address your concerns. We would be glad to answer any further questions.

Reviewer_PLzVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TriHAR, a hierarchical adaptive routing framework for multimodal clinical prediction with EHR, ECG, and CXR data. The contributions are (1) Representation-Level Routing, which selects one of three encoder depths per modality and patient through Straight-Through Gumbel-Softmax, (2) Modality-Expert Routing, which adaptively combines modality-specific predictions, and (3) Interaction-Expert Routing, which combines EHR-centered cross-modal interaction predictions, with the two branches fused for the final prediction. Experiments on ICU length-of-stay prediction from MIMIC-IV, MIMIC-IV-ECG, and MIMIC-CXR-JPG cover 27 encoder combinations against four trimodal baselines, plus EHR–ECG and EHR–CXR settings against four bimodal baselines. TriHAR achieves the best average rank in Accuracy, AUROC, and Precision in both settings, and routing analyses show depth preferences that vary across modalities and encoders and complementary behavior between the two expert routers.

Critical strengths
  • The experiments are comprehensive, including 27 combinations of EHR, ECG, and CXR encoders, comparisons against four trimodal and four bimodal baselines, and analyses of representation-level selection and expert utilization.

  • The paper is easy to follow, and the method figure is clear.

Critical weaknesses

The evaluation metrics do not show how many long-stay patients are missed. The paper reports accuracy, AUROC, and precision, but no recall or F1. In Figure 3, the full model has more false negatives (23.9%) than true positives (15.0%) on average across encoder combinations, so it misses more long-stay patients than it identifies, which precision alone does not show.

Questions for authors

Can you clarify what "three valid seeds" means in the captions of Figures 5 to 7.

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer PLzV

Thank you for the positive assessment and the helpful suggestion on evaluation metrics.

W1. Recall and F1.

We now report recall and F1 together with the original metrics. ACC, AUROC, and PREC are the Table 1 values (threshold 0.5). Recall and F1 use the threshold that maximizes F1 on the validation set (t = 0.4), applied unchanged to the test set, following the standard practice of tuning the decision threshold for F1 on held-out data (Rajkomar et al., 2018). PR-AUC is threshold-free.

MethodACCAUROCPRECPR-AUCRecallF1
I²MoE0.6290.6510.6090.5300.6200.545
FuseMoE0.6260.6440.6030.5090.6250.546
DDMF0.6220.6420.6020.5210.6160.540
MulT0.6270.6530.6080.5340.6060.542
TriHAR0.6350.6560.6150.5310.6500.538

(Mean over 27 encoder combinations and three runs.)

TriHAR has the highest ACC, AUROC, PREC, and recall, while PR-AUC and F1 are on par with the baselines (F1 within 0.01). The false negatives in Fig. 3 therefore mainly reflect the default threshold of 0.5: at a validation-selected operating point, TriHAR identifies more long-stay patients than the baselines. The revised Appendix also reports these metrics for the new analyses.

Reference: Rajkomar et al. (2018). Scalable and accurate deep learning with electronic health records. npj Digital Medicine, 1, 18.


Q1. "Three valid seeds."

These are three independent runs with different data splits and initializations; Figs. 5 to 7 average over them, and the same three runs are used for all methods. We revised the captions to "averaged over three independent runs".

We thank the reviewer again for the positive evaluation and the helpful suggestions, which helped us clarify the contributions of the paper. We would be glad to answer any further questions.

Thanks for your rebuttal. I will maintain my original positive score.

AuthorsOpenReview ↗

Thank You for Your Response

Thank you for your response and for maintaining your positive assessment. Your constructive suggestions helped us improve the paper. Please feel free to let us know if you have any further questions.

Reviewer_AQVvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper describes a three-level herarchical attention system for multimodal electronic health records. The first level has hard attention over which layer of the modality encoder representations to use. The second level performs a weighted sum over the different modalities. The third performs cross-attention in different directions across modality pairs. Both levels 2 and 3 make their own independent predictions which are then combined into a join prediction and trained end-to-end.

Critical strengths

The architecture is interesting and seems sensible, blending together 3 different modalities. Evaluation is performed across many different encoder combinations, which can give useful information also beyond this specific architecture. There is a good number of baselines compared against.

Critical weaknesses

Given that modality attention and cross-attention are pretty well established, then one of the main novel aspects in this model is attending to different layers of the encoder. This component is also argued to be crucially important. However, the contribution of this component is never investigated or evaluated in isolation. For some reason, the ablation experiments ablate the higher levels of attention but don't include an experiment that ablates the layer attention, using a fixed encoder representation instead (e.g. from the top level of the encoder).

Evaluation is only performed on 1 binary classification task and on one dataset. It is difficult to draw conclusions that this method generalises beyond that specific case. Yes, it may be difficult to obtain other datasets that contain EHR+CXR+ECG, but the architecture is in no way specific to those modalities. Datasets with other modality combinations could be used. And if nothing else, MIMIC itself can be used for many different tasks beyond the one constructed in this paper.

Experiments in Section 4.2.3 are used as evidence to show that the model focuses on different layers and different modalities. But the model has been explicitly regularised to divide its attention as evenly as possible (Eq 11). The benefit of this regularisation should be evaluated. If without this regularisation the model ends up collapsing to a single layer and single modality while achieving better results, then that would invalidate a lot of the conclusions in the paper.

There should be a compute requirements comparison. How does the number of parameters and number of operations (or wall clock time) compare to the baselines? If the proposed model is bigger and requires substantially more compute then the comparison might not be fair.

In equation 2 there needs to be much more detail about r and z come from. What are the logits in the case of encoders? These encoders have been presumably trained for different tasks. And r is not discussed at all, even though that is the information based on which the attention decisions are made. Where does it come from and what does it contain?

There are several components of the paper that are left vague. R_L in eq 2 "adjusts the sequence length" but doesn't specify how. g_{m,k} is a "scoring function" without any details, F_{fuse} in eq 10 just says it's a "learnable fusion function".

Many citations in the paper are using latex citet when they should be using citep. Please format them correctly for readability.


Update after author response:

Thank you for providing some ablation experiments. This did improve my opinion of the paper but not evaluating on any other tasks or datasets is still and important issue and I can't raise my score to recommend acceptance.

It seems the justification for not reporting on other tasks and datasets is that the method would not work on other tasks and datasets. These negative results should then be reported, not experiments omitted based on results.

When evaluating only on one task and dataset, the method could easily be overfitted to that particular dataset. Some generalisation beyond one particular dataset is needed to demonstrate that the method actually works beyond one particular test set.

Re hard selection: concatenation is not the same as soft attention. I don't think you can use that to draw conclusions about how soft-attention might perform over layers. Besides you ran that experiment for the author response so something else must have motivated the original design decision.

Questions for authors

Hard attention is performed over the layers but soft attention (weighted sum) is performed at the other levels. This choice doesn't seem to have been discussed. Did you consider soft attention also on the layer level and what was the result?

Were all these baselines reimplemented from scratch? If so, how can we be sure they are correct and working as designed?

6 comments
AuthorsOpenReview ↗

Author Response to Reviewer AQVv (1/2)

Thank you for your valuable review and suggestions. We conducted all the requested experiments and incorporated the key results into the revised manuscript. These additions strengthened both the paper and its evaluation.


W1. Contribution of R1 in isolation.

We added two ablations on all 27 combinations: Fixed-High (R1 removed, top-level features) and Level Concat (R1 replaced by the concatenation of all levels).

MethodACCAUROCPRECPR-AUCRecallF1
Fixed-High0.6230.6500.6120.5310.6200.520
Level Concat0.6180.6490.6020.5290.6120.518
TriHAR0.6350.6560.6150.5310.6500.538

(Mean over 27 combinations. TriHAR: ACC, AUROC, PREC from Table 1. Recall and F1 at t = 0.4.)

TriHAR has the best average rank in ACC, AUROC, and PREC (1.80 / 1.89 / 1.85 vs. 1.87 / 2.00 / 2.07 for Fixed-High and 2.33 / 2.11 / 2.07 for Level Concat). The margins are modest, but they clarify the role of R1. R1 is the entry point of TriHAR: both R2 and R3 operate on the representations it selects. Without R1, the same R2 and R3 routing performs at the level of the strongest baselines (0.623 / 0.650 / 0.612 vs. 0.629 / 0.653 / 0.609), whereas with R1 TriHAR is the best method in all three metrics. Level Concat adds 0.07M parameters and performs worse, so using all depths jointly does not help. We revised Sec. 4.2.4 to state this role of R1.

W2. Single task and single dataset.

We agree that our evaluation is limited to a single task (ICU LoS) within on a single MIMIC cohort, and we do not claim generalization to other datasets or tasks. We chose ICU LoS because the other common MIMIC tasks are considerably more imbalanced. This is a property of the tasks, not of our trimodal selection: the positive rates are similar among all patients with EHR+ECG or with EHR+CXR available (first ICU stay per patient):

TaskTrimodal (ours, n = 7,504)Naturally EHR+ECGNaturally EHR+CXR
ICU LoS > 3 days (ours)38.9%34.9%33.8%
In-hospital mortality11.7%9.9%11.7%
30-day readmission22.2%17.4%20.3%

With fewer positives, comparisons across 27 encoder combinations and 8 baselines become less stable, as we also observe for LoS with a 5-day cutoff (22% positives). Within this setting, our experiments test robustness rather than dataset generalization:

  • Encoders: the 27 combinations of 9 encoders from different families (tabular transformers, temporal CNN/RNN, 2D CNNs) evaluate robustness to the choice of representation, not generalization across datasets.
  • Modality configurations: in EHR+ECG and EHR+CXR, TriHAR has the best average rank among four bimodal methods (ACC / AUROC / PREC 2.028 / 2.056 / 2.222 vs. 2.861 / 2.889 / 2.778 for the best baseline in each metric, Table 2).
  • Interaction anchor: R3 also works with CXR or ECG as the anchor (AUROC 0.664 to 0.669 on the highest combination), so the interaction branch is not tied to EHR.
  • Label definition: with 2- and 5-day cutoffs, AUROC stays within 0.62 to 0.68.

To our knowledge, no EHR–ECG–CXR resource independent of these MIMIC databases currently exists, and evaluation on other tasks and datasets remains important future work.

W3. Effect of the regularizer (Eq. 11).

The KL term is applied only to the R2/R3 gates, not to R1, so the level-selection results in Sec. 4.2.3 are not caused by it. Its purpose is not to improve performance but to guard against a known failure mode of soft MoE gates, collapse onto a single expert (Shazeer et al., 2017). We retrained TriHAR without it on all 27 combinations with three runs each:

MethodACCAUROCPRECPR-AUCRecallF1Runs with collapsed R2 gate
TriHAR (w/o reg)0.6310.6540.6110.5310.5780.5248.6%
TriHAR (ours)0.6350.6560.6150.5310.6500.5380%

(Collapsed: for over half of the test patients, one modality receives over 90% of the R2 gate weight. Recall and F1 at t = 0.4 for both.)

Collapse is rare even without the regularizer (8.6% of runs, always onto CXR), and AUROC and PR-AUC are nearly unchanged. The regularizer therefore serves mainly as a safeguard that removes these cases (0%), although recall and F1 are also higher with it. Recomputing Table 4 without the regularizer (Appendix D.8):

RoutingModelEHR (%)ECG (%)CXR (%)
R1+R2TriHAR (ours)22.718.558.8
R1+R2TriHAR (w/o reg)22.013.464.6
R1+R3TriHAR (ours)39.430.430.1
R1+R3TriHAR (w/o reg)32.723.444.0

R1 still uses all levels for every modality, so level selection does not depend on the regularizer. Without it, R2 shifts further toward CXR and the leading R3 expert changes from the enhanced EHR to the EHR-conditioned CXR expert, although R3 stays far more balanced than R2. We revised Sec. 4.2.3 to state the R3 emphasis as a property of the regularized model.

Reference: Shazeer et al. (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. ICLR.

AuthorsOpenReview ↗

Author Response to Reviewer AQVv (2/2)

W4. Compute.

Averaged over 27 combinations (parameters receiving gradients; FLOPs at batch size 1). Mean AUROC is from Table 1.

MethodParams (M)Fusion params (M)GFLOPsFusion GFLOPsMean AUROC
I²MoE12.410.5620.680.4820.651
FuseMoE12.981.1320.240.0420.644
DDMF12.440.5820.240.0420.642
MulT13.251.3920.390.1960.653
TriHAR13.141.2220.350.1550.656

The differences in capacity are small (parameters within 7%, total FLOPs within 2%) and do not track performance. MulT has the most parameters and I²MoE the most FLOPs, yet both rank below TriHAR, and FuseMoE has a fusion module of similar size to TriHAR's (1.13M vs. 1.22M) but the second-lowest AUROC. Within TriHAR, Level Concat adds 0.07M parameters without any gain (W1). The improvements of TriHAR are therefore not explained by additional parameters or computation.

W5 / W6. Definitions.

zm,kz_{m,k}: logits of a level-specific prediction head; rm,kr_{m,k}: pooled feature of level kk, used by R1 to score levels. All encoders and heads are trained end-to-end on the LoS task without pretrained weights. RLR_L: adaptive average pooling to a fixed length LL. gm,kg_{m,k}: level-specific linear scoring layer whose logits feed the Straight-Through Gumbel-Softmax. FfuseF_{fuse}: a small MLP over the concatenated logits of R2 and R3. These definitions are now in Sec. 3.

W7. Citation format.

We agree with the reviewer. To make the text easier to read, we replaced \citet with \citep wherever a citation is parenthetical throughout the manuscript.


Q1. Hard selection at the level stage.

As shown in W1, using all levels jointly does not help. Level Concat, which combines all three levels instead of selecting one, gives lower ACC, AUROC, and PREC than hard selection (0.618 / 0.649 / 0.602 vs. 0.635 / 0.656 / 0.615) while adding 0.07M parameters (W1). Hard selection with the Straight-Through Gumbel-Softmax (Jang et al., 2017) also makes the depth used for each patient explicit, which enables the analyses in Sec. 4.2.3. In R2 and R3, by contrast, the experts carry complementary evidence from different modalities, so soft weighting is natural there. As the soft, non-selective alternative at the level stage, we evaluated Level Concat in place of a softmax-weighted sum over levels.

Reference: Jang, Gu & Poole (2017). Categorical reparameterization with Gumbel-Softmax. ICLR.

Q2. Baselines.

They were not reimplemented. For I²MoE, FuseMoE, DDMF, and MulT we import the fusion, routing, and loss modules from the official repositories unchanged and only connect them to the shared encoders. No pretrained weights are used, and all methods share the same 27 encoder combinations, so the comparison reflects differences in fusion and routing design.


We thank the reviewer again for the constructive comments, which helped us clarify the role of each routing component and strengthen the evaluation. We hope our responses address your concerns and would be glad to answer any further questions.

AuthorsOpenReview ↗

Reminder: Our Response to Your Review

Dear Reviewer AQVv,

Thank you for your thoughtful review and constructive feedback. Your comments have been valuable in helping us strengthen our paper.

As the author–reviewer discussion period is coming to a close, we would like to kindly follow up on our rebuttal. We have addressed all of your comments and questions, including conducting the additional experiments you suggested. We would greatly appreciate it if you could review our response before the discussion period ends. We would also be happy to provide further clarification if needed.

Thank you again for your time and consideration.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their constructive feedback. We uploaded a revised PDF with changes marked in blue. Main additions:

  1. R1 ablations (AQVv W1, hmbm W1): Fixed-High and Level Concat on all 27 encoder combinations (Appendix D.7).
  2. Effect of the gate regularizer (AQVv W3; Appendix D.8).
  3. Compute comparison: parameters and FLOPs (AQVv W4; Appendix D.9).
  4. Cutoff sensitivity at 2 and 5 days (hmbm W2; Appendix D.10).
  5. Patient-level routing analysis with clinical variables (hmbm Q4, in our response).
  6. PR-AUC, recall, and F1 at a validation-selected threshold for the new analyses (PLzV W1, hmbm W2; Appendix C, D.7–D.10).
  7. Clarified definitions (rr, zz, RLR_L, gg, FfuseF_{fuse}), the scope of the KL regularizer, figure captions, and citation format.

We sincerely thank all reviewers again for their time and valuable comments, which have substantially improved the paper. We would be glad to answer any further questions.

AuthorsOpenReview ↗

Thank You to All Reviewers and the Area Chair

Dear Reviewers and Area Chair,

We sincerely thank all reviewers and the Area Chair for their time and constructive feedback throughout the review and discussion. We are also grateful that the reviewers recognized the strengths of our work, including the design of the routing framework and the comprehensive evaluation across encoder combinations. During the author–reviewer discussion, we carried out all of the additional experiments requested by the reviewers and incorporated the main results into the revised paper. These additions clarified the role of each component, strengthened the evaluation, and substantially improved the paper.

Thank you very much for taking the time to review our paper despite your busy schedules.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.