- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces MILM, which classifies multimodal irregular time series. it first searializes every observation as an tine-ordered XML time–channel–value triplet, and fine-tunes Qwen3-4B with LoRA. Stage 1 of the process trains on value-redacted sequences so the model learns from sampling patterns. Stage 2 continues on full sequences. Predictions are read from logits scores after an <answer> tag in the output. Experiments are conducted on MIMIC-IV and eICU for in-hospital mortality and length-of-stay prediction. MILM obtains an average rank of 1.1 against baselines on 4 EHR datasets.. Several ablations are done. Value-redaction and a new "value pending" evaluation are also reported in the paper, as well as a Yelp restaurant-closure experiment in the appendix showing transferability of the approach to another domain.
Critical strengths
Simple method with well-designed control and useful eval. MILM-Direct-Eq and the channel- vs time-ordered ablation are the right methods to support the claims. Good reporting practices are employed: 5 runs with SDs, hyperparameter search based on validation AUROC, wall-clock and memory are reported for all methods along with prompts, code release. Data was used under DUA with local compute only.
Critical weaknesses
There is only weak evidence that the proposed two-stage "informative sampling" strategy is the key point here. Most of the advantage over baselines may come from LLM serialization itself. MILM-Direct already beats the best baseline on 6 of 8 metrics and MILM improves over MILM-Direct by only 0.17–1.46 points.
cohort construction and splits should be revisited and limit results generalizability. Splits are random at ICU-stay level and not by patient, however, MIMIC and eICU contain multiple stays per patient, so patient-level leakage is possible. It is not stated whether the five runs vary the split or only the seed. eICU spans 100s of hospitals, a random split lets a model learn site-specific practices. The conclusion lists cross-hospital shift only as future work.
The cohorts are heavily filtered.
Questions for authors
Are splits patient-disjoint? If not, please re-run all experiments with patient-grouped splits (and, for eICU, a hospital-held-out split).
Were eICU note types audited for treatment-limitation content?
Would appreciate Table 1 to also be reported under realistic value-pending setting for all methods (not only MILM vs MILM-Direct).
4 comments
We also re-ran the evaluation on larger test sets built from patients who appear in neither training nor validation using relaxed cohort filters. Each set is three times the size of the original test set. For eICU, we additionally built hospital-held-out test sets that exclude every hospital present in training or validation.
Table R3a. AU-ROC on the new test sets. Best results in bold, second-best in italics.
| Method | MIMIC-IV-IHM unseen patient | MIMIC-IV-LOS unseen patient | eICU-IHM unseen patient | eICU-LOS unseen patient | eICU-IHM unseen hospital | eICU-LOS unseen hospital |
|---|---|---|---|---|---|---|
| GRU-D | 76.52 ± 0.33 | 64.35 ± 0.32 | 65.69 ± 0.86 | 56.09 ± 1.75 | 66.65 ± 1.12 | 56.81 ± 1.79 |
| GRU-D + Note | 76.51 ± 0.81 | 64.87 ± 1.55 | 67.48 ± 0.88 | 56.41 ± 0.74 | 67.57 ± 1.04 | 54.58 ± 2.76 |
| mTAND | 76.23 ± 1.30 | 63.53 ± 0.69 | 64.41 ± 0.26 | 58.65 ± 1.51 | 68.62 ± 1.17 | 58.61 ± 0.78 |
| mTAND + Note | 78.29 ± 0.81 | 64.03 ± 0.76 | 65.26 ± 0.53 | 59.19 ± 0.59 | 68.36 ± 0.96 | 55.96 ± 1.46 |
| KEDGN | 78.34 ± 0.58 | 64.04 ± 1.92 | 70.12 ± 0.66 | 59.54 ± 1.47 | 72.31 ± 1.65 | 59.80 ± 1.59 |
| KEDGN + Note | 74.32 ± 2.65 | 56.97 ± 4.48 | 70.01 ± 1.00 | 60.20 ± 0.63 | 73.49 ± 0.59 | 60.13 ± 1.44 |
| UTDE | 77.60 ± 2.85 | 67.55 ± 1.88 | 67.03 ± 0.66 | 60.31 ± 0.83 | 67.69 ± 0.82 | 55.93 ± 1.24 |
| FuseMoE | 80.11 ± 1.52 | 68.24 ± 0.70 | 67.85 ± 0.50 | 60.69 ± 0.30 | 67.99 ± 0.80 | 56.49 ± 1.48 |
| VITAL-stats | 78.24 ± 0.75 | 65.47 ± 0.82 | 68.61 ± 0.82 | 59.11 ± 1.20 | 70.87 ± 1.50 | 58.67 ± 0.37 |
| VITAL-stats + Note | 78.77 ± 0.51 | 65.44 ± 1.29 | 68.87 ± 1.17 | 59.64 ± 1.40 | 70.55 ± 0.76 | 59.31 ± 1.00 |
| VITAL-LLM | 71.42 ± 1.00 | 59.31 ± 1.18 | 57.77 ± 0.49 | 58.95 ± 1.21 | 57.82 ± 0.83 | 56.02 ± 1.06 |
| VITAL-LLM + Note | 70.83 ± 0.89 | 60.53 ± 1.06 | 58.16 ± 0.72 | 59.64 ± 1.03 | 56.76 ± 0.56 | 54.06 ± 0.73 |
| ISTS-PLM | 78.78 ± 0.60 | 65.83 ± 0.91 | 68.84 ± 0.13 | 60.40 ± 0.48 | 73.12 ± 1.53 | 61.91 ± 0.85 |
| ISTS-PLM + Note | 78.87 ± 0.52 | 65.73 ± 0.57 | 69.40 ± 0.52 | 60.44 ± 0.78 | 72.81 ± 0.80 | 61.81 ± 0.41 |
| Qwen3-4B (zero-shot) | 75.67 | 62.05 | 64.87 | 53.12 | 70.56 | 59.17 |
| MILM | 84.20 ± 0.28 | 70.36 ± 0.43 | 71.63 ± 0.51 | 62.69 ± 0.66 | 75.57 ± 0.78 | 60.66 ± 0.79 |
Table R3b. AP on the new test sets. Best results in bold, second-best in italics.
| Method | MIMIC-IV-IHM unseen patient | MIMIC-IV-LOS unseen patient | eICU-IHM unseen patient | eICU-LOS unseen patient | eICU-IHM unseen hospital | eICU-LOS unseen hospital |
|---|---|---|---|---|---|---|
| GRU-D | 42.86 ± 0.96 | 50.61 ± 0.91 | 24.54 ± 1.05 | 51.49 ± 0.83 | 22.13 ± 1.59 | 59.14 ± 1.24 |
| GRU-D + Note | 40.82 ± 3.40 | 50.74 ± 2.55 | 26.15 ± 1.05 | 50.98 ± 0.57 | 22.07 ± 1.12 | 57.28 ± 2.70 |
| mTAND | 45.87 ± 1.36 | 49.30 ± 0.82 | 23.06 ± 0.85 | 53.59 ± 1.04 | 20.30 ± 1.50 | 60.85 ± 0.42 |
| mTAND + Note | 46.73 ± 1.57 | 49.03 ± 1.28 | 23.35 ± 0.87 | 52.65 ± 1.00 | 20.01 ± 1.08 | 58.41 ± 1.26 |
| KEDGN | 43.87 ± 1.96 | 48.37 ± 2.49 | 30.44 ± 1.31 | 52.58 ± 1.32 | 24.21 ± 3.02 | 61.50 ± 1.29 |
| KEDGN + Note | 36.09 ± 5.29 | 42.94 ± 3.60 | 30.37 ± 1.77 | 53.97 ± 0.87 | 25.44 ± 1.40 | 61.72 ± 1.51 |
| UTDE | 42.15 ± 6.80 | 53.57 ± 2.25 | 24.65 ± 0.42 | 53.39 ± 1.43 | 19.70 ± 0.59 | 58.35 ± 1.25 |
| FuseMoE | 47.04 ± 3.87 | 54.39 ± 0.48 | 25.67 ± 0.92 | 54.17 ± 0.86 | 18.31 ± 1.54 | 58.58 ± 1.86 |
| VITAL-stats | 45.25 ± 1.77 | 52.53 ± 0.87 | 26.98 ± 1.09 | 52.41 ± 1.25 | 21.48 ± 2.78 | 60.19 ± 0.70 |
| VITAL-stats + Note | 47.55 ± 0.78 | 51.45 ± 1.20 | 27.79 ± 1.03 | 53.47 ± 1.74 | 22.63 ± 1.42 | 60.71 ± 0.97 |
| VITAL-LLM | 28.44 ± 0.99 | 44.75 ± 1.90 | 20.18 ± 0.50 | 52.94 ± 1.71 | 16.17 ± 2.11 | 58.42 ± 1.47 |
| VITAL-LLM + Note | 27.09 ± 1.00 | 45.53 ± 1.32 | 21.19 ± 1.15 | 53.32 ± 2.00 | 16.59 ± 0.76 | 56.45 ± 1.34 |
| ISTS-PLM | 46.25 ± 1.25 | 51.94 ± 0.50 | 28.34 ± 0.93 | 53.64 ± 0.61 | 25.62 ± 0.87 | 63.01 ± 0.75 |
| ISTS-PLM + Note | 47.17 ± 0.63 | 52.17 ± 1.06 | 28.45 ± 0.57 | 53.58 ± 0.87 | 24.96 ± 0.84 | 62.61 ± 0.61 |
| Qwen3-4B (zero-shot) | 38.05 | 48.21 | 22.78 | 48.72 | 17.94 | 60.61 |
| MILM | 53.17 ± 0.66 | 57.47 ± 1.03 | 32.50 ± 0.95 | 55.94 ± 0.82 | 30.00 ± 0.76 | 61.56 ± 0.70 |
MILM ranks first in both metrics on 5 of the 6 new test sets.
The five runs use the same split and differ in the training seed. All methods share this split and these seeds, so the comparisons are matched. We have updated Section 5.1 of the revised paper to state this more explicitly.
Q2: Treatment-limitation content in eICU notes
Our eICU extraction uses only the patient, lab, and note tables. Code status and end-of-life care plans are recorded in the carePlanGeneral and carePlanEOL tables, which no method in our study reads. The sentence in Appendix B.1 that describes the care plan as covering end-of-life planning refers to these unused tables, and we have corrected it in the revised paper. We also audited the notes we use with a keyword search. A line noting an end-of-life or care-limitation discussion appears in 5 of the 551 eICU-IHM test stays and 2 of the 408 eICU-LOS test stays. Excluding these stays changes AU-ROC by less than 0.7 points for every method. The ranking of all methods is unchanged on eICU-LOS, and the top four are unchanged on eICU-IHM.
Q3: Value pending results for all methods
Appendix E (Table 15) reports all baselines under drop observation. Show presence requires representing an observation without its value, which these baselines cannot do because they need a numeric value for every observed entry. In the revised paper, Section 5.5 now refers to Table 15 by name for better visibility.
We have added Tables R1 to R3 to the appendix in the revised paper.
We thank the reviewer again for suggesting the patient- and hospital-held-out evaluations, which strengthened the paper. We hope these results and clarifications address the reviewer's questions.
Thank you for the additional analysis. The patient-disjoint evaluation addresses my concern about leakage. The eICU note audit and Table 15 largely address my other questions. I remain confident in my rating.
We thank the reviewer for the detailed review and for recognizing our experiment design. We address each point below.
W1: Contribution of Stage 1
Time-ordered serialization is itself a component of MILM alongside two-stage fine-tuning. MILM-Direct is our ablation that removes Stage 1. The question is therefore whether Stage 1 adds a reliable gain on top of the serialization. To test this, we ran two paired analyses that pool all 20 dataset-seed pairs: a paired t-test over seeds and a paired bootstrap over test stays (10,000 resamples).
Table R1. MILM minus the comparison method in points, with 95% intervals.
The AU-ROC gain is significant under both analyses, including against the epoch-matched MILM-Direct-Eq. Although its intervals are wider, MILM's mean AP is also higher in both comparisons. Stage 1 also helps when some values are not yet available at prediction time (Table 4). MILM keeps its lead over MILM-Direct when pending observations are dropped, and the lead widens in all four dataset-metric combinations when their time and channel are shown.
Q1 & W2: Patient-level splits, the five runs, and hospitals
The splits are random at the ICU-stay level, so some test stays belong to patients who also have stays in training or validation. These account for 9.0%, 6.7%, 9.8%, and 9.1% of the test stays on MIMIC-IV-IHM, MIMIC-IV-LOS, eICU-IHM, and eICU-LOS. We re-ran the evaluation of all methods after removing these stays, so that every test patient is unseen during training.
Table R2a. AU-ROC on the patient-disjoint test sets. Best results in bold, second-best in italics.
Table R2b. AP on the patient-disjoint test sets. Best results in bold, second-best in italics.
MILM remains first in 6 of the 8 columns and second in the other 2.