acceptodds
Under review as a conference paper at ICLR 2027

Multimodal Irregular Time-series Analysis on Clinical Databases

Abstract

Clinical datasets are very useful to drive patient outcome prediction. State-of-the-art approaches typically aggregate patient-wise time-series from such databases and build analysis models. However clinical time-series analysis poses unique challenges due to their irregular inputs under interleaved multi-modalites. Current methods are either unimodal, or fail to capture cross-modal interactions across the interleaved modalities. To bridge this gap, we propose to achieve seamless cross-modal interaction, that is, allowing bidirectional information flow across interleaved multimodal inputs. Our model, MIST (Multimodal Irregular Seamless Transformer), serves as a novel clinical time-series model that supports seamless cross-modal interaction under a unified token-stream representation. We evaluate MIST across four tasks on two real-world databases, showing that MIST outperforms state-of-the-art baselines, with 8.8% and 5.4% relative AUROC gains on adverse-outcome prediction, a 4.8% AUROC gain on MIMIC phenotyping, 12.3% and 18.5% MSE reductions on length-of-stay, and 0.7% and 0.8% MSE reductions on vital-sign forecasting averaged over all horizons, rising above 10% at short horizons. We further demonstrate the multimodal gains through component ablations and case studies.

Primary area:
learning on time series and dynamical systems
Keywords:
Time-series Analysis, Clinical Databases, Multimodal, Irregular Time-series

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PPHzOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

MIST combines irregular numerical measurements and short timestamped chart events in a unified token stream. ClinicalBERT supplies type and text embeddings, a scalar principal-component projection represents textual values in the triplet stream, and gated cross-attention reinjects full text content alongside self-attention. A separate vital-sign encoder supplies additional physiological features. Experiments cover adverse outcomes, log length of stay, vital forecasting, and MIMIC phenotyping across adult and pediatric cohorts. The reported gains are substantial for several patient-level targets and smaller for aggregate forecasting. The study supports the usefulness of short clinical events, but the specific role of temporal interleaving needs stronger evidence.

Critical strengths
  1. The method addresses a useful clinical data type. Short categorical chart events are explicitly distinguished from long clinical documents, and the model accepts numerical and textual records on an irregular timeline. The design offers a practical way to combine these events with an established physiological encoder rather than requiring aligned modalities.

  2. The empirical package is broader than a single-task demonstration. On MIMIC, mortality AUROC increases from 0.725 to 0.789 and log-LOS MSE falls from 0.470 to 0.412. The study includes a pediatric cohort, alternative vital encoders, and a forecasting breakdown. Appendix B.4 matches IMM-TSF's ClinicalBERT and GraFITi backbones to MIST, which strengthens the comparison. These are separate within-cohort experiments, not an external transfer evaluation.

Critical weaknesses
  1. The evidence does not isolate the central interleaving claim. Table 4 describes replacing seamless fusion with naive concatenation, whereas Section 5.3 and Appendix B.6 describe two streams followed by gated cross-attention. These are different controls. Even under the appendix description, the intervention changes stream organization and the placement of cross-modal interactions together. The .092 mortality AUROC drop therefore cannot be attributed specifically to interleaving or bidirectionality. Please identify the actual comparator, report its capacity and seed-wise results, and separate joint-stream attention from repeated semantic injection and late fusion. A control perturbing relative event timing while retaining content would directly address the motivating claim. The full-text timing mechanism also needs clarification. The triplet tokens carry time, but Equation 2 describes C as content embeddings without an explicit time term. Ordering key/value rows alone does not preserve their event-time binding under attention. The scalar PCA value may supply a content cue, so this is not an absence of all temporal information. Please state whether C contains time or type information and how the full semantics are associated with their original events (Sections 4.1 to 4.2, Table 4, Appendix B.6).

  2. The outcome definitions and cohort analysis need a more precise clinical interpretation. Section 5.1 uses the ICU expire flag, whereas Table 1 and Appendix C.2 call the target in-hospital mortality. The exact endpoint must be identified. For CHoA, positive labels include transfer to ICU, but it is unclear whether transfers before the 24-hour prediction landmark are excluded. Truncating inputs alone does not make an already observed transfer a future outcome. In addition, both cohorts require stays longer than 48 hours, and MIMIC eligibility depends on observations between hours 24 and 48. These criteria define a selected population and limit interpretation as prediction for general admissions. Table 7 does not establish absence of meaningful selection bias merely by comparing a few distributions. Appendix C.4 is a useful attempt to examine excluded patients, but training on the full cohort and evaluating on the held-out cohort leaves the patient-disjoint split ambiguous. Please specify the outcome times, landmark eligibility, and expanded-cohort partitions before using these analyses to support prospective prediction and robustness (Section 5.1, Appendices A.1 and C.4, Tables 7, 11, and 12).

Questions for authors
  1. Appendix C.2 reports paired tests, but its interpretation as excluding seed-level randomness needs justification. Please explain how three training runs enter each test and define the Wilcoxon pairing unit for repeated forecast observations. The phenotype aggregation should explain how dependence among labels is handled. Patient-level paired intervals that preserve within-patient dependence would clarify the relevant effect sizes. Corresponding uncertainty on CHoA is particularly useful because escalation AUPRC is 0.073 with displayed variation 0.021, versus 0.068 with 0.004 for GRU-D. This does not establish nonsignificance, but Table 9's MIMIC-only tests do not settle that comparison.

  2. Please reconcile Figure 2 with Equations 2 and 3. The figure places self-attention before gated cross-attention, while the equations specify the reverse. Figure 3 also uses undefined settings mm=2 versus mm=0, whereas the text describes MIST versus GraFITi. Identify both models and define the percentage calculation and aggregation weights. The claim of superiority over every baseline should be qualified by metric, since Table 3 gives better MIMIC forecasting MAE for IMM-TSF, 0.526 versus 0.530 for MIST.

  3. Which implementation produced the first row of Table 4, and does the gain remain against a capacity-comparable joint-stream control with the same inputs and readout? What time, type, or positional information is present in C, and how is full text content bound to each event?

  4. Which exact mortality field was used? For CHoA, how many positive transfers or deaths occur before 24 hours, and how are those patients handled? Please specify prediction landmarks and event-availability timestamps.

  5. In Appendix C.4, does full-cohort training mean training only on the training partition of an expanded population? Please provide retained/excluded patient counts in training, validation, and testing, and confirm that evaluated patients did not enter fitted preprocessing.

  6. What are the paired units and seed-handling rules behind Table 9, including phenotype dependence and repeated forecasting targets? Can you provide corresponding patient-level effect intervals and CHoA uncertainty? These answers would materially affect the scope of the statistical claims.

  7. The perturbed patient cases are illustrations, not strong proof of captured cross-modal reasoning. Please state whether Table 5's probabilities were recomputed on the perturbed records and how cases were selected. Without a matched event-removal or timing intervention, the difference between two models does not identify which highlighted events caused the prediction change. The conclusion should reflect this evidential boundary (Section 5.4, Appendix B.5).

2 comments

We thank the reviewer for sharing the detailed comments and concerns, and we kindly address them as following:

W1 & Q3: Dual-stream ablation concerns, and stream specifications

We apologize for the confusion caused by the wording. The first row of Table 4 follows Section 5.3 and Appendix B.6: we replace MIST's unified stream with a dual-stream, late-fusion mechanism, following prior approaches such as UTDE, to show that the unified-stream pipeline is what contributes. The phrase "naive concatenation" in the caption of Table 4 is inaccurate and we will correct it. The requested per-seed results of this variant are: forecasting MSE .588/.591/.592; mortality AUROC .699/.705/.687; LOS MSE .465/.473/.487; phenotype AUROC .692/.696/.692.

We acknowledge that this ablation changes both the stream organization and the fusion technique at once. We therefore add three controls, reported in the table below together with the late-fusion variant: (1) removing the cross-attention layers while keeping the unified stream and its self-attention; (2) removing the self-attention layers while keeping the cross-attention; and (3) randomly permuting the times of textual events while retaining their content. We do not include a variant that changes only the stream organization, because the unified stream is the basis of both attention layers, and removing it is equivalent to the late-fusion variant.

Forecasting MSEMortality AUROCLOS MSEPhenotype AUROC
MIST Full.586.789.412.725
late-fusion.590.697.475.693
no cross attn.587.738.428.712
no self attn.588.709.468.696
perturb time.589.714.463.694

As shown in the table, each control degrades performance relative to full MIST, though none as much as late fusion. Keeping the unified stream and its self-attention without cross-attention ("no cross attn") already recovers a large part of the gap between late fusion and full MIST (mortality AUROC .738, against .697 and .789), which indicates that the unified stream itself carries cross-modal information; cross-attention then adds the full text semantics on top. Removing self-attention ("no self attn") or perturbing event times ("perturb time") brings performance close to the late-fusion level, which indicates that the gain depends on self-attention over a correctly time-ordered interleaved stream. All numbers are means over three seeds.

Regarding C: it contains the full text embedding only, with no temporal term. The time of each event is carried by its token in Z. Cross-attention retrieves text content from C and adds it, as a residual, to the time-stamped tokens in Z. The temporal relations between events are modeled by the self-attention layers over the unified stream. We will make this explicit around Equation 2.

W2 & Q4 & Q5: outcome definitions and cohort analysis details

The mortality label on MIMIC-III is the in-hospital expire flag, and we will use one consistent name throughout the paper. On CHoA, both adverse-outcome labels occur after the prediction landmark: to align with prior work, we exclude patients whose adverse outcome occurs during the input window, so no positive transfer or death falls within the first 24 hours. We will state this explicitly in the revision. We chose the cohort filters and input lengths on clinical grounds that prior work has also adopted, as stated in Section 5.1 and Appendix A.1. We agree that these criteria define a selected population: our results apply to stays longer than 48 hours with a 24-hour prediction landmark, not to general admissions, and we will state this scope and soften the wording around Table 7. For Tables 11 and 12, we evaluate on 2,365 patients (about 30%) of the held-out cohort and train on the remaining held-out patients merged with the training set of our main experiments; the split is patient-disjoint. Normalization statistics and the PCA projection are fit on training patients only. The held-out patients follow the same rule that outcome labels arise after the input window.

Q1. Missing details and CHoA significance tests

We provide the significance tests on CHoA here, and clarify the test settings as follows. Each model's per-patient predictions are averaged over its three seeds before the test, and the Wilcoxon pairing unit is the patient. The tests therefore measure uncertainty from patient sampling; seed variability is reported separately as the spread in Tables 1–3, and we will reword Appendix C.2 accordingly. For phenotyping, Fisher's method treats the 25 labels as independent, which we consider a reasonable approximation: the average pairwise correlation among the 25 labels across all patients is 0.04. We do not run seed-to-seed tests, because seeds are not paired across different architectures.

(The results below are the same as in our response to Reviewer kzhs, since both reviewers raise the same question.)

TaskEscalationEscalationLOSForecasting
BaselineRAINDROPGRU-DGRU-DGraFITi
p-value9.9e-71.7e-103.2e-411.8e-26

The table above follows the same protocol as Appendix C.2 and is computed on escalation AUROC, LOS MSE, and forecasting MSE. The baselines are the strongest ones on CHoA in Tables 1–3, and MIST significantly outperforms each of them. We note in particular that the forecasting p-value is very small even though the gap itself is small: despite the inherent difficulty of improving forecasting, MIST's improvement over the baseline is consistent across patients.

Regarding AUPRC and patient-level effect intervals, we additionally run a paired patient-level bootstrap (2,000 resamples) on escalation AUPRC and AUROC, and report the 95% percentile interval of the difference (MIST minus baseline) and a two-sided p-value. We acknowledge that MIST's AUPRC gain is borderline, especially against GRU-D. We also note that the labels are extremely imbalanced (1.1% positive, Table 6), which makes AUPRC a noisy metric on this task. We will qualify our claims accordingly in the revision.

95% interval of difference (p-value)Escalation AUPRCEscalation AUROC
vs. RAINDROP[.007, .044] (.006)[.034, .084]
vs. GRU-D[.000, .044] (.051)[.057, .109]

Q2. Attention order clarification

Thank you for pointing these out. The implemented attention order is the one in Figure 2: self-attention followed by gated cross-attention in each layer. We will correct Equations 2–3 and the description in Section 4.2. In Figure 3, mm=0 denotes unimodal GraFITi alone and mm=2 denotes full MIST; we will relabel the figure. Each cell reports the relative change in MSE, (MSE_MIST − MSE_GraFITi) / MSE_GraFITi, computed over the forecast targets of that vital channel and horizon bucket. The aggregate in Table 3 averages over all observed targets, so channels and horizons with more observations carry more weight. We agree that the superiority claim should be qualified by metric (IMM-TSF has a better MIMIC forecasting MAE, .526 vs. .530), and we will correct it in the revision.

Q7. Case study concerns

We acknowledge that the perturbed patients are illustrations rather than quantitative proof. Section 5.4 presents them as case studies "to investigate how MIST benefits downstream patient-outcome prediction": examples of how the cross-modal patterns we describe appear in patient histories, and of how MIST behaves in such scenarios, without revealing actual patient records. We will revise the wording in the conclusion accordingly. The probabilities are computed by feeding the perturbed histories into MIST. The cases were selected manually: we looked for patients with such highly indicative events, identified from clinical knowledge, so they are not a random sample. We also performed an event-removal experiment in which we mask the important events described in Section 5.4. MIST's prediction changes from .63 to .43 for Patient A and from .42 to .57 for Patient B, in both cases moving toward the unimodal GraFITi prediction (.23 and .65).

Reviewer_qsHHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper explores how to effectively utilize the irregularly recorded data such as heart rate and blood pressure in hospitals, along with short textual records of medication and nursing care. The authors propose MIST, a framework that integrates these records into a unified chronological sequence, allowing the numerical and textual data to inform one another. This sequence is then processed by a model specifically designed for vital signs to predict a patient’s clinical status.

Critical strengths
  1. The method is designed in alignment with the features of irregular clinical records. The paper organizes numerical measurements and timestamped short text events into a unified event sequence, while preserving the complete textual semantics for subsequent interaction. This design does not require original records to correspond one-to-one in time, and can jointly handle different types of medical events that occur in succession. In addition, the model retains a dedicated vital sign encoder, combining numerical variation patterns with multimodal context.

  2. The experiments are relatively comprehensive. The experiment involved two clinical cohorts and four prediction tasks. The comparison included not only numerical models but also multimodal methods such as UTDE and IMM-TSF. IMM-TSF used the same ClinicalBERT and GraFITi as MIST, which helped reduce the impact of encoder selection on the comparison.

Critical weaknesses
  1. The gains across different tasks are uneven, and the trade-off between performance and computational cost remains insufficiently clear. Although MIST performs well in predicting adverse outcomes and length of stay, its overall benefit for predicting vital signs is relatively limited. Subgroup analyses provided by the authors indicate that improvements for certain metrics are more pronounced over shorter prediction horizons; however, these gains are not consistent across the entire prediction range. Also, the existing cost analysis only reports on MIST itself and does not compare it with the main baseline, so it is unclear how much additional computational cost is required for these projected benefits.
Questions for authors
  1. What is the computational cost of MIST relative to the main baseline?

  2. What specific textual information does “-text-value” block?

2 comments

We thank the reviewer for sharing the insightful comments and recognizing the methodological contribution and experiment comprehensiveness, and we kindly address the remaining concerns:

W1. & Q1.

We acknowledge that MIST's gains are uneven across tasks. We believe this mainly reflects the differing inherent difficulty of the tasks, as also indicated by the differing spread of baseline performance across them. On vital-sign forecasting in particular, several baselines plateau at an MSE of .590–.592 (MIMIC) or .482–.484 (CHoA), while MIST reaches .586 and .478, with both gaps beyond seed variance. We also acknowledge that the forecasting gain is not uniformly positive, as Figure 3 shows. However, the short-horizon gain is still considerable and is consistent with how clinical text events influence vital signs, as discussed in Section 5.2.

We provide the computational cost of MIST and the multimodal baselines below, all at batch size 64 and for unified-stream lengths of 256/512/1024 tokens. We focus on the multimodal baselines because they process the same inputs as MIST. MIST is comparable to the multimodal UTDE pipeline (faster in training, slightly slower in inference) and slower than IMM-TSF. Even at 1024 tokens, which is above the average stream length in our cohorts (Table 6), training at batch size 64 takes 26.4 GB and fits on a single 32 GB or 40 GB GPU; a smaller batch size reduces this further. In time, MIST stays in the same range as previous dual-stream approaches. Its higher memory use is the price of modeling the full interleaved stream, and it remains practical for clinical cohorts.

ModelTrain ms/step (256/512/1024)Inference ms/step (256/512/1024)Peak GPU memory, GB (256/512/1024)
MIST (as in C.1)66.0/98.9/208.523.3/33.7/71.46.6/11.3/26.4
UTDE79.7/144.9/257.220.3/32.6/61.80.30/0.48/0.86
IMM-TSF26.7/33.4/45.59.5/11.8/16.31.6/2.0/3.1

Q2. The "text-value" component refers to the value conversion described in Section 4.1: "we embed the context phrase with ClinicalBERT and take its projection onto the first principal component". The ablation zero-masks this value, as stated in the caption of Table 4.

Thank you for the response and the additional results. The GPU memory cost is quite high. Compared with IMM-TSF, MIST uses substantially more memory and is slower in both training and inference.

Reviewer_kzhsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes MIST, a model designed for irregular multimodal clinical time-series data. Instead of processing numerical measurements and textual events in separate streams, MIST places them into a shared temporal sequence and uses cross-attention and self-attention to model their interactions. The model also includes a separate vital-sign encoder, whose representation is later combined with the multimodal features for downstream prediction.

The method is evaluated on MIMIC-III and a pediatric CHoA dataset on four tasks, including adverse-outcome prediction, phenotyping, length-of-stay prediction, and vital-sign forecasting. MIST generally outperforms the compared baselines, with the largest improvements appearing in patient-level prediction tasks. The authors also provide ablation studies and several supplementary experiments to examine the contribution and robustness of the proposed design.

Critical strengths
  1. This paper aims to address the significant challenge of modeling irregular and interleaved numerical and textual clinical events. The proposed unified label flow provides a structured way to represent these two modalities along a shared timeline, while a cross-attention mechanism integrates additional textual information.
Critical weaknesses
  1. The CHoA adverse outcome assessment task defines ICU transfer or death as a positive outcome, but it is unclear whether these events will necessarily occur after the 24-hour observation period, raising concerns about potential time loopholes. Furthermore, only MIMIC-III reported results for statistical significance testing. Given the relatively small improvement in AUPRC and the high variability of CHoA, the reliability of the reported benefits remains questionable.
  2. Several noticeable differences exist between the manuscript and the published repository, including event-type encoding, attention order, and CHoA tag construction. While these differences may reflect different configurations, the specific implementation used to obtain the reported results remains unclear.
Questions for authors
  1. This evaluation is limited to two clinical datasets: MIMIC-III and CHoA. Could the authors provide additional external validation based on independent datasets (such as MIMIC-IV) to demonstrate its generalizability across different hospitals and clinical settings?
  2. The most recent baseline included in the experiments is IMM-TSF (2025). Could the authors clarify whether relevant methods from 2025-2026 were considered, and explain the reasons for excluding such methods where applicable?
  3. Could the authors clarify whether adverse outcomes in CHoA strictly occur after the 24-hour observation window, and how potential time leakage can be prevented? Furthermore, given the relatively high variability of reported results, could you provide statistical significance tests for CHoA, particularly AUROC and AUPRC for adverse outcomes?
  4. Could the authors confirm whether the published repository is completely identical to the implementation used in the experiments? Significant differences in event-type encoding, attention order, and CHoA label construction require further clarification to assess experimental reproducibility.
1 comment

We appreciate the reviewer for sharing the comments, and we kindly address them as following:

Q1. Additional external validation

We respectfully note that MIMIC-III and MIMIC-IV come from the same hospital and overlap in years, so MIMIC-IV would not serve as an independent dataset. CHoA and MIMIC-III, in contrast, come from different hospitals and different patient populations (pediatric vs. adult), and both cohorts are of considerable size. We therefore believe the current evaluation already shows that MIST works across distinct clinical settings. We do not claim cross-hospital transfer: the two cohorts are trained and evaluated separately, and we will state this explicitly in the limitations.

Q2. Baseline Selection

We selected baselines under a common standard: they are peer-reviewed, have public code, and fit the irregular multimodal time-series setting. As noted in our related work, irregular multimodal time series remain largely underexplored, which has slowed the development of baseline methods and further supports the contribution of MIST. We are nevertheless happy to run additional comparisons if the reviewer can suggest candidate methods.

W1 & Q3. Leakage Prevention

All adverse outcomes occur after the observation window. The CHoA records in our cohort are not taken from the ICU, and, following previous studies, we exclude all patients whose adverse outcome occurs during the input window. We will state this explicitly in the revision. We provide statistical significance tests for CHoA below.

TaskEscalationEscalationLOSForecasting
BaselineRAINDROPGRU-DGRU-DGraFITi
p-value9.9e-71.7e-103.2e-411.8e-26

The table above follows the same protocol as Appendix C.2 and is computed on escalation AUROC, LOS MSE, and forecasting MSE. The baselines are the strongest ones on CHoA in Tables 1–3, and MIST significantly outperforms each of them. We note in particular that the forecasting p-value is very small even though the gap itself is small: despite the inherent difficulty of improving forecasting, MIST's improvement over the baseline is consistent across patients.

Regarding AUPRC and patient-level effect intervals (also raised by Reviewer PPHz), we additionally run a paired patient-level bootstrap (2,000 resamples) on escalation AUPRC and AUROC, and report the 95% percentile interval of the difference (MIST minus baseline) and a two-sided p-value. We acknowledge that MIST's AUPRC gain is borderline, especially against GRU-D. We also note that the labels are extremely imbalanced (1.1% positive, Table 6), which makes AUPRC a noisy metric on this task. We will qualify our claims accordingly in the revision.

95% interval of difference (p-value)Escalation AUPRCEscalation AUROC
vs. RAINDROP[.007, .044] (.006)[.034, .084]
vs. GRU-D[.000, .044] (.051)[.057, .109]

W2 & Q4. Repository

We acknowledge that the configuration files and comments in the repository are not well maintained, which may lead to duplicated settings and confusion, and we thank the reviewer for pointing this out. The event-type encoding (Section 4.1) and the CHoA label construction (Section 5.1, Appendix A.1) described in the paper are the ones used for all reported results. For the attention order, the implementation applies self-attention followed by gated cross-attention in each layer, as drawn in Figure 2. We will correct Equations 2–3 and the description in Section 4.2 so that the paper is consistent. We will also clean up the repository so that it contains only this configuration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.