Multimodal Irregular Time-series Analysis on Clinical Databases

Clinical datasets are very useful to drive patient outcome prediction. State-of-the-art approaches typically aggregate patient-wise time-series from such databases and build analysis models. However clinical time-series analysis poses unique challenges due to their irregular inputs under interleaved multi-modalites. Current methods are either unimodal, or fail to capture cross-modal interactions across the interleaved modalities. To bridge this gap, we propose to achieve seamless cross-modal interaction, that is, allowing bidirectional information flow across interleaved multimodal inputs. Our model, MIST (Multimodal Irregular Seamless Transformer), serves as a novel clinical time-series model that supports seamless cross-modal interaction under a unified token-stream representation. We evaluate MIST across four tasks on two real-world databases, showing that MIST outperforms state-of-the-art baselines, with 8.8% and 5.4% relative AUROC gains on adverse-outcome prediction, a 4.8% AUROC gain on MIMIC phenotyping, 12.3% and 18.5% MSE reductions on length-of-stay, and 0.7% and 0.8% MSE reductions on vital-sign forecasting averaged over all horizons, rising above 10% at short horizons. We further demonstrate the multimodal gains through component ablations and case studies.