RosFormer: Robust Stochastic Continuous-Time Transformer for Irregular Multivariate Time Series

Irregular multivariate time series (IMTS) forecasting is a challenging problem due to the distribution shift incurred by changes in the observation density and missingness patterns. A stochastic continuous-time Transformer incorporates the linear-noise stochastic differential equations (LNSDEs) with a continuous-time attention to attenuate the influence of input perturbations and capture dependencies across observations, which, however, requires a costly numerical calculation of the integration. Patch-based methods, on the other hand, can reduce computation by compressing the local observations into patch embeddings, while the patch compression alone does not ensure robustness to such a shift. This thus motivates us to incorporate the patch-based modeling with a stochastic continuous-time Transformer for a robust and efficient forecasting under distribution shifts. Such an incorporation is non-trivial, as it requires adapting the discrete patch representations to continuous-time modeling, while preserving robustness to input perturbations across both the trajectory evolution and attention aggregation. In this paper, we propose RosFormer, a robust stochastic continuous-time Transformer for an efficient IMTS forecasting under distribution shifts. RosFormer captures the inter-variable dependencies within each patch through a relative-time-aware graph, and uses the resulting embeddings to initialize the LNSDE trajectories at their temporal anchors. These trajectories are then aggregated through compactness-aware attention, with the queries, keys and values sharing one trajectory per variable-level patch to reduce numerical integration costs. We further theoretically analyze the stability of the attention output under bounded perturbations of the initial patch embeddings. Experiments demonstrate the robustness of RosFormer to distribution shifts in the controlled test-time missingness and hospital-disjoint eICU-CRD settings, along with a competitive forecasting and event prediction performance and lower computational cost than the other continuous-time baselines.