- Rating
- 2
- Confidence
- 3(−1)
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RosFormer, a stochastic continuous-time Transformer for irregular multivariate time series forecasting under observation-process distribution shifts. The method constructs patch-level representations and uses a relative-time-aware graph to model cross-variable dependencies. These representations initialize linear-noise stochastic differential equations, with queries, keys, and values derived from shared stochastic trajectories. A compactness-aware continuous-time attention mechanism then aggregates the trajectories for prediction. The paper further provides a theoretical analysis of attention-output stability under bounded perturbations. Experiments evaluate robustness under controlled test-time missingness and hospital-disjoint distribution shifts, alongside standard forecasting and event prediction benchmarks, ablation studies, and computational efficiency analyses.
Critical strengths
The paper provides a comprehensive empirical evaluation across forecasting, event prediction, and distribution-shift settings. RosFormer achieves competitive predictive performance, particularly under severe test-time missingness on certain datasets. The efficiency analysis also demonstrates substantial computational savings while largely preserving predictive accuracy, supporting the practical effectiveness of the overall framework.
Critical weaknesses
1. Limited Methodological Novelty
The paper presents the integration of patch-based graph representations with LNSDE-driven continuous-time attention, along with three associated integration challenges, as its central methodological contribution. While the paper acknowledges the relevant foundations, the novelty of this integration appears limited. In particular, SDEFormer (Wang et al., 2025) has already incorporated stable Neural SDEs, including an LNSDE variant, into continuous-time Transformer attention for irregular time series modeling. This closely related work is not discussed, despite directly addressing a central component of RosFormer's proposed framework. On the patch side, the proposed approach builds on established mechanisms in T-PatchGNN (Zhang et al., 2024), Hi-Patch (Luo et al., 2025), and VIMTS (Hu et al., 2025), which the paper appropriately cites. Although the three integration challenges motivate specific design choices, their solutions primarily involve adapting existing patch representations to stochastic trajectories, extending attention with temporal weighting and stability analysis, and reducing computational overhead through patch-level evolution and shared Q/K/V trajectories. These are meaningful architectural refinements, but they do not appear to introduce a substantially distinct modeling mechanism. Consequently, the methodological contribution seems to lie mainly in a particular integration and localized extensions of established techniques, rather than a major methodological advance.
2. Insufficient Validation of the Distinctive Contributions
Although the experiments demonstrate the effectiveness of the overall framework, they do not sufficiently isolate the benefits of RosFormer's specific integration and design choices. The comparisons with existing models and the component-removal ablations do not clearly establish how much of the improvement arises from the proposed adaptations, rather than the established mechanisms themselves. This limits the empirical support for the paper's methodological contributions.
Questions for authors
The paper presents the non-trivial integration of patch-based representations with LNSDE-driven continuous-time attention as its main methodological contribution. Could the authors provide convincing evidence that RosFormer's specific integration offers substantial robustness benefits over reasonable alternative combinations of these established techniques? Such comparisons should incorporate both patch-based modeling and LNSDE-driven attention, rather than relying on component-removal ablations. While Appendix B evaluates a ContiFormer + LNSDE variant in terms of computational efficiency, this variant does not incorporate patch-based representations, and its predictive performance and robustness are not reported. More direct evidence would help clarify the methodological value of RosFormer's specific designs beyond combining existing mechanisms.
9 comments
The experimental settings and detailed results are presented below.
Forecasting under controlled test-time missingness
We adapted SDEFormer-LN to our forecasting experiments and compared it with RosFormer and RosFormer (Linear Q), a patch-based variant that constructs Q by linear interpolation. Full denotes the original test inputs. The other settings randomly remove 30%, 50%, or 70% of the observed test inputs while keeping the prediction targets unchanged. Each entry reports MSE / MAE, with the metric scales shown alongside each dataset.
Table C.1. Forecasting performance under controlled test-time missingness.
| Dataset | Model | Full | 30% | 50% | 70% |
|---|---|---|---|---|---|
| Human Activity (×10⁻³ / ×10⁻²) | SDEFormer-LN | 2.77 / 3.34 | 3.11 / 3.45 | 3.64 / 4.02 | 4.80 / 4.94 |
| RosFormer (Linear Q) | 2.67 / 3.11 | 2.94 / 3.39 | 3.34 / 3.80 | 4.49 / 4.58 | |
| RosFormer | 2.54 / 3.05 | 2.71 / 3.24 | 2.89 / 3.38 | 3.37 / 3.68 | |
| PhysioNet (×10⁻³ / ×10⁻²) | SDEFormer-LN | 5.06 / 3.96 | 6.21 / 4.85 | 6.86 / 5.15 | 8.72 / 5.45 |
| RosFormer (Linear Q) | 4.70 / 3.61 | 5.06 / 3.96 | 5.65 / 4.26 | 6.96 / 4.78 | |
| RosFormer | 4.55 / 3.46 | 5.00 / 3.86 | 5.61 / 4.08 | 6.73 / 4.60 | |
| USHCN (×10⁻¹ / ×10⁻¹) | SDEFormer-LN | 4.96 / 3.00 | 5.10 / 3.13 | 5.17 / 3.26 | 5.25 / 3.40 |
| RosFormer (Linear Q) | 4.90 / 2.95 | 4.94 / 3.02 | 5.00 / 3.08 | 5.04 / 3.11 | |
| RosFormer | 4.84 / 2.92 | 4.86 / 2.92 | 4.92 / 2.95 | 4.96 / 2.98 | |
| MIMIC-III (×10⁻² / ×10⁻²) | SDEFormer-LN | 1.79 / 7.51 | 1.94 / 8.20 | 2.07 / 8.56 | 2.12 / 8.66 |
| RosFormer (Linear Q) | 1.44 / 6.56 | 1.52 / 6.74 | 1.57 / 6.94 | 1.67 / 7.16 | |
| RosFormer | 1.30 / 6.32 | 1.43 / 6.68 | 1.49 / 6.78 | 1.53 / 7.12 |
RosFormer maintains lower prediction errors than both alternatives across all four datasets. The comparison with Linear Q supports the value of trajectory-based query construction under sparse inputs. On Human Activity, for example, MSE increases by 32.7% from Full to 70% missingness for RosFormer, compared with 68.2% for Linear Q, providing additional evidence of reduced sensitivity to observation removal on this dataset.
Efficiency and predictive performance of trajectory sharing
To evaluate trajectory sharing, we compare RosFormer with RosFormer (Independent LNSDEs for Q/K/V). The independent variant retains patch-based representations but solves three separate LNSDE trajectories for Q, K, and V. The shared variant generates all three through projections of one latent trajectory.
The efficiency comparison uses Human Activity with a batch size of 32. Training and inference times are measured per epoch.
Table C.2. Computational efficiency of independent and shared trajectories on Human Activity.
| Model | Training time (s/epoch) | Inference time (s/epoch) | GPU memory (MB) |
|---|---|---|---|
| RosFormer (Independent LNSDEs for Q/K/V) | 25.32 | 9.26 | 5828 |
| RosFormer (Shared trajectory) | 8.55 | 3.76 | 4048 |
We also evaluate both variants under the controlled test-time missingness protocol to examine whether these efficiency gains affect forecasting performance.
Table C.3. Forecasting performance of independent and shared trajectories under controlled test-time missingness. Each entry reports MSE / MAE, with the lowest confirmed MSE in bold.
| Dataset | Model | Full | 30% | 50% | 70% |
|---|---|---|---|---|---|
| Human Activity (×10⁻³ / ×10⁻²) | RosFormer (Independent LNSDEs for Q/K/V) | 2.52 / 3.05 | 2.73 / 3.20 | 2.95 / 3.38 | 3.42 / 3.76 |
| RosFormer (Shared trajectory) | 2.54 / 3.05 | 2.71 / 3.24 | 2.89 / 3.38 | 3.37 / 3.68 | |
| PhysioNet (×10⁻³ / ×10⁻²) | RosFormer (Independent LNSDEs for Q/K/V) | 4.53 / 3.59 | 4.99 / 3.88 | 5.47 / 4.09 | 6.77 / 4.63 |
| RosFormer (Shared trajectory) | 4.55 / 3.46 | 5.00 / 3.86 | 5.61 / 4.08 | 6.73 / 4.60 | |
| USHCN (×10⁻¹ / ×10⁻¹) | RosFormer (Independent LNSDEs for Q/K/V) | 4.86 / 2.94 | 4.88 / 2.95 | 4.93 / 2.97 | 5.00 / 3.01 |
| RosFormer (Shared trajectory) | 4.84 / 2.92 | 4.86 / 2.92 | 4.92 / 2.95 | 4.96 / 2.98 | |
| MIMIC-III (×10⁻² / ×10⁻²) | RosFormer (Independent LNSDEs for Q/K/V) | 1.32 / 6.36 | 1.44 / 6.70 | 1.52 / 7.02 | 1.57 / 7.23 |
| RosFormer (Shared trajectory) | 1.30 / 6.32 | 1.43 / 6.68 | 1.49 / 6.78 | 1.53 / 7.12 |
The two variants achieve similar forecasting errors across the evaluated missingness levels. Together with the efficiency results, this supports trajectory sharing as a way to reduce computation while retaining comparable predictive performance under changing observation patterns. We will include these additional evaluations in the revised manuscript.
Weakness 1.2: Methodological Novelty
Reviewer comment:
1. Limited Methodological Novelty
On the patch side, the proposed approach builds on established mechanisms in T-PatchGNN (Zhang et al., 2024), Hi-Patch (Luo et al., 2025), and VIMTS (Hu et al., 2025), which the paper appropriately cites. Although the three integration challenges motivate specific design choices, their solutions primarily involve adapting existing patch representations to stochastic trajectories, extending attention with temporal weighting and stability analysis, and reducing computational overhead through patch-level evolution and shared Q/K/V trajectories. These are meaningful architectural refinements, but they do not appear to introduce a substantially distinct modeling mechanism. Consequently, the methodological contribution seems to lie mainly in a particular integration and localized extensions of established techniques, rather than a major methodological advance.
Response:
RosFormer builds on established patch encoders and LNSDE dynamics. Its methodological contribution lies in jointly accounting for observation-process shifts when constructing initial patch states and aggregating their stochastic trajectories. Temporal anchors determine both the relative-time graph corrections and the starting times of continuous evolution, while compactness modulates both these corrections and attention weights. These connections make the temporal structure of the observed data an explicit part of the modeling process from cross-variable interaction to trajectory aggregation.
1). Observation timing links patch-state construction with stochastic attention.
Changes in observation patterns can alter both the representative time of a patch and the temporal spread of its observations. RosFormer retains these properties explicitly through temporal anchors and compactness scores after patch encoding.
When a patch's anchor changes, its temporal offsets from other variables change accordingly. These offsets enter graph construction, affecting the initial state supplied to the LNSDE, while the same anchor specifies when that state begins to evolve. Cross-variable aggregation and continuous-time initialization therefore use a consistent description of the patch's temporal position.
Changes in temporal dispersion are handled across these stages as well. When observations are more dispersed around an anchor, compactness reduces the magnitude of the relative-time edge correction and adjusts the patch's contribution to attention. At equal trajectory similarity, a more concentrated patch receives greater weight. Compactness thus connects how a patch participates in constructing initial states with how its evolved trajectory contributes to the prediction.
2). A direct combination baseline assesses the additional value of this formulation.
The added Patch + SDEFormer-LN baseline combines VIMTS's [C2] patch-processing module with SDEFormer-LN. This comparison retains both established components and evaluates the benefit of RosFormer's specific formulation.
As shown in Tables C.4 and C.5 (see our response to Weakness 2 & Question), RosFormer achieves lower MSE and MAE across all four datasets and evaluated missingness levels, as well as on hospital-disjoint eICU. For example, at 70% missingness on Human Activity, MSE decreases from to . These results support the practical value of the proposed design under observation-process shifts, complementing the methodological distinctions described above.
The revised manuscript will clarify the connections between these designs, distinguish them from the inherited components, and include the additional comparison.
Reviewer comment:
1. Limited Methodological Novelty
The paper presents the integration of patch-based graph representations with LNSDE-driven continuous-time attention, along with three associated integration challenges, as its central methodological contribution. While the paper acknowledges the relevant foundations, the novelty of this integration appears limited. In particular, SDEFormer (Wang et al., 2025) has already incorporated stable Neural SDEs, including an LNSDE variant, into continuous-time Transformer attention for irregular time series modeling. This closely related work is not discussed, despite directly addressing a central component of RosFormer's proposed framework.
Response:
Thank you for highlighting SDEFormer [C1]. We will include it in the related work and clarify its relationship to RosFormer in the revised manuscript. Both methods use neural SDEs within continuous-time attention. We summarize their differences and the additional designs in RosFormer below.
1). Different motivations.
RosFormer focuses on forecasting robustness to distribution shifts in the observation process, including changes in sampling density and missingness between training and deployment. SDEFormer-LN focuses on modeling stochastic continuous-time dynamics in irregular time series. RosFormer additionally introduces time-aware patch modeling, compactness-aware attention, and LNSDE-based query dynamics to improve forecasting robustness under these shifts.
2). Different model architectures.
RosFormer constructs continuous-time attention from variable-level patch representations. Each patch evolves along a latent LNSDE trajectory, from which Q, K, and V are obtained through separate projections. Thus, all three representations follow learned stochastic dynamics. SDEFormer-LN constructs Q by linear interpolation between observations and models K and V using separate LNSDEs.
The difference in query construction is particularly relevant when observations become sparse. Larger observation gaps constrain the interpolated Q to vary linearly over longer intervals. RosFormer allows Q to follow learned nonlinear dynamics over these intervals, providing greater flexibility in modeling queries under changes in observation patterns.
3). Improved computational efficiency.
Compared with SDEFormer-LN, which evolves K and V using two separate LNSDEs, RosFormer generates Q, K, and V from a single shared LNSDE trajectory per variable-level patch. SDEFormer-LN obtains Q through linear interpolation, and introducing a separate LNSDE for Q would require a third trajectory. RosFormer's shared construction enables stochastic modeling of all three representations without this additional integration cost.
4). Additional theoretical analysis.
RosFormer and SDEFormer-LN both incorporate LNSDEs into continuous-time attention. RosFormer further analyzes how perturbations to initial patch representations propagate through the latent trajectories and attention weights to affect the attention output. The analysis holds temporal anchors and compactness scores fixed under the stated assumptions.
5). Experimental validation.
RosFormer achieves lower MSE and MAE than SDEFormer-LN across all four datasets and evaluated missingness levels. Additional experiments evaluate the query modeling and trajectory sharing designs discussed above. The comparison with RosFormer (Linear Q) shows that LNSDE-based query modeling improves forecasting performance under test-time missingness. Shared and independent trajectories achieve comparable predictive performance, while sharing reduces training time, inference time, and GPU memory usage by 66.2%, 59.4%, and 30.5%, respectively, on Human Activity.