acceptodds
Under review as a conference paper at ICLR 2027

CRFormer: Covariance-Enhanced Rough Transformer for Multivariate Time Series Modeling

Abstract

Multivariate time series modeling faces the dual challenges of capturing complex spatial dependencies and maintaining computational efficiency over long sequences. Existing signature-based methods, including the Rough Transformer (RFormer), often overlook the covariance structure of multivariate sequences, leading to inadequate modeling of spatial dependencies. In this work, we propose Covariance-Enhanced Rough Transformer (CRFormer), which extends RFormer by introducing a Covariance Rough Path Module (CRP Module) that integrates the geometric structure of covariance matrices with path signature theory. CRP Module explicitly constructs a dynamic covariance path alongside the raw temporal path, forming a dual-signature stream that effectively captures complex spatiotemporal inter-dependencies. We further establish the theoretical equivalence between CRFormer and Neural CDEs in the continuous-time limit, achieving comparable expressiveness with substantially fewer layers while providing rigorous interpretability and convergence guarantees. More importantly, the CRP Module is backbone-nonintrusive and can be seamlessly integrated into benchmark time series backbones to enhance their spatial dependency modeling capability. Extensive experiments demonstrate that CRFormer significantly outperforms competing baselines across diverse benchmarks, while CRP-enhanced models deliver consistent accuracy gains alongside dramatic reductions in computational cost and memory consumption across multiple backbone architectures, validating the effectiveness and broad applicability of the CRP Module. The code is available at a repository: https://anonymous.4open.science/r/CRFormer-FDC8/.

Primary area:
learning on time series and dynamical systems
Keywords:
Multivariate Time Series, Time Series Modeling, Path Signatures, Rough Path Theory

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ycjoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

CRFormer combines truncated signatures of the raw time-series path and a Cholesky-embedded sliding-covariance path for classification and forecasting with Transformer and other backbones. Experiments report predictive improvements and computational savings, alongside theoretical claims concerning noise robustness and equivalence to Neural CDEs.

Critical strengths
  1. Encoding dynamic covariance as a separate signature path offers a useful representation for explicitly modeling changing dependencies between variables.
  2. Evaluations across six backbones and comparisons with direct covariance inputs provide broad empirical evidence for assessing the representation’s transferability.
Critical weaknesses
  1. Equation (79) in Theorem 4 equates an ODE depending on control values with a CDE driven by control increments without the necessary construction, leaving the claimed equivalence and resulting guarantees unsupported.
  2. Theorem 2 omits the σ2I\sigma^2I covariance bias induced by additive noise, leaving the uncorrected estimator with a mean-squared error of at least dσ4d\sigma^4, contrary to its claimed vanishing-error conclusion.
Questions for authors
  1. Given that Table 6 truncates baseline inputs while CRP processes complete sequences, do the main predictive gains persist under an equal input-information budget?
  2. How does Table 10’s O(Td2)O(Td^2) overhead account for the Cholesky decomposition and O(Td4)O(Td^4) second-order covariance signatures in Theorem 3, and do the efficiency gains survive complete cost accounting?
  3. Since fixed-count covariance windows depend on sampling density, what windowing and sampling assumptions make Theorem 7 applicable to irregularly sampled observations?
Reviewer_btXbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CRFormer, a model that combines signatures of raw temporal paths and dynamic covariance paths to model MTS. The CRP Module is also evaluated as an input representation for several Transformer-based and state-space-based backbones. Experiments cover classification, forecasting, and synthetic tasks, together with theoretical claims concerning approximation, robustness, and Neural CDE equivalence.

Critical strengths
  1. The paper includes multiple backbone evaluations, stream ablations, and direct covariance-input controls.
  2. Explicitly modeling evolving cross-variable covariance is a well-motivated direction.
Critical weaknesses

Major:

  1. In lines 303 ~ 335 of scripts/train_baseline.py, the classification pipeline loads the official training/test splits but does not create a separate validation set, which is inconsistent with the Val/Test Split configuration reported in Table 5, although it is unclear whether this script was used for those experiments. In addition, test accuracy is evaluated after every epoch and used for early stopping and checkpoint selection, and this checkpoint is subsequently evaluated on the same test set. The authors are suggested to clarify which reported results were produced using this pipeline and how an independent held-out evaluation was ensured.

  2. In lines 266–300 of scripts/train_baseline.py, the direct data loader fits a StandardScaler on the entire set before training/validation/test split, which introduces preprocessing leakage as the statistics of vaildation and test influence the transformation applied to the training data. The scaler should be fitted exclusively on the training set.

  3. The configuration of the Regression Datasets in Table 6 in Appendix F.2 indicates that the model predicts only one numeric variable, while the script predicts only the first numeric variable one step ahead. Although this could be regarded as a single-step forecasting setting, this evaluation does not establish performance across multiple-step horizons or on multivariate-output forecasting tasks. Please clarify the scope of the forecasting claims.

  4. Table 6 caption states that plain backbones are trained on truncated time series (T=2,000) while CR-enhanced variantes process the full sequences, which makes unfair and confounded comparison. Comparisons using matched input lengths would help isolate the contribution of the CRP module.

  5. In Table 7, the highest mean classification accuracy of the presented model does not show consistent performance (the std values of its performance on SCP-2, Motor, and Heartbeat are relatively high) and does not establish consistent superiority across datasets. Therefore, the authors are encouraged to provide uncertainty estimates for the performance differences and discuss the conditions under which the method is beneficial or less effective, preferably in a separate Limitations or Discussion section.

  6. The CRP module is presented as broadly applicable to time-series modeling in both Abstract and Introduction, but its empirical evaluation is limited to classification and one-step forecasting. Its applicability for temporal localization (such as anomaly detection) and pointwise reconstruction (such as imputation) remains underexplored. The authors could either provide supporting evidence for these broader applications or clarify the scope of their claims and discuss how signature compression might affect such tasks.

  7. The following reference entries require correction:

  • "The great multivariate time series classification bake off: a review and experimental evaluation of recent algorithmic advances": The journal name is lost.
  • "Dynamic covariance models for multivariate financial time series": The third author should be listed as Zoubin Ghahramani rather than Ghahramani Zoubin.

Minor:

  1. The expanded classification comparison in Table 7 covers only four datasets. Evaluating a broader subset of the UEA archive would strengthen the evidence and facilitate comparison with prior MTSC studies, such as TVNet [1], TimesNet [2], and ConvTimeNet [4].
  2. The computational efficiency, cost, and memory comparisons in Appendices G.2 and G.3 primarily cover state-space- and attention-based models. Broader efficiency claims would benefit from comparisons with more diverse architectures, including convolution- and frequency-based models such as TVNet [1], TimesNet [2], TSLANet [3], and ConvTimeNet [4]. Otherwise, the claim should be limited to efficiency improvements over the evaluated state-space- and attention-based baselines.
  3. The citation for Mamba in line 77 is unresolved and appears as "Mamba (?)". Please correct the citation.
  4. Figure 9 contains an isolated “t” that appears unrelated to the surrounding content. Please check whether it is unintended and remove it if appropriate.

References for reviews: [1] Li, Chenghan, Mingchen Li, and Ruisheng Diao. "TVNet: a novel time series analysis method based on dynamic convolution and 3D-variation." International Conference on Learning Representations. Vol. 2025. 2025.

[2] Wu, Haixu, et al. "Timesnet: Temporal 2d-variation modeling for general time series analysis." arXiv preprint arXiv:2210.02186 (2022).

[3] Eldele, Emadeldeen, et al. "Tslanet: Rethinking transformers for time series representation learning." AI for Time Series. CRC Press, 2024. 35-57.

[4] Cheng, Mingyue, et al. "Convtimenet: A deep hierarchical fully convolutional model for multivariate time series analysis." Companion Proceedings of the ACM on Web Conference 2025. 2025.

Questions for authors
  1. Do the authors correct the reference entries?
  2. Can the authors reconcile the released implementation with the split configuration in Table 5?
  3. Were the reported forecasting results obtained using the direct data loader, which fits StandardScaler before splitting the data? If so, how do the results change when normalization statistics are estimated exclusively from the training set?
  4. How are the performance when plain models and CR-enhanced variants are trained using the MTS with the same input lengths? Can the authors explicitly provide the gains when CRP Module is applied?
  5. Do the authors provide uncertainty estimates for the performance and discuss the substantial variability in Table 7?
  6. How do signature compression affect temporal localization and pointwise reconstruction tasks?
  7. Do the efficiency advantages extend to competitive convolution- and frequency-based models, or should the efficiency claims be restricted to the evaluated state-space- and attention-based baselines?
Reviewer_HQ5vOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes CRFormer, which extends Rough Transformer with a covariance-signature branch. It constructs sliding-window covariance matrices, maps them to Cholesky coordinates, and combines their truncated path signatures with raw-path signatures before Transformer processing. The resulting CRP module can also be attached to other sequence models. The paper claims approximation, robustness, and Neural CDE equivalence guarantees, and evaluates the approach on classification, forecasting, and synthetic tasks, including backbone comparisons and stream ablations.

Critical strengths
  1. A reasonable modeling idea with some supporting evidence. Explicit covariance dynamics could provide a useful inductive bias when raw-path signatures are truncated. Tables 13–14 show improvements over the raw-signature-only variant across the tested backbones, supporting an incremental benefit from the covariance branch.
  2. Evaluation across multiple architectures. The plug-in experiments cover both Transformer and state-space backbones and report predictive performance and computational costs. This broadens the potential relevance of the proposed representation beyond CRFormer itself.
Critical weaknesses
  1. Several central theoretical guarantees appear unsupported or incorrect as stated. In Theorem 2, additive isotropic noise introduces a covariance bias of (\sigma^2 I), so the expected squared error relative to the clean covariance cannot generally vanish as (O(\sigma^2/w)). Equation (52) subtracts this bias, although the proposed estimator does not. In the proof of Theorem 4, Eq. (79) equates an ODE depending on (M(s)) with a CDE driven by (dM(s)); these are generally different, particularly because (M) is piecewise constant. Theorem 1 also relies on recovering a time integral from a first-order signature, which instead gives a path increment. These issues affect the paper’s main theoretical contributions and seem to require substantial revision.
  2. The experiments do not yet establish the claimed strength and generality of the contribution. The expanded comparison reports only about a 0.45% improvement in mean regression MAE over the closest baseline, with substantial losses on some individual datasets. Although means and standard deviations are provided, the number of repeated runs and statistical significance are unclear. Attribution is also complicated by the disclosed settings: Heartbeat disables the dual stream in Table 5, plug-in forecasting uses a one-step horizon, and some plain backbones receive truncated sequences while CRP variants receive full sequences. The existing ablations support adding a covariance branch, but do not isolate the necessity of Cholesky geometry or separate its benefits from compression and additional input history.
Questions for authors
  1. Theoretical clarification: Is the covariance estimator actually debiased by subtracting (\sigma^2 I)? If not, how does Theorem 2 account for the nonvanishing covariance bias?
  2. Theoretical clarification: What additional assumptions justify Eq. (79), given that (dM(s)=0) inside each constant segment while the ODE vector field may remain nonzero? How is the time integral used in Appendix D.1 recovered from the stated signature representation?
  3. Experimental clarification: How many runs underlie Tables 7–8, and were competing models evaluated using matched splits, input lengths, and tuning budgets? Please also explain the disabled covariance branch for Heartbeat and whether the reported gains persist under equal-input comparisons.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.