acceptodds
Withdrawn from ICLR 2027

Scaling Multivariate Time Series Forecasting with Context-Modulated Prototype Attention

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multivariate time series forecasting is fundamental to a wide range of real-world applications, from traffic and energy systems to large-scale retail platforms, where hundreds to tens of thousands of interacting variables must be forecast jointly. Full pairwise attention captures rich cross-variate interactions but incurs quadratic complexity in the number of variates, while efficient alternatives reduce this cost by restricting or compressing cross-variate communication, potentially sacrificing flexible, input-dependent interactions. Real-world forecasting further requires incorporating known covariates, such as calendar events and promotions, and characterizing predictive uncertainty. We propose SLIM, a unified Transformer architecture built on context-modulated prototype attention (CoPA). CoPA compresses cross-variate interactions into a small set of latent prototypes formed from the history, and lets known covariates modulate only the queries that retrieve from them, so that the same shared representation is read differently under different forecast contexts. This yields cost linear in the number of variates, adds a covariate path that is independent of the prototype budget, and supports quantile forecasting within a single backbone. Extensive experiments across short-term, long-term, high-dimensional, and context forecasting benchmarks demonstrate state-of-the-art accuracy with substantially lower memory and computational costs.

Primary area:
learning on time series and dynamical systems
Keywords:
multivariate time series forecasting, prototype attention, cross-variate modeling, scalable transformers

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Z2CXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces SLIM, a scalable Transformer for multivariate time series forecasting based on Context-Modulated Prototype Attention. The proposed method replaces quadratic cross-variable attention with a prototype-based latent interaction mechanism, where prototypes are constructed from historical observations and known covariates modulate only the retrieval process. This enables linear scaling with respect to the number of variables while preserving input-dependent cross-variate interactions.

Critical strengths

s1: The paper has a strong motivation. It addresses a critical scalability issue in multivariate time series forecasting by identifying the trade-off between expressive cross-variate modeling and computational efficiency.

s2: Experiments on various forecasting benchmarks show that SLIM achieves strong predictive performance with significantly improved computational and memory efficiency, especially in high-dimensional settings.

Critical weaknesses

w1: The paper is relatively difficult to read and provides limited intuitive insights into the proposed mechanism. More explanations and qualitative analyses are needed to clarify why the proposed design works.

w2: The experiments mainly show that SLIM achieves better performance than existing methods, but provide insufficient understanding of the underlying factors contributing to its effectiveness.

w3: Despite the emphasis on scalability, the evaluation does not sufficiently cover truly large-scale datasets. More experiments with real-world datasets containing much larger numbers of variables are needed.

w4: The paper lacks a discussion of the limitations of SLIM, including potential failure cases, sensitivity to prototype selection, and trade-offs between scalability and representation capacity.

Questions for authors

Please refer to the weaknesses.

Reviewer_kKKdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SLIM, a framework for large-scale multivariate time series forecasting. Its core component, Context-Modulated Prototype Attention (CoPA), compresses cross-variate information into a small set of latent prototypes to reduce computational costs. Known covariates are used to modulate retrieval queries, allowing the model to adjust cross-variate interactions based on the forecasting context.

The authors evaluate SLIM on short-term, long-term, high-dimensional, and covariate-aware forecasting tasks, showing good forecasting accuracy and efficiency.

Critical strengths

S1: The paper addresses efficiency and covariate modeling in large-scale multivariate forecasting, which is a practical and relevant problem.

S2: The model is clearly designed and reduces the cost of cross-variate attention through a small set of shared prototypes.

S3: The experiments cover various forecasting tasks, with additional efficiency analysis, sensitivity analysis, and ablation studies.

Critical weaknesses

W1. Incremental novelty compared with existing efficient cross-variate modeling methods.

The proposed method mainly combines existing techniques, including prototype aggregation, latent attention, and FiLM-based query modulation. Similar ideas of compressing cross-variate interactions through latent representations have been explored in Crossformer and SOFTS. Therefore, CoPA seems more like a combination of existing techniques than a fundamentally new approach.

Recent methods such as FACT and VPNet have also explored efficient cross-variate modeling. The paper does not clearly explain what makes CoPA better or different.

W2. Insufficient justification and validation of the core design.

The paper separates history-based prototype construction from covariate-based retrieval. Prototypes are built only from historical data, while known covariates only affect the retrieval queries.

However, it is unclear why covariates should not be involved in prototype construction, or when this separation would work better than joint modeling.

W3. Missing recent baselines and insufficient ablation studies.

Although the paper compares against many baselines, most are from 2022–2025. It lacks comparisons with recent ICLR 2026 methods such as FACT, GTR, and VPNet, which explore fine-grained cross-variate modeling, global temporal retrieval, and local cross-variate dependencies. Without these comparisons, it is difficult to assess the advantages of SLIM over recent methods.

The long-term forecasting experiments also miss the widely used Traffic dataset (862 variables). Although Traffic-CA and Traffic-GBA are included in the high-dimensional experiments, they are different datasets and cannot replace the standard Traffic benchmark.

The ablation studies mainly focus on the number of prototypes and where covariates are applied. Comparisons with fixed or learnable prototypes, simple low-rank attention, and direct covariate fusion are still missing, making it difficult to identify the contribution of each component.

Questions for authors

Q1: What is the main contribution of CoPA compared with existing prototype aggregation, low-rank attention, and efficient cross-variate modeling methods? Why is it necessary to separate prototype construction from covariate-based retrieval?

Q2: Could the authors add comparisons with FACT, VPNet, and GTR, as well as results on the standard Traffic dataset? Could they also compare computational efficiency with these methods?

Q3: Could the authors compare fixed prototypes, learnable prototypes, removing Gumbel noise, and direct covariate fusion to better show the contribution of each component? Does query modulation also work better on other covariate-aware datasets?

Reviewer_Xxq7OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SLIM, a Transformer forecaster that replaces dense cross-variate attention with a small set of history-dependent latent prototypes. Context-modulated prototype attention pools keys and values from historical variate representations, then uses known covariates to modulate only the final-layer retrieval queries. A nine-quantile head supplies point and interval predictions. Validation covers 29 datasets across short-term, long-term, high-dimensional, and covariate-aware forecasting, together with common-shape GPU measurements and a controlled conditioning-placement study on Favorita.

Critical strengths
  • Substantial empirical coverage and measured scalability.
  • The paper measures runtime and memory on the same device and common tensor shapes, and acknowledges cases where alternatives are faster or more memory-efficient.
  • The paper has a specific, testable conditioning design. Section 4.7 goes beyond adding covariates to a model: it compares no covariates, look-back covariates, full query conditioning, and prototype conditioning under a shared backbone.
Critical weaknesses
  • Limited methodological novelty. The latent-token communication mechanism follows the attention families acknowledged in Section 2, while query modulation is a standard feature-wise affine operation. Propositions 1–2 largely formalize the architecture: they do not establish an approximation, expressivity, generalization, or optimal-placement advantage. This is a substantive contribution concern. The paper should identify the precise advance that remains beyond combining these established ingredients and narrow its theoretical claims accordingly.

  • The conditioning-placement conclusion exceeds the evidence. Section 4.7 studies the central placement choice on Favorita alone. Moreover, the prototype-conditioning alternative changes three sites simultaneously, so it does not cleanly isolate placement under matched parameterization. The evidence supports one useful configuration more convincingly than a general rule that covariates should modify retrieval while leaving prototypes unchanged. The authors should clarify the control and scope the conclusion to what it establishes.

  • The point-forecast definition is ambiguous after quantile sorting. Section 3.4 claims that sorting nine quantile outputs removes crossings without affecting the median. Sorting can change the value originally assigned to the 0.5 quantile. Appendix F does not resolve whether Tables 1–4 evaluate that original output or the fifth order statistic. This is a fixable reporting issue with direct consequences for interpreting the reported point errors: specify the exact prediction used and correct the median-preservation claim.

  • The uncertainty contribution lacks aggregate validation. Although interval prediction is part of the contribution, Appendix G supplies selected examples rather than aggregate quantile-loss, coverage, and sharpness comparisons. Those examples cannot establish a general uncertainty-quality advantage. The authors should identify any existing aggregate evidence or qualify the uncertainty claim; resolving this fully may require additional evaluation beyond a short rebuttal.

  • Baseline provenance contains an unexplained numerical inconsistency. Table 9 repeats all four SCINet MSE/MAE pairs exactly between ETTm2 and Electricity. This does not establish that the overall results are incorrect, but it requires an explanation before the comparison is auditable. The authors should verify the entries, correct any transcription errors, and identify the source or execution protocol for these baseline results.

  • Training variability and selected configurations are insufficiently documented. The main accuracy results lack training-seed variability and sufficiently complete selected configurations. The placement-study bootstrap conditions on trained models and therefore do not establish robustness across training runs. These are fixable reporting limitations where records already exist; otherwise, the strength of the empirical conclusions should be reduced until training variability is assessed.

Questions for authors
  1. Are point errors computed from the original output labelled 0.5 or from the fifth value after quantile sorting? Please correct or qualify the claim that sorting preserves the median output.

  2. Why do the SCINet entries in Table 9 match exactly between ETTm2 and Electricity at every horizon? Were these baselines rerun, and can you identify the selected configurations and training-seed protocol?

  3. What explains Favorita query-conditioning MSE of 0.4210 in Table 4 versus 0.4283 in Figure 4? Please identify any differences in configuration, checkpoint, or seed.

  4. Which already available result separates query placement from parameterization or training variability, and which aggregate probabilistic metrics support the uncertainty claim? No additional runs are requested for rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.