acceptodds
Under review as a conference paper at ICLR 2027

Select, Don't Compress: Sparse Attention for High-Dimensional Time Series Forecasting

Abstract

In high-dimensional multivariate time series, forecasting accuracy may depend on specific interactions between individual channels. Dense cross-channel attention captures them, but its cost grows quadratically with the number of channels. Traditional alternatives either compress these interactions or remove them altogether, potentially losing the fine-grained dependencies that matter. We introduce Native Sparse Time Attention (NSTA), which preserves direct interactions between individual channels while scaling linearly in their number for a fixed sparsity budget. On both evaluated high-dimensional datasets, NSTA shifts the accuracy–efficiency Pareto frontier toward lower forecasting error.

Primary area:
learning on time series and dynamical systems
Keywords:
Multivariate Time Series Forecasting, Transformer, Forecasting

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mGLPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Native Sparse Time Attention (NSTA), a three-branch attention mechanism for high-dimensional multivariate time-series forecasting. NSTA combines channel-wise temporal attention, compressed cross-channel representations, and sparse token selection through shared hierarchical routing. With a fixed sparsity budget, its computational complexity scales linearly with the number of channels. Experiments using Timer-XL demonstrate substantial inference speedups and improved accuracy-latency tradeoffs, particularly on high-dimensional datasets.

Critical strengths
  1. NSTA introduces a well-motivated three-branch architecture that combines efficient global cross-channel information aggregation with sparse direct channel interactions.

  2. The shared hierarchical routing mechanism achieves linear complexity with respect to the number of channels while preserving access to selected channel-specific information.

  3. The experiments demonstrate significant practical speedups, reaching up to 23× faster inference, along with favorable accuracy-latency trade-offs. The ablation studies also support the effectiveness of the proposed components.

Critical weaknesses
  1. The channel-wise temporal attention branch still has quadratic complexity with respect to the number of patches, potentially limiting efficiency when both the context length and channel count are large.

  2. The experimental comparisons are limited. The paper lacks direct comparisons with recent high-dimensional forecasting methods such as STHD, ContextRNN, U-Cast, and Unicorn. Additionally, GPU memory consumption is not reported, making it difficult to fully assess practical efficiency.

  3. The shared routing mechanism may fail to capture channel-specific dependencies, while the discrete Top-(k) selection could affect training stability. The paper provides limited analysis of these potential limitations and the sensitivity to sparsity budgets.

Questions for authors
  1. How sensitive is NSTA to the routing prior coefficient (lambda)? Does removing this term cause training instability or premature convergence of routing decisions?

  2. Can the authors report peak GPU memory consumption and the actual runtime overhead of Top-(k) selection and Gather operations, particularly in high-dimensional settings?

  3. How does the shared routing mechanism handle channels with distinct cross-channel dependencies? Would partially channel-specific routing improve performance?

  4. How does NSTA compare with recent efficient high-dimensional forecasting methods, particularly U-Cast, Unicorn, STHD, and ContextRNN, under comparable experimental settings?

2 comments
AuthorsOpenReview ↗

Response to Reviewer mGLP (1/2)

We sincerely thank the reviewer for the detailed review and the concrete questions, which led to several additions to the paper.

W1: The channel-wise temporal attention branch still has quadratic complexity with respect to the number of patches, potentially limiting efficiency when both the context length and channel count are large.

Answer: You are right, and Proposition 1 states this P² term explicitly. We would only add that it is not the bottleneck in the regime we target:

  • Every patch-based Transformer pays the P2P^2 term, including channel-independent PatchTST (Table 2). What NSTA removes is the C2C^2 factor.
  • Patching keeps PP small (P=84P = 84 at our longest context P=84P=84).
  • This setting, with both a long context and a high channel count, is trained and reported (Figure 6, Table 10).

W2: The experimental comparisons are limited. [...] Additionally, GPU memory consumption is not reported, making it difficult to fully assess practical efficiency.

Answer: Thank you for pointing out both gaps. Memory is addressed in Q2 and baselines in Q4.

W3: The shared routing mechanism may fail to capture channel-specific dependencies, while the discrete Top-k selection could affect training stability. The paper provides limited analysis of these potential limitations and the sensitivity to sparsity budgets.

Answer: These are fair concerns. Shared routing is discussed in Q3 and training stability in Q1. Budget sensitivity is analysed in Appendix C.5:

  • Traffic: MSE stays within 0.345–0.348 over the kck_c sweep (Figure 16).
  • GTWSF-Temp: MSE improves steadily with kck_c (Figure 15).
  • Across datasets: Table 4 uses one untuned budget (kc=32k_c=32) on five datasets, retaining from 100% down to 3.7% of the channels (Table 3).
AuthorsOpenReview ↗

Response to Reviewer mGLP (2/2)

Q1: How sensitive is NSTA to the routing prior coefficient (lambda)? Does removing this term cause training instability or premature convergence of routing decisions?

A1: Removing the prior costs accuracy, not stability.

  • No tuning: λ\lambda is learned (Section 3.4), so it adds no hyperparameter.
  • Ablation: Table 9 removes the prior on Traffic at five budgets (kc=32k_c=32 to 512512). MSE rises by 1.3–3.0% at every budget and seeds vary slightly more, with no instability or divergence.
  • Why training is unaffected: without the prior, the channel scorer is still trained through the compression branch (Eqs. 1–2), and top-k is never differentiated through.
  • Routing does not collapse: with the prior, ranked selection beats random selection of the same size (Table 8), and the selected patches and channels still vary across test windows (Figures 12–14).
  • Stability overall: NSTA uses Timer-XL's optimizer and learning rate unchanged (Table 6), and its seed-to-seed standard deviation is no larger than the dense baseline's (Table 10: 0.0020 vs. 0.0032 on Traffic, 0.034 vs. 0.048 on GTWSF-Temp).

Q2: Can the authors report peak GPU memory consumption and the actual runtime overhead of Top-k selection and Gather operations, particularly in high-dimensional settings?

A2: Thank you for this suggestion. Peak memory is 21× lower at C=4096C=4096, and the selection overhead is small and already included in every reported latency.

  • Memory: Figure 5 (left) covers C=256C=256 to 40964096 for a single attention layer. Dense attention grows quadratically and NSTA linearly.
  • Overhead: at high channel counts, top-k takes under 1% of an attention layer's latency and the gathers about 5%.
  • Accounting: our FLOP counts exclude both (Appendix A.2), but all latencies time the full forward pass (Appendix B.4), so the speedups include them.
  • Where it shows: at C=256C=256, where kc=Ck_c=C and nothing is pruned, NSTA is slightly slower than dense attention (Section 4.1).

Q3: How does the shared routing mechanism handle channels with distinct cross-channel dependencies? Would partially channel-specific routing improve performance?

A3: This is an important question, and the honest answer is that sharing has a measurable cost. Only the candidate set is shared, and only within one head, but a channel whose dependencies differ from the others is served less well.

  • Query-specific weights: each query computes its own attention over the shared candidates (Eq. 5) and keeps its own univariate and compression paths.
  • Per-head routing: routing is independent per head and layer, so a channel sees up to H=8H=8 candidate sets per layer. On GTWSF-Temp, heads split into about three groups with complementary station subsets (Appendix C.4, Figures 11–12).
  • Measured loss: the shared set contains a query's own top-1 token for 55% (Traffic) and 21% (GTWSF-Temp) of queries, over 6× the random baseline. At the matched budget, recall is 26% and 10%, about 3× random (Section 4.4, Figure 9). Lower-ranked preferences are not covered, as we state there.
  • Remedy today: a larger kck_c recovers accuracy at linear cost: 7.120 to 6.887 MSE on GTWSF-Temp for kc=256k_c=256 to 10241024 (Table 10).
  • Extension: we agree that partially channel-specific routing is a natural next step. Routing per cluster of channels would stay linear in CC for a fixed number of clusters; we have not evaluated it and leave it to future work.

Q4: How does NSTA compare with recent efficient high-dimensional forecasting methods, particularly U-Cast, Unicorn, STHD, and ContextRNN, under comparable experimental settings?

A4: Thank you for these references. Unfortunately, we could not train these four methods under our matched protocol (three seeds, lookback sweep, same latency measurement) within the rebuttal period. We added SOFTS and DUET instead.

  • Coverage now: the Pareto comparison includes channel-independent (PatchTST), compressed or intermediate (UniTST, SOFTS, DUET), dense (Timer-XL) and sparse (NSTA) designs.
  • U-Cast, Unicorn: both pass cross-channel information through learned intermediates (hierarchical latent queries; a latent prototype codebook), the family of UniTST, SOFTS and DUET. Unicorn also targets multi-dataset pretraining, which we list as future work.
  • Scope of our claim: we do not claim superiority over these methods. With the backbone fixed and only the attention operator changed, NSTA matches dense attention at 2–3× lower latency, and the frontier claim is limited to the evaluated models.

Presentation. We are sorry that the paper was difficult to follow, and we would like to improve its clarity. If you could point us to the passages that were hardest to read, we will gladly revise them.

We hope these answers address your concerns, and we would be glad to discuss any point further.

Reviewer_FjRTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Native Sparse Time Attention, a replacement for Timer-XL's dense patch-channel attention. It combines exact same-channel temporal attention, compressed cross-sectional attention, and direct attention to selected original channel tokens. Historical patches and channels are routed through candidate sets shared across the channel queries of a patch; final attention weights remain query-specific. This gives linear channel scaling for fixed temporal and selection budgets. Experiments assess fixed-budget accuracy, compiled runtime, accuracy-latency frontiers on Traffic and GTWSF-Temp, and three-seed branch, ranking, and routing-prior ablations.

Critical strengths
  • The paper studies an important problem in time series forecasting.
  • The efficiency result is supported by a credible matched implementation comparison.
  • The analysis directly tests whether learned selection contributes.
Critical weaknesses
  • The architectural novelty is a specialization of an existing template. Section 5 explicitly inherits Native Sparse Attention's compressed, selected, and local branch decomposition. The new contribution is primarily its adaptation to the patch-channel grid and shared candidate routing. That specialization is useful, but the framing should identify it precisely rather than imply a broadly new sparse-attention principle.

  • The comparison does not establish superiority across efficient-attention designs. The Pareto study compares PatchTST, UniTST, Timer-XL, and NSTA, but does not isolate the three-branch design against other equally budgeted routing or linear-attention operators within the same backbone. This limits the breadth of the significance claim. The demonstrated frontier should be scoped to the evaluated alternatives; broader architectural comparisons are a direction for further evaluation.

  • Shared routing restricts target-specific direct retrieval. Equations 3–4 give channel queries at a patch the same candidate set, with channel ranking determined by the candidate patch's pooling weights rather than the individual target query. Consequently, the selected-token branch cannot directly retrieve a channel excluded from that set, even if it is important to a particular target. The authors should state this representational restriction explicitly; preserved direct access to retained tokens is not preserved direct access to every potentially relevant token.

  • The diagnostics do not establish when the shared-routing assumption is adequate. PCA spectra and branch-contribution statistics do not determine whether different target channels need different candidates. The paper motivates shared dependencies but does not characterize the boundary at which heterogeneous dependencies make shared routing inadequate. This is a limitation of the explanatory evidence, not a demonstrated inconsistency in the complexity analysis. The interpretation of those diagnostics should be narrowed.

  • The strongest efficiency conclusions have limited empirical scope. The high-dimensional Pareto studies cover two datasets at one focal horizon each, while fixed-budget errors increase on ECL and Traffic in Table 4. These findings support a useful operating regime rather than uniform accuracy gains. The paper should foreground that tradeoff and distinguish the matched-latency or matched-accuracy result from a claim of improvement at every fixed budget.

Questions for authors
  1. What is the smallest architectural change from the cited Native Sparse Attention template that is necessary for the reported advantage? Is any existing matched comparison available beyond the three-branch instantiation?

  2. Can a target query retrieve a channel outside the shared top-ranked set in that layer? Please state precisely which class of target-specific interactions is excluded by shared routing.

  3. Which validation results determined the Pareto-study budgets and model settings, and do existing seed-level results support the closest frontier comparisons? Please distinguish configuration selection from subsequent test reporting; no new benchmark is requested.

1 comment
AuthorsOpenReview ↗

Response to Reviewer FjRT

We sincerely thank the reviewer for the thorough and constructive review. Your comments helped us state the contribution and its limits more precisely.

Q1: What is the smallest architectural change from the cited Native Sparse Attention template that is necessary for the reported advantage? Is any existing matched comparison available beyond the three-branch instantiation?

A1: We agree that NSTA is a specialization of the NSA template rather than a new sparse-attention principle, and we revised the framing accordingly: the first contribution now describes NSTA as an adaptation of NSA to the patch–channel grid, and Section 5 states precisely what differs.

The smallest necessary change is what the selection is shared across. In NSA, selection stays specific to each query token and is shared only across the heads of a GQA group. In NSTA, the candidate set is shared across all channel queries of a patch. This sharing is what makes the cost linear in the number of channels; the remaining differences only adapt the compression and local branches to the patch–channel grid.

You are right that we have no matched comparison against other routing or linear-attention operators inside this backbone. We therefore scoped the frontier claim to the evaluated models.

Q2: Can a target query retrieve a channel outside the shared top-ranked set in that layer? Please state precisely which class of target-specific interactions is excluded by shared routing.

A2: No, and thank you for asking us to state this precisely. Within the selection branch of a given head and layer, a query cannot attend directly to a token of another channel outside the shared set. The excluded class is exactly the target-specific direct cross-channel interactions outside that set. As you note, preserved direct access to retained tokens is not preserved direct access to every relevant token. This restriction is now stated explicitly in Sections 3.4 and 5.

Three paths partly compensate for it:

  • Selection is computed independently per head and per layer, so each layer provides 8 candidate sets. These sets are complementary on GTWSF-Temp and largely overlap in deeper layers on Traffic (Figures 10–11).
  • Every channel enters the pooled summary of the compression branch (Eq. 1), so an excluded channel stays reachable in compressed form.
  • Each query keeps exact access to its own channel through the univariate branch.

We also agree that PCA spectra and branch contributions do not show when shared routing is adequate, and Section 4.4 now says explicitly that this analysis is descriptive. To measure the mismatch directly, we added a recall analysis (Section 4.4, Figure 9):

TrafficGTWSF-Tempvs. random set
Query's top-1 token is in the shared set55%21%> 6×
Recall of the query's top-512 (matched budget)26%10%≈ 3×

The shared set is far better than chance, but sharing has a real cost, especially on GTWSF-Temp.

Q3: Which validation results determined the Pareto-study budgets and model settings, and do existing seed-level results support the closest frontier comparisons? Please distinguish configuration selection from subsequent test reporting; no new benchmark is requested.

A3: Thank you for raising this, as the original text did not separate selection from reporting clearly enough.

Configuration selection. No validation or test result determined the Pareto budgets. The budget k_p = 2, k_c = 256 was fixed before the runs as a round power-of-two value and is used on both datasets at every lookback. Model and optimization settings are Timer-XL's per-dataset configurations, without modification (Table 6). The k_c = 1024 point on GTWSF-Temp is reported as one additional higher-compute configuration to illustrate the trade-off at a larger budget. Section 4.3 now states all of this.

Seed-level support. Table 10 reports mean ± standard deviation over three seeds for the key frontier points:

DatasetConfigurationMSELatency
TrafficTimer-XL, L = 6720.345 ± 0.0032218 ms
TrafficNSTA, L = 6720.345 ± 0.002066 ms
TrafficNSTA, L = 80640.339 ± 0.0019778 ms
GTWSF-TempTimer-XL, L = 487.157 ± 0.048423 ms
GTWSF-TempNSTA, L = 96, k_c = 2567.120 ± 0.034215 ms
GTWSF-TempNSTA, L = 96, k_c = 10246.887 ± 0.005377 ms

The two lower-error points (L = 8064 on Traffic, k_c = 1024 on GTWSF-Temp) are separated from the dense baseline by more than the seed variation.

On empirical scope. We also agree with your remark that the results support a useful operating regime rather than uniform accuracy gains, and the abstract and conclusion now condition the claims on a sufficient selection budget and on the two evaluated high-dimensional datasets.

We hope these clarifications address your concerns, and we would be glad to discuss any point further.

Reviewer_dk4COpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Native Sparse Time Attention (NSTA), a three-branch replacement for dense patch-channel attention: per-channel temporal attention, attention to pooled cross-sectional summaries, and direct attention to a hierarchically selected set of patch-channel tokens. The candidate set is shared across channel queries, giving attention-score complexity linear in the channel count for fixed budgets. Experiments in Timer-XL cover standard datasets and GTWSF-Temp (3,850 channels), with three-seed accuracy results and H100 timing. The paper reports matched accuracy at lower latency on Traffic and GTWSF-Temp, and uses the savings to evaluate longer contexts.

Critical strengths

1.The paper targets a practical gap between channel-independent/compressed interaction and dense cross-channel attention. The three paths have clear, complementary roles, and sharing routing across channel queries is a simple way to preserve direct token access while reducing the attention-score count.

2.The core comparison is well controlled: NSTA replaces Timer-XL's attention operator while retaining the backbone and training configuration. The evaluation includes multiple channel scales, a genuinely high-dimensional dataset, three-seed forecasting results, and both synthetic scaling and trained-model latency measurements.

Critical weaknesses
  1. The selection policy is intentionally shared across all channel queries: patch scores are averaged across queries, and the top channels are chosen from a scalar pooling weight. This may exclude a channel that is useful to a particular query even when that query's final attention could rank it highly. Ranked-versus-random selection shows that the router carries signal, but does not establish how much query-specific useful information is lost or how robust the shared ranking is across datasets.

  2. The strongest high-dimensional accuracy/latency evidence is concentrated on Traffic and GTWSF-Temp and within Timer-XL. The efficiency comparisons do not include a matched evaluation of several efficient cross-channel alternatives discussed in the paper. This limits the evidence for NSTA as a broadly preferable sparse interaction design, rather than a useful drop-in replacement for dense Timer-XL attention.

Questions for authors

1.Could you reconcile Eq. 7 with Appendix A and Table 2, and state explicitly whether the reported complexity counts all heads and all channel queries in the selection branch?

2.Can you quantify candidate recall or compare the shared router with a query-specific or otherwise stronger sparse router at matched compute? In particular, how often would a query-specific scoring rule select useful channels omitted by the shared top-k_c set?

3.How do the accuracy/latency points compare under a matched protocol with efficient multivariate alternatives such as Crossformer, SOFTS, or DUET? If such comparisons are out of scope, please narrow the broader efficiency positioning accordingly.

1 comment
AuthorsOpenReview ↗

Response to Reviewer dk4C

We thank the reviewer for the careful reading and constructive questions.

Q1: Could you reconcile Eq. 7 with Appendix A and Table 2, and state explicitly whether the reported complexity counts all heads and all channel queries in the selection branch?

A1: Thank you for catching this inconsistency. Eq. 7, Appendix A and Table 2 now all state the same bound, $O\left( HCP\left(P+\tilde{k}_p\tilde{k}_c\right) \right)$, and the derivation in Appendix A explicitly counts all H heads and all PC channel queries in the selection branch. The scaling remains linear in the number of channels C for fixed budgets.

Q2: Can you quantify candidate recall or compare the shared router with a query-specific or otherwise stronger sparse router at matched compute? In particular, how often would a query-specific scoring rule select useful channels omitted by the shared top-k_c set?

A2: This is a fair concern, and we agree the original submission did not quantify it. Following your suggestion, we added a recall analysis to Section 4.4 (Figure 9) that compares the shared candidate set with a query-specific rule at the same budget.

Setup: Using the trained models (k_p = 2, k_c = 256, so |S_p| = 512), we score every visible token for each query with the selection logits of Eq. 5 and take the query’s own top-n set. We then report the fraction of that set contained in the shared set, averaged over queries, heads, layers and test windows, and compare it with a random set of the same size.

TrafficGTWSF-Tempvs. random set
Query's top-1 token is in the shared set55%21%> 6×
Recall of the query's top-512 (matched budget)26%10%≈ 3×

At matched budget, a query-specific rule would pick 74% (Traffic) and 90% (GTWSF-Temp) of its tokens from outside the shared set. The shared router is far better than chance and agrees most with each query on its highest-ranked tokens, but recall falls as n grows and is clearly lower on GTWSF-Temp. Your intuition is therefore right that sharing has a real cost, and we now state this openly in the paper as a limitation of the design.

Q3: How do the accuracy/latency points compare under a matched protocol with efficient multivariate alternatives such as Crossformer, SOFTS, or DUET? If such comparisons are out of scope, please narrow the broader efficiency positioning accordingly.

A3: We agree these comparisons were missing, and we are grateful for the suggestion, as they make the positioning of NSTA much clearer. We added SOFTS and DUET to both latency–accuracy figures (Traffic, GTWSF-Temp). Both were run under our matched protocol: the same lookbacks, horizon, three seeds and architecture configuration, with forward+backward latency measured at batch size 4 on an H100.

  • DUET is dominated on both datasets. At the same lookback, NSTA is 2.5–3.6× faster and has 5–13% lower MSE. For example, on Traffic at L=672, NSTA reaches 0.345 MSE in 66 ms, while DUET reaches 0.363 in 175 ms.
  • SOFTS is faster but less accurate. Its latency barely depends on the lookback (about 6 ms on Traffic and 29 ms on GTWSF-Temp), which places it at the low-latency end of both Pareto fronts. NSTA is at the low-error end of both fronts, with 4.4% lower MSE than the best SOFTS result on Traffic and 10% lower on GTWSF-Temp. SOFTS also degrades with long context (0.430 at L=8064), whereas NSTA continues to improve.

In line with your second point, we narrowed our claims accordingly: the paper now positions NSTA as defining the lower-error region of the Pareto front among the evaluated models, not as the most efficient method overall.

We hope these additions address your concerns, and we would be glad to clarify anything further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.