- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Native Sparse Time Attention (NSTA), a three-branch attention mechanism for high-dimensional multivariate time-series forecasting. NSTA combines channel-wise temporal attention, compressed cross-channel representations, and sparse token selection through shared hierarchical routing. With a fixed sparsity budget, its computational complexity scales linearly with the number of channels. Experiments using Timer-XL demonstrate substantial inference speedups and improved accuracy-latency tradeoffs, particularly on high-dimensional datasets.
Critical strengths
-
NSTA introduces a well-motivated three-branch architecture that combines efficient global cross-channel information aggregation with sparse direct channel interactions.
-
The shared hierarchical routing mechanism achieves linear complexity with respect to the number of channels while preserving access to selected channel-specific information.
-
The experiments demonstrate significant practical speedups, reaching up to 23× faster inference, along with favorable accuracy-latency trade-offs. The ablation studies also support the effectiveness of the proposed components.
Critical weaknesses
-
The channel-wise temporal attention branch still has quadratic complexity with respect to the number of patches, potentially limiting efficiency when both the context length and channel count are large.
-
The experimental comparisons are limited. The paper lacks direct comparisons with recent high-dimensional forecasting methods such as STHD, ContextRNN, U-Cast, and Unicorn. Additionally, GPU memory consumption is not reported, making it difficult to fully assess practical efficiency.
-
The shared routing mechanism may fail to capture channel-specific dependencies, while the discrete Top-(k) selection could affect training stability. The paper provides limited analysis of these potential limitations and the sensitivity to sparsity budgets.
Questions for authors
-
How sensitive is NSTA to the routing prior coefficient (lambda)? Does removing this term cause training instability or premature convergence of routing decisions?
-
Can the authors report peak GPU memory consumption and the actual runtime overhead of Top-(k) selection and Gather operations, particularly in high-dimensional settings?
-
How does the shared routing mechanism handle channels with distinct cross-channel dependencies? Would partially channel-specific routing improve performance?
-
How does NSTA compare with recent efficient high-dimensional forecasting methods, particularly U-Cast, Unicorn, STHD, and ContextRNN, under comparable experimental settings?
2 comments
Response to Reviewer mGLP (2/2)
Q1: How sensitive is NSTA to the routing prior coefficient (lambda)? Does removing this term cause training instability or premature convergence of routing decisions?
A1: Removing the prior costs accuracy, not stability.
- No tuning: is learned (Section 3.4), so it adds no hyperparameter.
- Ablation: Table 9 removes the prior on Traffic at five budgets ( to ). MSE rises by 1.3–3.0% at every budget and seeds vary slightly more, with no instability or divergence.
- Why training is unaffected: without the prior, the channel scorer is still trained through the compression branch (Eqs. 1–2), and top-k is never differentiated through.
- Routing does not collapse: with the prior, ranked selection beats random selection of the same size (Table 8), and the selected patches and channels still vary across test windows (Figures 12–14).
- Stability overall: NSTA uses Timer-XL's optimizer and learning rate unchanged (Table 6), and its seed-to-seed standard deviation is no larger than the dense baseline's (Table 10: 0.0020 vs. 0.0032 on Traffic, 0.034 vs. 0.048 on GTWSF-Temp).
Q2: Can the authors report peak GPU memory consumption and the actual runtime overhead of Top-k selection and Gather operations, particularly in high-dimensional settings?
A2: Thank you for this suggestion. Peak memory is 21× lower at , and the selection overhead is small and already included in every reported latency.
- Memory: Figure 5 (left) covers to for a single attention layer. Dense attention grows quadratically and NSTA linearly.
- Overhead: at high channel counts, top-k takes under 1% of an attention layer's latency and the gathers about 5%.
- Accounting: our FLOP counts exclude both (Appendix A.2), but all latencies time the full forward pass (Appendix B.4), so the speedups include them.
- Where it shows: at , where and nothing is pruned, NSTA is slightly slower than dense attention (Section 4.1).
Q3: How does the shared routing mechanism handle channels with distinct cross-channel dependencies? Would partially channel-specific routing improve performance?
A3: This is an important question, and the honest answer is that sharing has a measurable cost. Only the candidate set is shared, and only within one head, but a channel whose dependencies differ from the others is served less well.
- Query-specific weights: each query computes its own attention over the shared candidates (Eq. 5) and keeps its own univariate and compression paths.
- Per-head routing: routing is independent per head and layer, so a channel sees up to candidate sets per layer. On GTWSF-Temp, heads split into about three groups with complementary station subsets (Appendix C.4, Figures 11–12).
- Measured loss: the shared set contains a query's own top-1 token for 55% (Traffic) and 21% (GTWSF-Temp) of queries, over 6× the random baseline. At the matched budget, recall is 26% and 10%, about 3× random (Section 4.4, Figure 9). Lower-ranked preferences are not covered, as we state there.
- Remedy today: a larger recovers accuracy at linear cost: 7.120 to 6.887 MSE on GTWSF-Temp for to (Table 10).
- Extension: we agree that partially channel-specific routing is a natural next step. Routing per cluster of channels would stay linear in for a fixed number of clusters; we have not evaluated it and leave it to future work.
Q4: How does NSTA compare with recent efficient high-dimensional forecasting methods, particularly U-Cast, Unicorn, STHD, and ContextRNN, under comparable experimental settings?
A4: Thank you for these references. Unfortunately, we could not train these four methods under our matched protocol (three seeds, lookback sweep, same latency measurement) within the rebuttal period. We added SOFTS and DUET instead.
- Coverage now: the Pareto comparison includes channel-independent (PatchTST), compressed or intermediate (UniTST, SOFTS, DUET), dense (Timer-XL) and sparse (NSTA) designs.
- U-Cast, Unicorn: both pass cross-channel information through learned intermediates (hierarchical latent queries; a latent prototype codebook), the family of UniTST, SOFTS and DUET. Unicorn also targets multi-dataset pretraining, which we list as future work.
- Scope of our claim: we do not claim superiority over these methods. With the backbone fixed and only the attention operator changed, NSTA matches dense attention at 2–3× lower latency, and the frontier claim is limited to the evaluated models.
Presentation. We are sorry that the paper was difficult to follow, and we would like to improve its clarity. If you could point us to the passages that were hardest to read, we will gladly revise them.
We hope these answers address your concerns, and we would be glad to discuss any point further.
Response to Reviewer mGLP (1/2)
We sincerely thank the reviewer for the detailed review and the concrete questions, which led to several additions to the paper.
Answer: You are right, and Proposition 1 states this P² term explicitly. We would only add that it is not the bottleneck in the regime we target:
Answer: Thank you for pointing out both gaps. Memory is addressed in Q2 and baselines in Q4.
Answer: These are fair concerns. Shared routing is discussed in Q3 and training stability in Q1. Budget sensitivity is analysed in Appendix C.5: