acceptodds
Under review as a conference paper at ICLR 2027

Efficient Zero-shot Spatio-Temporal Forecasting via Multi-scale Expert Distilled Mixed Hashing Attention

Abstract

Zero-shot spatio-temporal forecasting is crucial for applications such as smart cities and IoT, and while Large Language Models (LLMs) show promising capability in this setting, their high computational cost limits real-time deployment. Designing an efficient student model to distill knowledge from LLMs offers a promising solution, though it remains challenging. In this paper, we propose Multi-scale Expert Distilled Mixed Hashing Attention (MSED-MHA). We first employ Linear Attention to capture global spatio-temporal trends with high efficiency. In addition, we introduce Spectral Hashing Attention to model local dependencies through hashing-based Top-k retrieval over relevant neighbors. To address the intractable optimization with multiple hash constraints, we adopt a spectral learning approach to enable accurate and efficient retrieval. In addition, a selective Multi-Scale Expert Distillation guides the student across temporal scales, applying teacher supervision only when beneficial, thereby mitigating error propagation. Experiments on traffic, finance, and weather benchmarks demonstrate that MSED-MHA achieves competitive forecasting performance with high computational efficiency compared to state-of-the-art methods.

Primary area:
learning on time series and dynamical systems
Keywords:
Spatio-Temporal Forecasting, Zero-Shot Forecasting, Time Series Forecasting, Knowledge Distillation, Efficient Attention, Hashing-Based Attention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4gBsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes MSED-MHA, which is a lightweight student model for within-domain cross-dataset zero-shot spatio-temporal forecasting, distilled from a frozen GPT-2 teacher. The student model attends over spatial nodes with Mixed Hashing Attention, which is the sum of a Performer-style linear attention and a Spectral Hashing Attention (SHA). SHA picks Top-k keys by learned hash codes, and the hash projection is trained to match a spectral target B = ZW (W are leading eigenvectors of a batch-averaged matrix). The student model is multi-scale (average pooling), and distillation at each scale is turned on only when the student’s loss at that scale is above the teacher’s loss. Experiments pretrain on several source datasets of one domain (traffic, finance, weather) and test on unseen target datasets. The model gets the best average rank and is claimed to be cheaper than LLM-based baselines.

Critical strengths

Strengths 1. The problem setting is quite practical. Distilling an LLM-based forecaster into a small model for zero-shot transfer to new sensor networks is a reasonable deployment story.

Strengths 2. The authors did a lot of extra controls in the appendix (w/o-Distill, LSH replacement, exact Top-k, teacher swap, four seeds). The selective distillation idea is simple and in my understanding the w/o-Distill comparison (Table 5) gives some evidence that the teacher signal helps.

Critical weaknesses

Weakness 1. Main results look inconsistent with the multi-seed results. Appendix I reports four-seed means, and they do not match Table 1 well. On SZ_TAXI horizon 48 the MSED-MHA mean is 0.3487 ± 0.0272 but Table 1 gives 0.3092, more than one std below the mean, so I think Table 1 may report a favorable run. For baselines the gap goes the other way and is much bigger: TimeCMA on SZ_TAXI-12 is 0.2576 in Table 1 but 0.7268 ± 0.021 in Table 19, ST-LLM+ is 0.2686 vs 0.7707. Also on METR-LA (Table 20) FSCA beats MSED-MHA in MSE at every horizon (for example 0.8935 vs 1.0562 at 48). So the “best average rank” claim depends on which table you trust.

Weakness 2. The method contribution is thinner than presented. M = Zc^T Z Z^T Zc is a d x d matrix, so W is basically a PCA-like projection and B = ZW is close to PCA hashing, which is old. Proposition 2 is a triangle inequality on residuals that are never measured. The efficiency argument for hashing is also weak at these sizes: with k = floor(N/30) the PEMS08 student uses k = 5 out of 170 nodes, and exact Top-k over a few hundred nodes should be cheap. Rep-Exact is about as accurate (Table 9), so the gain is only speed, and 81s vs 101s looks more like implementation difference. In Table 3, ST-LLM+ is faster than MSED-MHA on PEMS08 (71.05s vs 81.39s), and TimeFilter with 0.13M params takes 1376s, which is odd. The conclusion says lower inference time than all evaluated LLM-based approaches, which is not true for PEMS08.

Other points. The ablation in Table 2 is mixed, w/o-Selective is better than the full model on METR-LA horizons 24 to 48 and SZ_TAXI 36/48, but the text says it degrades. No time series foundation models are used as zero-shot baselines (Chronos, Moirai, Sundial are cited, and Moirai is even used as a teacher), and no naive or historical-average baseline. On finance, RePST beats MSED-MHA on all CSI300 and HSI cells and all errors are near 1.0 after normalization, so I am not sure there is much signal there. Teacher-alone zero-shot numbers are also missing, so I can not tell if the student “surpasses its teacher” as claimed.

Questions for authors

Q1. How exactly were the Table 1 numbers obtained (which seed, which baseline configs), and why are baseline numbers in Table 1 so different from Tables 18 to 20? Can you give Table 1 with the four-seed means?

Q2. What is the zero-shot accuracy of the GPT-2 teacher alone on the target datasets, and how often is the selective gate actually active per scale during training?

Q3. Were k = N/30 and the loss weights chosen only on source validation data? Tables 12 to 15 show target-set results, and in zero-shot setting choosing by these would be leakage.

Q4. Can you compare with zero-shot foundation models (for example Chronos or Moirai applied directly) and a naive last-value baseline, at least on the traffic targets?

Reviewer_NGvhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MSED-MHA for within-domain cross-dataset zero-shot spatio-temporal forecasting. The student combines global linear attention with spectral hashing attention and processes inputs at multiple temporal scales. Selective distillation transfers predictions and hidden representations from a teacher only when the teacher outperforms the corresponding student branch.

Critical strengths

1.The motivation is good: transferring forecasting models to unseen datasets while keeping inference costs manageable. The teacher is removed at inference, and the deployed model is substantially smaller than the evaluated LLM-based baselines.

2.The paper is clearly written and easy to follow.

3.Extensive experiments are conducted.

Critical weaknesses

1.The technical contribution appears incremental. The method combines global and local attention, multi-scale pooling, and selective distillation. Each component has a reasonable purpose, but the paper does not clearly establish what new modeling insight their combination provides for zero-shot transfer. In particular, the interpretation of linear attention as a global trend smoother and hashing attention as a high-frequency local refiner is insufficiently supported: selecting relevant spatial nodes does not directly establish sensitivity to high-frequency temporal changes.

2.The accuracy gains are uneven across domains. On CSI300_2 and HSI_2, MSED-MHA ranks second but trails RePST in both MSE and MAE at every reported horizon, while its results are very close to FSCA. These are competitive results, but they do not demonstrate an accuracy advantage in finance. The main component ablations focus on traffic, leaving it unclear whether spectral hashing and selective distillation provide comparable benefits in the other domains.

3.The spectral analysis does not establish better retrieval quality. Proposition 1 characterizes the spectral optimization, while Proposition 2 conditionally bounds balance and orthogonality residuals. Neither establishes that Hamming-distance retrieval preserves query–key similarity rankings or retrieves the keys most important to attention. The random-hashing comparison provides some empirical support, but spectral hashing wins only 14 of the 24 metric–horizon comparisons. Direct retrieval measurements are needed to explain when and why the spectral objective helps.

Questions for authors

What evidence supports the proposed division between global trend smoothing and local high-frequency refinement? Can the two branches be evaluated separately on smooth periods and abrupt changes?

Do spectral hashing and selective distillation improve performance on CSI300_2 and HSI_2? Component ablations on these datasets would help explain the differences across domains.

Compared with random hashing, does spectral hashing improve recall of exact Top-k attention neighbors, retained attention mass, or attention-output approximation error? Do these improvements persist on unseen target datasets?

Reviewer_eSghOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MSED-MHA for within-domain, cross-dataset, zero-shot spatiotemporal forecasting. The method employs a lightweight student model that combines Global Linear Attention with spectral hashing-based Top-k attention. The method constructs an eigenvector-derived spectral target to guide optimization of learnable hash codes. The student uses a multi-scale architecture to process inputs at different temporal resolutions, generating scale-specific forecasts that are then averaged to produce the final prediction. Meanwhile, the authors introduce a selective knowledge distillation strategy, applying teacher prediction and representation supervision only when the student's prediction loss at a given temporal is higher than the teacher's. During inference, the teacher model is completely removed, and only the student model is used for forecasting. Experiments are conducted on six target datasets across three domains: traffic, finance, and weather. The results demonstrate competitive forecasting performance and favorable average rankings, while achieving lower inference costs than several LLM-based baselines.

Critical strengths

S1. An interesting efficiency-oriented architecture. The combination of global linear attention and local hash-based retrieval is well motivated for balancing broad spatiotemporal trends and sparse local interactions. The teacher-free inference design addresses a practically relevant deployment constraint. S2. A reasonably broad and carefully controlled empirical evaluation. The evaluation spans three domains and six unseen target datasets, with comparisons to multiple baselines. The supplementary ablations include selective versus non-selective/no distillation, spectral hashing versus LSH and exact Top-k, and four-seed traffic results. These controls provide useful evidence for the contributions of the components.

Critical weaknesses

W1. Unclear justification for the near-linear complexity of hashing attention. Although the paper demonstrates favorable inference efficiency, its complexity analysis includes an O(Nk) term, while the number of retrieved neighbors is set to k=[N⁄30]. Under this setting, the term scales quadratically rather than linearly with N. Furthermore, the computational cost of hash-based Top-k retrieval is not explained. The authors should provide a more complete complexity analysis and further validate computational scalability across different numbers of spatial nodes. W2. Insufficient direct validation of the learned hash-code quality. The proposed spectral hashing method aims to learn effective binary codes for efficient neighbor retrieval. While the paper compares with LSH and exact Top-k attention, these experiments mainly track forecasting accuracy and inference efficiency, without directly evaluating the quality of the learned hash codes. It would be convincing to report quantitative results on bit balance, orthogonality, quantization loss, and Top-k retrieval accuracy.

Questions for authors

Q1. With k=[N⁄30], what are the actual asymptotic computational complexity and measured end-to-end retrieval latency as N increases? Does the implementation use an index or other method that avoids all-pairs Hamming-distance comparisons? Please clarify how this overhead is counted in the reported inference time. Q2. How do the authors check that the spectral hashing module learns meaningful binary hash codes? Can the authors report metrics on bit balance, orthogonality, quantization loss, and retrieval recall compared to exact Top-k selection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.