- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes MSED-MHA, which is a lightweight student model for within-domain cross-dataset zero-shot spatio-temporal forecasting, distilled from a frozen GPT-2 teacher. The student model attends over spatial nodes with Mixed Hashing Attention, which is the sum of a Performer-style linear attention and a Spectral Hashing Attention (SHA). SHA picks Top-k keys by learned hash codes, and the hash projection is trained to match a spectral target B = ZW (W are leading eigenvectors of a batch-averaged matrix). The student model is multi-scale (average pooling), and distillation at each scale is turned on only when the student’s loss at that scale is above the teacher’s loss. Experiments pretrain on several source datasets of one domain (traffic, finance, weather) and test on unseen target datasets. The model gets the best average rank and is claimed to be cheaper than LLM-based baselines.
Critical strengths
Strengths 1. The problem setting is quite practical. Distilling an LLM-based forecaster into a small model for zero-shot transfer to new sensor networks is a reasonable deployment story.
Strengths 2. The authors did a lot of extra controls in the appendix (w/o-Distill, LSH replacement, exact Top-k, teacher swap, four seeds). The selective distillation idea is simple and in my understanding the w/o-Distill comparison (Table 5) gives some evidence that the teacher signal helps.
Critical weaknesses
Weakness 1. Main results look inconsistent with the multi-seed results. Appendix I reports four-seed means, and they do not match Table 1 well. On SZ_TAXI horizon 48 the MSED-MHA mean is 0.3487 ± 0.0272 but Table 1 gives 0.3092, more than one std below the mean, so I think Table 1 may report a favorable run. For baselines the gap goes the other way and is much bigger: TimeCMA on SZ_TAXI-12 is 0.2576 in Table 1 but 0.7268 ± 0.021 in Table 19, ST-LLM+ is 0.2686 vs 0.7707. Also on METR-LA (Table 20) FSCA beats MSED-MHA in MSE at every horizon (for example 0.8935 vs 1.0562 at 48). So the “best average rank” claim depends on which table you trust.
Weakness 2. The method contribution is thinner than presented. M = Zc^T Z Z^T Zc is a d x d matrix, so W is basically a PCA-like projection and B = ZW is close to PCA hashing, which is old. Proposition 2 is a triangle inequality on residuals that are never measured. The efficiency argument for hashing is also weak at these sizes: with k = floor(N/30) the PEMS08 student uses k = 5 out of 170 nodes, and exact Top-k over a few hundred nodes should be cheap. Rep-Exact is about as accurate (Table 9), so the gain is only speed, and 81s vs 101s looks more like implementation difference. In Table 3, ST-LLM+ is faster than MSED-MHA on PEMS08 (71.05s vs 81.39s), and TimeFilter with 0.13M params takes 1376s, which is odd. The conclusion says lower inference time than all evaluated LLM-based approaches, which is not true for PEMS08.
Other points. The ablation in Table 2 is mixed, w/o-Selective is better than the full model on METR-LA horizons 24 to 48 and SZ_TAXI 36/48, but the text says it degrades. No time series foundation models are used as zero-shot baselines (Chronos, Moirai, Sundial are cited, and Moirai is even used as a teacher), and no naive or historical-average baseline. On finance, RePST beats MSED-MHA on all CSI300 and HSI cells and all errors are near 1.0 after normalization, so I am not sure there is much signal there. Teacher-alone zero-shot numbers are also missing, so I can not tell if the student “surpasses its teacher” as claimed.
Questions for authors
Q1. How exactly were the Table 1 numbers obtained (which seed, which baseline configs), and why are baseline numbers in Table 1 so different from Tables 18 to 20? Can you give Table 1 with the four-seed means?
Q2. What is the zero-shot accuracy of the GPT-2 teacher alone on the target datasets, and how often is the selective gate actually active per scale during training?
Q3. Were k = N/30 and the loss weights chosen only on source validation data? Tables 12 to 15 show target-set results, and in zero-shot setting choosing by these would be leakage.
Q4. Can you compare with zero-shot foundation models (for example Chronos or Moirai applied directly) and a naive last-value baseline, at least on the traffic targets?