- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies knowledge distillation for spatiotemporal forecasting. The proposed TAM is a relational distillation method similar to CIRKD. The main insight of the paper is that the features aligned in relational distillation should be task-dependent, and so should the way references are selected. Based on this, TAM designs new features and reference-selection rules for video, weather, and traffic forecasting, respectively. In the experiments designed by the authors, TAM+KD shows some accuracy gains over KD without sacrificing efficiency.
Critical strengths
-
Introducing task-specific design into distillation for spatiotemporal forecasting is relatively novel and well-motivated. I appreciate this starting point and the insight behind it.
-
The experiments cover a broad range of task scenarios, and TAM is effective to some extent. This forms the basis of my positive assessment of this work.
Critical weaknesses
-
The core motivation of TAM is not strongly supported by evidence, and its accuracy gains are small and narrow in scope. The central insight of the paper relies mainly on Fig. 1 and Table 1. However, in both, most of the accuracy gains of TAM+KD over KD fall within one standard deviation. Therefore, there is no conclusive evidence that the task-specific idea benefits distillation for spatiotemporal forecasting. Moreover, in most of the main experiments, the paper only compares TAM+KD with the teacher, the student, and KD, without direct comparison against other state-of-the-art KD baselines (AB KD is compared only on TaxiBJ in Table 2, where it even outperforms TAM+KD). Hence, the experiments only show that TAM has some effect, but do not demonstrate that it is more effective than existing methods. These two issues significantly weaken my confidence in the validity of the paper's insight and the effectiveness of the method.
-
The actual novelty of TAM is unclear, and its effectiveness is hard to attribute. The paper neither discusses nor compares with the most directly related relational or memory-based distillation methods for spatiotemporal / time-series forecasting [1, 2]. The paper also does not ablate the two components of TAM, i.e., the new features and the new reference-selection rules, to identify the source of the gains. These two issues significantly weaken my confidence in the methodological contribution of the paper.
[1] Qianru Zhang et al. Efficient Traffic Prediction Through Spatio-Temporal Distillation. AAAI 2025.
[2] Sisuo Lyu et al. TS-Memory: Plug-and-Play Memory for Time Series Foundation Models. arXiv:2602.11550, 2026.
Questions for authors
-
In Table 1, the parameter compression ratio is small for most students, and on KittiCaltech the teacher is even smaller than the student. The efficiency advantage of the student therefore seems unclear. Could the authors explain which settings lead to this? Besides parameter counts, could the authors also report FLOPs, inference latency, or memory usage, which are common efficiency metrics?
-
Is TAM orthogonal to other KD baselines? If so, can the authors show that it still brings gains on top of stronger baselines? If not, why do the authors not at least compare with methods of the same kind or with the strongest baseline?
-
Why does TAM use a bounded queue? MoCo uses a bounded queue because the momentum encoder that produces the keys is continuously updated. TAM, however, is similar to CIRKD in that the teacher is frozen, so building a one-time teacher memory bank offline before training is entirely feasible. Furthermore, the authors do not demonstrate any advantage of the bounded queue, e.g., in efficiency.
11 comments
1. Response to Q1: Student Efficiency Advantage, KittiCaltech Parameter Count, and Full Efficiency Metrics
Why does the student ResNet-FCN have slightly more parameters () than the teacher UniFormer () on KittiCaltech, yet remain much faster?
In OpenSTL spatiotemporal predictors, teachers (UniFormer, TAU, IncepU, gSTA) process multi-frame sequences through high-resolution spatial-temporal translators without spatial downsampling, whereas our students (ResNet-FCN, U-Net-Base, Tiny U-Net) aggressively downsample feature maps via strided convolutions. On KittiCaltech (), ResNet-FCN concentrates its parameters in convolutions at the spatial bottleneck ( resolution, which requires minimal FLOPs), whereas UniFormer-hs64 shares compact projection weights across all spatiotemporal tokens to compute dense self-attention. Consequently, ResNet-FCN reduces FLOPs by (), reduces inference latency by (), and cuts inference peak memory by ().
Moreover, to evaluate high parameter compression alongside U-Net-Base, we also evaluate Tiny U-Net (, up to parameter compression and latency speedup):
| Benchmark | Role & Model | Params (M) | FLOPs (G) | Infer Latency (ms) | Infer VRAM (MB) | Latency Speedup |
|---|---|---|---|---|---|---|
| KittiCaltech () | Teacher: UniFormer-hs64 | |||||
Student: ResNet-FCN | ||||||
| Moving MNIST / FMNIST () | Teacher: TAU | |||||
Student: ResNet-FCN | ||||||
| KTH () | Teacher: IncepU | |||||
Student: U-Net-Base | ||||||
| HMDB51 () | Teacher: TAU | |||||
Student: U-Net-Base | ||||||
| BAIR () | Teacher: TAU | |||||
Student: Tiny U-Net | ||||||
| WeatherBench () | Teacher: gSTA | |||||
Student: U-Net-Base / Tiny | / | / | / | / | / | |
| TaxiBJ () | Teacher: TAU | |||||
Student: U-Net-Base |
2. Response to Q2 (and W1, W2): Comparison and Orthogonality with Stronger SOTA KD Baselines (Including [1, 2])
2.1 Discussion and Direct Comparison with Related Spatiotemporal KD & Memory Methods We thank the reviewer for pointing out two highly relevant works, which we now discuss and compare experimentally:
- ST-KD (Zhang et al., AAAI 2025 [1]): Transfers intra-sample spatial-temporal graph/feature correlations from teacher to student for traffic prediction, without maintaining a cross-sample historical memory bank.
- TS-Memory (Lyu et al., 2026 [2]): Uses a plug-and-play key-value memory of historical patches to augment time-series foundation models at inference time, whereas TAM uses teacher memory strictly as training-time supervision with zero student inference overhead.
- CIRKD (Yang et al., 2022), Feature-AB KD (Heo et al., 2019), and Frequency-Aligned KD (FAKD, Li et al., 2025).
2.2 Direct Comparison AND Orthogonality on Top of Stronger KD Baselines
Because TAM operates on cross-sample historical references, it is strictly orthogonal to per-sample KD objectives (Feature-MSE KD, Feature-AB KD, FAKD, and ST-KD). Below we show both direct comparisons and the effect of adding TAM on top of the strongest baselines across four representative benchmarks (4 paired seeds 42–45):
| Method (4 Paired Seeds 42–45) | TaxiBJ (MSE ) | WeatherBench-gSTA (MSE ) | KTH (MSE ) | KittiCaltech (MSE ) |
|---|---|---|---|---|
| Supervised Student | ||||
| Output / Feature-MSE KD | ||||
| CIRKD (Task-Agnostic Queue, Yang et al., 2022) | ||||
| ST-KD Adaptation (Zhang et al., AAAI 2025 [1]) | ||||
| TS-Memory Distill Adaptation (Lyu et al., 2026 [2]) | ||||
| Frequency-Aligned KD (FAKD, Li et al., 2025) | ||||
| Feature-AB KD (Heo et al., 2019) | ||||
| Output / Feature-MSE KD + TAM (Ours) | ||||
| ST-KD [1] + TAM (Ours) | ||||
| FAKD + TAM (Ours) | ||||
| Feature-AB KD + TAM (Ours) |
Key findings:
- On TaxiBJ, adding TAM on top of
Feature-AB KDreduces MSE from ( paired gain, winning on 4/4 seeds). - Adding TAM onto
FAKDandST-KD [1]consistently improves every baseline across all four datasets, achieving the best results overall.
We thank Reviewer jnRf for appreciating our core insight of introducing task-specific representations and reference selection into distillation for spatiotemporal forecasting, as well as our broad experimental coverage. Below we address all weaknesses and questions with comprehensive efficiency metrics, comparisons and orthogonality experiments on top of stronger SOTA KD baselines (including [1, 2]), decoupled component ablations, and an empirical analysis of the bounded queue versus an offline memory bank.