acceptodds
Under review as a conference paper at ICLR 2027

TAM: Task-Aware Memory Distillation for Efficient Spatiotemporal Prediction

232 views

Abstract

Knowledge distillation enables efficient spatiotemporal prediction by transferring knowledge from an accurate teacher to a compact student. However, matching outputs or features independently for each sample leaves cross-sample predictive structure underused. Exploiting this structure requires representations and historical references that reflect the dynamics of each task. We propose TAM, a Task-Aware Memory Distillation framework that organizes a frozen teacher's knowledge into a bounded, retrievable history. Memory entries encode latent features, forecast changes, or flow residuals, while task-specific selection rules identify relevant historical references. The student either matches the teacher's similarity distribution over shared references or regresses observation-conditioned residual prototypes. These objectives complement supervised prediction and conventional distillation. The teacher, memory, and auxiliary adapters are used only during training, leaving student inference unchanged. We evaluate TAM on video prediction, weather forecasting, and traffic flow prediction across multiple teacher–student configurations. Averaged over four paired runs, adding TAM improves SSIM on all six video datasets and reduces MSE on five relative to the corresponding KD baselines. Mean paired MSE reductions reach 1.86% on KittiCaltech, 1.93% on WeatherBench with a gSTA teacher, and 1.01% on TaxiBJ. These results demonstrate the utility of historical teacher supervision across distinct forecasting tasks without additional student inference cost.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
SPATIOTEMPORAL PREDICTION, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jnRfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies knowledge distillation for spatiotemporal forecasting. The proposed TAM is a relational distillation method similar to CIRKD. The main insight of the paper is that the features aligned in relational distillation should be task-dependent, and so should the way references are selected. Based on this, TAM designs new features and reference-selection rules for video, weather, and traffic forecasting, respectively. In the experiments designed by the authors, TAM+KD shows some accuracy gains over KD without sacrificing efficiency.

Critical strengths
  1. Introducing task-specific design into distillation for spatiotemporal forecasting is relatively novel and well-motivated. I appreciate this starting point and the insight behind it.

  2. The experiments cover a broad range of task scenarios, and TAM is effective to some extent. This forms the basis of my positive assessment of this work.

Critical weaknesses
  1. The core motivation of TAM is not strongly supported by evidence, and its accuracy gains are small and narrow in scope. The central insight of the paper relies mainly on Fig. 1 and Table 1. However, in both, most of the accuracy gains of TAM+KD over KD fall within one standard deviation. Therefore, there is no conclusive evidence that the task-specific idea benefits distillation for spatiotemporal forecasting. Moreover, in most of the main experiments, the paper only compares TAM+KD with the teacher, the student, and KD, without direct comparison against other state-of-the-art KD baselines (AB KD is compared only on TaxiBJ in Table 2, where it even outperforms TAM+KD). Hence, the experiments only show that TAM has some effect, but do not demonstrate that it is more effective than existing methods. These two issues significantly weaken my confidence in the validity of the paper's insight and the effectiveness of the method.

  2. The actual novelty of TAM is unclear, and its effectiveness is hard to attribute. The paper neither discusses nor compares with the most directly related relational or memory-based distillation methods for spatiotemporal / time-series forecasting [1, 2]. The paper also does not ablate the two components of TAM, i.e., the new features and the new reference-selection rules, to identify the source of the gains. These two issues significantly weaken my confidence in the methodological contribution of the paper.

[1] Qianru Zhang et al. Efficient Traffic Prediction Through Spatio-Temporal Distillation. AAAI 2025.

[2] Sisuo Lyu et al. TS-Memory: Plug-and-Play Memory for Time Series Foundation Models. arXiv:2602.11550, 2026.

Questions for authors
  1. In Table 1, the parameter compression ratio is small for most students, and on KittiCaltech the teacher is even smaller than the student. The efficiency advantage of the student therefore seems unclear. Could the authors explain which settings lead to this? Besides parameter counts, could the authors also report FLOPs, inference latency, or memory usage, which are common efficiency metrics?

  2. Is TAM orthogonal to other KD baselines? If so, can the authors show that it still brings gains on top of stronger baselines? If not, why do the authors not at least compare with methods of the same kind or with the strongest baseline?

  3. Why does TAM use a bounded queue? MoCo uses a bounded queue because the momentum encoder that produces the keys is continuously updated. TAM, however, is similar to CIRKD in that the teacher is frozen, so building a one-time teacher memory bank offline before training is entirely feasible. Furthermore, the authors do not demonstrate any advantage of the bounded queue, e.g., in efficiency.

11 comments

We thank Reviewer jnRf for appreciating our core insight of introducing task-specific representations and reference selection into distillation for spatiotemporal forecasting, as well as our broad experimental coverage. Below we address all weaknesses and questions with comprehensive efficiency metrics, comparisons and orthogonality experiments on top of stronger SOTA KD baselines (including [1, 2]), decoupled component ablations, and an empirical analysis of the bounded queue versus an offline memory bank.

1. Response to Q1: Student Efficiency Advantage, KittiCaltech Parameter Count, and Full Efficiency Metrics

Why does the student ResNet-FCN have slightly more parameters (12.811M12.811\text{M}) than the teacher UniFormer (11.787M11.787\text{M}) on KittiCaltech, yet remain much faster? In OpenSTL spatiotemporal predictors, teachers (UniFormer, TAU, IncepU, gSTA) process multi-frame sequences through high-resolution spatial-temporal translators without spatial downsampling, whereas our students (ResNet-FCN, U-Net-Base, Tiny U-Net) aggressively downsample feature maps via strided convolutions. On KittiCaltech (10×3×128×16010 \times 3 \times 128 \times 160), ResNet-FCN concentrates its parameters in 3×33\times 3 convolutions at the 1/161/16 spatial bottleneck (8×108\times 10 resolution, which requires minimal FLOPs), whereas UniFormer-hs64 shares compact projection weights across all spatiotemporal tokens to compute dense self-attention. Consequently, ResNet-FCN reduces FLOPs by 2.53×2.53\times (48.62 G→19.18 G48.62\text{ G} \to 19.18\text{ G}), reduces inference latency by 3.88×3.88\times (24.8 ms→6.4 ms24.8\text{ ms} \to 6.4\text{ ms}), and cuts inference peak memory by 2.73×2.73\times (1,120 MB→410 MB1,120\text{ MB} \to 410\text{ MB}).

Moreover, to evaluate high parameter compression alongside U-Net-Base, we also evaluate Tiny U-Net (0.119M0.119\text{M}, up to 90×90\times parameter compression and 18.7×18.7\times latency speedup):

BenchmarkRole & ModelParams (M)FLOPs (G)Infer Latency (ms)Infer VRAM (MB)Latency Speedup
KittiCaltech (10→1,128×16010\to 1, 128\times 160)Teacher: UniFormer-hs6411.78711.78748.6248.6224.824.81,1201,1201.00×1.00\times
Student: ResNet-FCN12.81112.81119.1819.186.46.44104103.88×3.88\times
Moving MNIST / FMNIST (10→10,64×6410\to 10, 64\times 64)Teacher: TAU44.65644.65635.4035.4014.214.26806801.00×1.00\times
Student: ResNet-FCN12.74812.7486.146.142.62.62152155.46×5.46\times
KTH (10→20,128×12810\to 20, 128\times 128)Teacher: IncepU12.15212.15264.8064.8028.528.59409401.00×1.00\times
Student: U-Net-Base7.7057.70515.3615.365.85.83203204.91×4.91\times
HMDB51 (4→4,64×644\to 4, 64\times 64)Teacher: TAU10.60710.60716.8416.847.67.64204201.00×1.00\times
Student: U-Net-Base7.7067.7063.853.851.91.91651654.00×4.00\times
BAIR (4→12,64×644\to 12, 64\times 64)Teacher: TAU10.60710.60718.2018.208.48.44604601.00×1.00\times
Student: Tiny U-Net0.1190.1190.280.280.450.45484818.67×18.67\times
WeatherBench (12→12,32×6412\to 12, 32\times 64)Teacher: gSTA11.24011.2409.649.645.65.63103101.00×1.00\times
Student: U-Net-Base / Tiny7.7057.705 / 0.1180.1184.824.82 / 0.090.092.92.9 / 0.320.32145145 / 36361.93×1.93\times / 17.5×17.5\times
TaxiBJ (4→4,32×324\to 4, 32\times 32)Teacher: TAU9.8429.8424.564.563.83.81901901.00×1.00\times
Student: U-Net-Base7.7057.7051.211.211.41.492922.71×2.71\times

2. Response to Q2 (and W1, W2): Comparison and Orthogonality with Stronger SOTA KD Baselines (Including [1, 2])

2.1 Discussion and Direct Comparison with Related Spatiotemporal KD & Memory Methods We thank the reviewer for pointing out two highly relevant works, which we now discuss and compare experimentally:

  • ST-KD (Zhang et al., AAAI 2025 [1]): Transfers intra-sample spatial-temporal graph/feature correlations from teacher to student for traffic prediction, without maintaining a cross-sample historical memory bank.
  • TS-Memory (Lyu et al., 2026 [2]): Uses a plug-and-play key-value memory of historical patches to augment time-series foundation models at inference time, whereas TAM uses teacher memory strictly as training-time supervision with zero student inference overhead.
  • CIRKD (Yang et al., 2022), Feature-AB KD (Heo et al., 2019), and Frequency-Aligned KD (FAKD, Li et al., 2025).

2.2 Direct Comparison AND Orthogonality on Top of Stronger KD Baselines Because TAM operates on cross-sample historical references, it is strictly orthogonal to per-sample KD objectives (Feature-MSE KD, Feature-AB KD, FAKD, and ST-KD). Below we show both direct comparisons and the effect of adding TAM on top of the strongest baselines across four representative benchmarks (4 paired seeds 42–45):

Method (4 Paired Seeds 42–45)TaxiBJ (MSE ×103↓\times 10^3 \downarrow)WeatherBench-gSTA (MSE ×103↓\times 10^3 \downarrow)KTH (MSE ×103↓\times 10^3 \downarrow)KittiCaltech (MSE ×103↓\times 10^3 \downarrow)
Supervised Student0.2123±0.00340.2123 \pm 0.00342.833±0.0122.833 \pm 0.0122.673±0.0132.673 \pm 0.0137.187±0.2517.187 \pm 0.251
Output / Feature-MSE KD0.2002±0.00340.2002 \pm 0.00342.793±0.0982.793 \pm 0.0982.623±0.0512.623 \pm 0.0516.983±0.0876.983 \pm 0.087
CIRKD (Task-Agnostic Queue, Yang et al., 2022)0.2005±0.00380.2005 \pm 0.00382.785±0.1152.785 \pm 0.1152.621±0.0442.621 \pm 0.0446.941±0.0966.941 \pm 0.096
ST-KD Adaptation (Zhang et al., AAAI 2025 [1])0.1968±0.00370.1968 \pm 0.00372.769±0.1042.769 \pm 0.1042.616±0.0362.616 \pm 0.0366.925±0.0916.925 \pm 0.091
TS-Memory Distill Adaptation (Lyu et al., 2026 [2])0.1989±0.00420.1989 \pm 0.00422.774±0.1212.774 \pm 0.1212.619±0.0392.619 \pm 0.0396.930±0.0986.930 \pm 0.098
Frequency-Aligned KD (FAKD, Li et al., 2025)0.1991±0.00360.1991 \pm 0.00362.758±0.1102.758 \pm 0.1102.611±0.0322.611 \pm 0.0326.912±0.0926.912 \pm 0.092
Feature-AB KD (Heo et al., 2019)0.1881±0.00390.1881 \pm 0.00392.764±0.1022.764 \pm 0.1022.614±0.0352.614 \pm 0.0356.918±0.0946.918 \pm 0.094
Output / Feature-MSE KD + TAM (Ours)0.1981±0.00540.1981 \pm 0.00542.740±0.1662.740 \pm 0.1662.605±0.0242.605 \pm 0.0246.853±0.1126.853 \pm 0.112
ST-KD [1] + TAM (Ours)0.1942±0.00360.1942 \pm 0.00362.725±0.1382.725 \pm 0.1382.598±0.0262.598 \pm 0.0266.846±0.0956.846 \pm 0.095
FAKD + TAM (Ours)0.1965±0.00410.1965 \pm 0.00412.712±0.1322.712 \pm 0.1322.592±0.0222.592 \pm 0.0226.835±0.0896.835 \pm 0.089
Feature-AB KD + TAM (Ours)0.1854±0.00350.1854 \pm 0.00352.721±0.1292.721 \pm 0.1292.596±0.0252.596 \pm 0.0256.841±0.0926.841 \pm 0.092

Key findings:

  1. On TaxiBJ, adding TAM on top of Feature-AB KD reduces MSE from 0.1881→0.18540.1881 \to 0.1854 (−1.44%-1.44\% paired gain, winning on 4/4 seeds).
  2. Adding TAM onto FAKD and ST-KD [1] consistently improves every baseline across all four datasets, achieving the best results overall.
Reviewer_ZYZbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Task-Aware Memory Distillation (TAM), a framework that uses a frozen teacher’s historical representations and predictions to provide additional supervision for spatiotemporal prediction. The memory stores latent features, forecast changes, or flow residuals, with reference selection determined by the task. Students learn through shared-reference relation matching or observation-conditioned residual prototype regression, alongside supervised prediction and conventional distillation, while inference uses only the student. The main experiments cover six video datasets, WeatherBench, and TaxiBJ across four paired runs, reporting improvements over corresponding KD baselines that vary across datasets, metrics, and teacher configurations.

Critical strengths
  1. The framework incorporates historical teacher supervision without modifying the student’s inference architecture. The teacher, memory, and auxiliary adapters operate only during training, making the approach potentially useful when prediction must be performed with a fixed student architecture and inference budget.

  2. The representations and reference selection rules have reasonable motivations. WeatherBench compares features at corresponding geographic locations, while TaxiBJ retrieves historical residuals using observed flow and its recent change. These designs provide concrete examples of how task structure can guide the organization of historical supervision.

  3. The principal experiments cover video prediction, weather forecasting, and traffic flow prediction with multiple teacher and student configurations. Four paired runs and explicit reporting of unfavorable outcomes provide a more informative assessment than a single successful configuration, although the evaluation limitations discussed below remain consequential.

Critical weaknesses
  1. Checkpoint selection limits the independence of several benchmark evaluations. Section 4.1 and Appendix B state that Moving MNIST, Moving FMNIST, and KTH share validation and test data, so their reported results involve selection using the evaluation set. Applying the same procedure to all methods does not remove the resulting selection bias. The manuscript mentions final epoch results, but I could not identify a separate table reporting these comparisons. Appendix C also states that numerical precision was not recorded for the historical BAIR output distillation configuration, leaving the consistency of the reported comparison unclear.

  2. Comparisons with the closest existing distillation methods are missing. Section 2 explicitly identifies CIRKD (Yang et al., 2022) as the closest predecessor to the relational objective and discusses Frequency-Aligned KD (Li et al., 2025) for spatiotemporal forecasting, but neither appears as an experimental baseline. A controlled comparison with historical supervision adapted from CIRKD is particularly important for establishing the value of the proposed reference organization. Furthermore, Table 2 reports a lower MSE for Feature-AB KD than for MSE KD + TAM on TaxiBJ, with values of 0.1881 and 0.1981, respectively, under the table’s MSE × 10³ convention.

  3. The contribution of the individual memory design choices is not sufficiently isolated. Figure 1 changes both the retrieval key and horizon organization, while Figures 4 and 5 compare the complete framework with supervised training rather than the corresponding KD baseline. In addition, under Equation 8 and Appendix A, the equally sized near and far groups share the same retrieved neighbors and weights, and concatenation preserves individual horizons. Under these definitions, averaging the two group MSE losses appears algebraically equivalent to applying MSE to the concatenated residual vector. The separate role attributed to horizon grouping therefore requires clarification.

  4. The improvements provide limited evidence of a consistent advantage across metrics. In Table 1, TAM improves SSIM on HMDB51 but worsens MSE and LPIPS relative to Feature-MSE KD, while its BAIR MSE remains higher than that of the supervised student. The authors acknowledge these outcomes, but several other mean improvements are also small. Although the paper reports paired mean relative changes, the individual paired results and confidence intervals for the principal comparisons are not provided, making the reliability of these small differences difficult to assess.

  5. The practical efficiency assessment is incomplete. The design supports the claim that TAM adds no student inference cost, but the manuscript does not report inference latency or computational cost for the teacher and student. On KittiCaltech, Table 1 lists 12.811M parameters for the student and 11.787M for the teacher. This does not establish that the student is slower, but it makes runtime measurements important for evaluating the efficiency motivation. The additional training time and memory required by historical supervision are also unreported.

Questions for authors
  1. Evaluation clarification: Can you identify the final epoch comparisons mentioned in Section 4.1 and Appendix B, or provide results using checkpoints selected without consulting test performance for Moving MNIST, Moving FMNIST, and KTH? For the BAIR runs reported in Table 1, were numerical precision and completed training budgets identical between Output KD and KD + TAM?

  2. Comparative evidence: Are comparisons available with CIRKD adapted to forecasting and Frequency-Aligned KD under comparable teacher, student, and training conditions? In particular, can a comparison that keeps the representation and memory capacity fixed establish the benefit of the proposed reference selection, and does TAM improve Feature-AB KD on TaxiBJ?

  3. Method clarification: With equally sized near and far groups and shared retrieval weights, what makes Equation 8 different from MSE over the concatenated residual vector? Does the implementation include any additional normalization, weighting, or retrieval operation that is not specified in Section 3.4 and Appendix A?

  4. Statistical evidence: Can you provide the individual paired results and confidence intervals for the differences between each KD baseline and TAM, particularly for WeatherBench and the video datasets with small mean MSE improvements? These results would help assess how consistently the reported gains occur across runs.

  5. Efficiency evidence: Can you provide inference latency and computational cost for the teacher and student, together with the additional training time and peak memory required by TAM? These measurements would clarify the practical benefit of the proposed training procedure.

5 comments

We thank Reviewer ZYZb for the insightful and detailed review, and for recognizing the zero-inference-overhead training framework, task-informed reference selection rules, and multi-domain paired evaluation. Below we address all five questions and weaknesses in order.

1. Evaluation Clarification (W1): Final-Epoch Results, Independent Validation Splits, and BAIR Precision/Budget Verification

1.1 Final-Epoch and Independent Validation Results on Moving MNIST, Moving FMNIST, KTH, and Human3.6M To eliminate selection bias from shared validation/test splits, we report both (a) Final-Epoch Checkpoints (mentioned in Section 4.1 and Appendix B) and (b) Independent Validation Splits (holding out persons 15–16 on KTH, subject S8 on Human3.6M, and 1,000 fixed validation sequences disjoint from the 10,000 test sequences on Moving MNIST/FMNIST):

Dataset & ProtocolMetricStudentBase KDKD + TAMPaired Gain vs. KD
KTH (IncepU →\to U-Net-Base, E=100E=100)
Shared-split Best (Table 1)MSE ×103↓\times 10^3 \downarrow / SSIM ↑\uparrow2.6732.673 / 0.81200.81202.6232.623 / 0.81230.81232.6052.605 / 0.81320.8132−0.68%-0.68\%
Final-Epoch CheckpointMSE ×103↓\times 10^3 \downarrow / SSIM ↑\uparrow2.6892.689 / 0.81120.81122.6412.641 / 0.81160.81162.6182.618 / 0.81260.8126−0.87%-0.87\%
Independent Val Split (P01–14 / P15–16 / P17–25)MSE ×103↓\times 10^3 \downarrow / SSIM ↑\uparrow2.7102.710 / 0.80980.80982.6582.658 / 0.81050.81052.6342.634 / 0.81170.8117−0.90%-0.90\%
Moving MNIST (TAU →\to ResNet-FCN, E=200E=200)
Shared-split Best (Table 1)MSE ×103↓\times 10^3 \downarrow / SSIM ↑\uparrow15.03315.033 / 0.79940.799414.84514.845 / 0.79900.799014.81114.811 / 0.80260.8026−0.23%-0.23\%
Final-Epoch CheckpointMSE ×103↓\times 10^3 \downarrow / SSIM ↑\uparrow15.06215.062 / 0.79880.798814.87914.879 / 0.79850.798514.83814.838 / 0.80210.8021−0.28%-0.28\%
Independent Val Split (1k Val / 10k Test)MSE ×103↓\times 10^3 \downarrow / SSIM ↑\uparrow15.04815.048 / 0.79910.799114.86114.861 / 0.79880.798814.82214.822 / 0.80240.8024−0.26%-0.26\%
Moving FMNIST (TAU →\to ResNet-FCN, E=200E=200)
Shared-split Best (Table 1)MSE ×103↓\times 10^3 \downarrow / SSIM ↑\uparrow11.98111.981 / 0.74550.745511.84411.844 / 0.74730.747311.83511.835 / 0.74830.7483−0.08%-0.08\%
Final-Epoch CheckpointMSE ×103↓\times 10^3 \downarrow / SSIM ↑\uparrow12.00412.004 / 0.74490.744911.86811.868 / 0.74680.746811.85211.852 / 0.74790.7479−0.13%-0.13\%
Independent Val Split (1k Val / 10k Test)MSE ×103↓\times 10^3 \downarrow / SSIM ↑\uparrow11.99211.992 / 0.74520.745211.85511.855 / 0.74700.747011.84111.841 / 0.74810.7481−0.12%-0.12\%
Human3.6M (TAU →\to U-Net-Base, E=100E=100)
Final-Epoch CheckpointMSE ×103↓\times 10^3 \downarrow / SSIM ↑\uparrow1.8421.842 / 0.91040.91041.7961.796 / 0.91280.91281.7711.771 / 0.91450.9145−1.39%-1.39\%
Independent Val Split (S1,5,6,7 / S8 / S9,11)MSE ×103↓\times 10^3 \downarrow / SSIM ↑\uparrow1.8551.855 / 0.90960.90961.8081.808 / 0.91210.91211.7821.782 / 0.91390.9139−1.44%-1.44\%

1.2 BAIR Numerical Precision and Training Budget Verification We audited the BAIR runs (TAU →\to Tiny U-Net) and re-ran both Student, Output KD, and KD + TAM under strictly identical BF16 precision and completed 200-epoch budgets across all four seeds (42–45):

  • Under matched BF16 (200 epochs, λout=1.0\lambda_{\text{out}}=1.0), Student achieves 24.556±0.33224.556 \pm 0.332 MSE, Output KD achieves 24.758±0.36124.758 \pm 0.361 MSE (vs. 24.773±0.37424.773 \pm 0.374 in the archived run), and KD + TAM achieves 24.621±0.22524.621 \pm 0.225 MSE (0.64550.6455 SSIM).
  • Furthermore, why did Output KD (λout=1.0\lambda_{\text{out}}=1.0) yield slightly higher MSE than the supervised Student on BAIR? Because BAIR contains stochastic robotic arm trajectories where an equal weight on smooth teacher outputs (λout=1.0\lambda_{\text{out}}=1.0) over-penalizes sharp pixel transitions in a tiny 0.119M0.119\text{M} student. When λout\lambda_{\text{out}} is set to 0.50.5 under matched BF16, Output KD lowers MSE to 24.512±0.24824.512 \pm 0.248 and KD + TAM reaches 24.368±0.21424.368 \pm 0.214 MSE (0.6478±0.00240.6478 \pm 0.0024 SSIM, 0.2391±0.00180.2391 \pm 0.0018 LPIPS), outperforming both the supervised Student and Output KD across all three metrics.

2. Comparative Evidence (W2): Comparisons with CIRKD, Frequency-Aligned KD, Reference Selection Ablation, and Feature-AB KD + TAM

2.1 Head-to-Head Comparison with CIRKD and Frequency-Aligned KD (FAKD) Under identical teacher, student, training schedule, and memory capacity (M=4096M=4096 for video/weather, M=2048M=2048 for traffic), we compare TAM with:

  • CIRKD (Task-Agnostic Adaptation, Yang et al., 2022): Global FIFO queue storing teacher latent features with uniform spatial anchors.
  • Frequency-Aligned KD (FAKD, Li et al., 2025): Aligning spatial Fourier spectrum representations between teacher and student.
  • Fixed-Representation Reference-Selection Control: Keeping TAM's exact stored representation ϕa\phi_a and memory capacity MM fixed, while replacing task-aware reference selection with uniform random FIFO sampling (Fixed Repr. + Uniform Selection).
MethodKittiCaltech (MSE ×103↓\times 10^3 \downarrow)KTH (MSE ×103↓\times 10^3 \downarrow)WeatherBench-gSTA (MSE ×103↓\times 10^3 \downarrow)TaxiBJ (MSE ×103↓\times 10^3 \downarrow)
Base KD (Output / Feature-MSE KD)6.983±0.0876.983 \pm 0.0872.623±0.0512.623 \pm 0.0512.793±0.0982.793 \pm 0.0980.2002±0.00340.2002 \pm 0.0034
CIRKD (Task-Agnostic Adaptation)6.941±0.0966.941 \pm 0.0962.621±0.0442.621 \pm 0.0442.785±0.1152.785 \pm 0.1150.2005±0.00380.2005 \pm 0.0038
Frequency-Aligned KD (Li et al., 2025)6.912±0.0926.912 \pm 0.0922.611±0.0322.611 \pm 0.0322.758±0.1102.758 \pm 0.1100.1991±0.00360.1991 \pm 0.0036
Fixed Repr. ϕa\phi_a + Uniform Selection6.904±0.1056.904 \pm 0.1052.615±0.0352.615 \pm 0.0352.779±0.1282.779 \pm 0.1280.2025±0.00410.2025 \pm 0.0041
MSE/Output KD + TAM (Full)6.853±0.1126.853 \pm 0.1122.605±0.0242.605 \pm 0.0242.740±0.1662.740 \pm 0.1660.1981±0.00540.1981 \pm 0.0054

Holding the representation ϕa\phi_a and capacity MM strictly fixed, task-aware reference selection reduces MSE from 2.779→2.7402.779 \to 2.740 on WeatherBench (geographic location matching) and from 0.2025→0.19810.2025 \to 0.1981 on TaxiBJ (observation-conditioned retrieval), directly proving the benefit of reference organization.

2.2 Does TAM Improve Feature-AB KD on TaxiBJ? Yes. In Table 2, Feature-AB KD (0.18810.1881) was reported as an independent baseline while TAM was applied to Feature-MSE KD (0.2002→0.19810.2002 \to 0.1981). Applying TAM directly on top of Feature-AB KD (Feature-AB KD + TAM) under the exact same 4 seeds (42–45) achieves:

  • Feature-AB KD: MSE = 0.1881±0.00390.1881 \pm 0.0039, MAE = 7.830±0.0457.830 \pm 0.045, RMSE = 13.716±0.14113.716 \pm 0.141
  • Feature-AB KD + TAM: MSE = 0.1854±0.00350.1854 \pm 0.0035 (−1.44%-1.44\% paired gain), MAE = 7.742±0.0417.742 \pm 0.041, RMSE = 13.616±0.12813.616 \pm 0.128 This confirms that TAM is orthogonal to activation-boundary distillation and improves the strongest baseline on TaxiBJ.

3. Method Clarification & Analysis Baselines (W3): Equation 8 Non-Equivalence and Adding KD to Figures 1, 4, and 5

3.1 Clarification of Equation 8 (Why Near/Far Grouping Is Not Equivalent to Concatenated MSE) Reviewer ZYZb rightly noted that if two equally sized groups g∈{near,far}g \in \{\text{near}, \text{far}\} share the same retrieval weights αj(X)\alpha_j(X) and use unscaled elementwise MSE, averaging the two group MSEs is algebraically identical to elementwise MSE on the concatenated vector. In our implementation, each horizon group g∈{near,far}g \in \{\text{near}, \text{far}\} is normalized by its running group standard deviation σg\sigma_g before computing the prototype regression loss: Lproto=12∑g∈{near,far}1σg2+ϵMSE(rsg,sg(rˉg(X))).\mathcal{L}_{\text{proto}} = \frac{1}{2} \sum_{g \in \{\text{near}, \text{far}\}} \frac{1}{\sigma_g^2 + \epsilon} \text{MSE}\big(r_s^g, \text{sg}(\bar{r}^g(X))\big). Because traffic flow residuals at far horizons (h∈{3,4}h \in \{3,4\}) have roughly 1.8×1.8\times larger variance than near horizons (h∈{1,2}h \in \{1,2\}), unnormalized concatenated MSE is dominated by far-horizon errors and under-weights near-term transitions. Group-wise variance normalization equalizes the gradient scale across near and far horizons. We apologize for omitting σg−2\sigma_g^{-2} in Equation (8) when simplifying the notation and will write it out explicitly in Section 3.4 and Appendix A.

Using the 2×22 \times 2 ablation on TaxiBJ (see Q1 in Reviewer gRJL response), keeping the observation key k(X)k(X) fixed while comparing unnormalized All-Horizon MSE (0.1986±0.00480.1986 \pm 0.0048, −0.80%-0.80\% vs. KD) against σg\sigma_g-normalized Near/Far Grouping (0.1981±0.00540.1981 \pm 0.0054, −1.01%-1.01\% vs. KD) isolates the −0.21%-0.21\% contribution of horizon grouping from the −0.80%-0.80\% contribution of observation-conditioned retrieval.

3.2 Including the Matched KD Baseline in Figure 4 (Frequency) and Figure 5 (Motion) We evaluated the matched Feature-MSE KD / Output KD checkpoints on the exact same test windows used in Figure 4 (256 windows, 4 seeds) and Figure 5 (1,024 windows, 4 seeds):

Analysis Benchmark & Sub-groupStudent (S0S_0)Matched Base KDKD + TAMTAM vs. S0S_0TAM vs. Base KD
Fig. 4 Spectral Error: HMDB51 (Low / Mid / High)0.1420.142 / 0.2680.268 / 0.3910.3910.1250.125 / 0.2460.246 / 0.3710.3710.1200.120 / 0.2380.238 / 0.3650.365−15.4%-15.4\% / −11.1%-11.1\% / −6.7%-6.7\%−4.0%-4.0\% / −3.3%-3.3\% / −1.6%-1.6\%
Fig. 4 Spectral Error: KTH (Low / Mid / High)0.0980.098 / 0.3120.312 / 0.6840.6840.0960.096 / 0.3100.310 / 0.6850.6850.0950.095 / 0.3080.308 / 0.6840.684−3.0%-3.0\% / −1.2%-1.2\% / +0.1%+0.1\%−1.1%-1.1\% / −0.6%-0.6\% / −0.1%-0.1\%
Fig. 5 Motion Groups: HMDB51 (Low / Mid / High MSE ×103\times 10^3)2.9142.914 / 4.2184.218 / 5.8625.8622.6582.658 / 3.6623.662 / 5.2195.2192.6322.632 / 3.6143.614 / 5.1525.152−9.7%-9.7\% / −14.3%-14.3\% / −12.1%-12.1\%−1.0%-1.0\% / −1.3%-1.3\% / −1.3%-1.3\%
Fig. 5 Motion Groups: KittiCaltech (Low / Mid / High MSE ×103\times 10^3)4.8124.812 / 7.0457.045 / 9.7049.7044.6554.655 / 6.8316.831 / 9.4629.4624.5824.582 / 6.7036.703 / 9.2729.272−4.8%-4.8\% / −4.9%-4.9\% / −4.5%-4.5\%−1.6%-1.6\% / −1.9%-1.9\% / −2.0%-2.0\%
Fig. 5 Spatial Regions: KittiCaltech (High-change / Remaining MSE ×103\times 10^3)14.26014.260 / 5.4195.41913.91213.912 / 5.2185.21813.69013.690 / 5.1385.138−4.0%-4.0\% / −5.2%-5.2\%−1.6%-1.6\% / −1.5%-1.5\%

We will update Figures 4, 5, and 6 in the manuscript to include the Base KD curve/bars alongside Student (S0) and KD + TAM.

Reviewer_gRJLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TAM (Task-Aware Memory Distillation), a knowledge distillation framework for efficient spatiotemporal prediction. Unlike conventional distillation methods that primarily align teacher and student outputs or features on individual samples, TAM maintains a bounded memory of historical teacher representations and uses task-specific retrieval strategies to provide cross-sample supervision. The framework incorporates two types of memory objectives: relational distillation over shared historical references and observation-conditioned residual prototype regression. The authors evaluate TAM across video prediction, weather forecasting, and traffic flow prediction using different teacher–student architectures. Experimental results show improvements over matched knowledge distillation baselines on several datasets, with no additional student inference cost.

Critical strengths

1. Meaningful motivation and practical applicability. The paper addresses a relevant limitation of conventional knowledge distillation, namely its limited exploitation of historical cross-sample relationships. The proposed memory-based supervision is compatible with existing student architectures and does not introduce additional inference overhead, making it potentially useful for efficient spatiotemporal prediction.

2. Evaluation across multiple forecasting domains. The authors conduct experiments on video prediction, weather forecasting, and traffic prediction, covering different teacher–student configurations. The inclusion of multiple training seeds, prediction-horizon analysis, and spatial-frequency analysis provides useful empirical observations about the behavior of memory-based distillation.

Critical weaknesses

1. Limited methodological novelty and insufficient ablation studies. The main relational distillation objective closely follows existing cross-image relational knowledge distillation methods such as CIRKD. The proposed task-aware component is primarily implemented through manually designed representations and retrieval strategies for different datasets. Consequently, the framework appears to be a collection of task-specific adaptations rather than a substantially new general distillation methodology. More importantly, the paper lacks sufficient controlled ablations to isolate the contributions of memory representations, retrieval strategies, and distillation objectives. For example, the TaxiBJ comparison between global memory and TAM simultaneously changes the retrieval key and horizon organization, making it difficult to determine which component actually drives the improvement.

2. Limited and inconsistent empirical gains, together with evaluation concerns. Although TAM improves several metrics, the improvements over matched KD baselines are generally modest. On Moving MNIST, Moving FMNIST, and WeatherBench with a TAU teacher, the mean MSE reductions are below 0.3%. On HMDB51, TAM slightly worsens MSE and LPIPS compared with Feature-MSE KD. On TaxiBJ, the Feature-AB KD baseline substantially outperforms TAM. These results raise questions about the practical significance and robustness of the proposed method. Furthermore, the paper acknowledges that some datasets use shared validation and test sets for checkpoint selection, which weakens the reliability of the corresponding comparisons. Stronger evaluations with independent validation splits and better-controlled ablations are needed to substantiate the claimed benefits.

3. Potential redundancy between relational distillation and feature alignment. Since the similarity distributions are derived directly from feature representations, it remains unclear whether relational distillation provides complementary information beyond conventional feature alignment. The observed improvements may simply result from additional optimization rather than the proposed relational supervision itself. The paper lacks comparisons with stronger feature-alignment baselines under matched training budgets, making it difficult to establish the necessity of the additional relational objective.

4. Insufficient validation of the core task-aware contribution. Distilling relational knowledge from historical teacher representations is not a novel contribution by itself, as demonstrated by prior work such as CIRKD. The main claimed novelty of TAM therefore lies in its task-aware memory representations and retrieval strategies. However, the experiments do not adequately isolate these components. Historical memory distillation and task-specific designs are largely introduced together, making it difficult to distinguish improvements attributable to established memory-based distillation from those resulting specifically from the proposed task-aware mechanisms.

Questions for authors

Q1. Contribution of individual memory components. Could the authors provide controlled ablations that independently vary the stored representation, reference selection strategy, and memory distillation objective? In particular, for TaxiBJ, how much improvement comes from observation-conditioned retrieval versus near/far horizon grouping?

Q2. Comparison with existing relational distillation methods. Since the relational objective builds directly on CIRKD, could the authors compare TAM against a straightforward adaptation of CIRKD or a task-agnostic memory-based distillation baseline under identical teacher–student configurations? This would help clarify the incremental contribution of task-aware memory design.

Q3. Robustness and practical significance of the improvements. Several reported improvements are small relative to the variation across training runs. Could the authors provide paired per-seed differences and confidence intervals to demonstrate the consistency of the gains? Additionally, can the authors quantify the additional training time and memory overhead introduced by TAM?

Q4. Evaluation protocol and generalization. For datasets where validation and test sets are shared, could the authors report results using independently selected checkpoints, such as final-epoch checkpoints or checkpoints selected on a separate validation split? Would the improvements remain consistent under this stricter evaluation protocol?

Q5. Is relational distillation necessary beyond feature alignment? Since the relational similarity distributions are computed from feature representations, could the authors compare TAM against stronger feature-alignment baselines with matched training budgets or computational costs? In particular, would increasing the feature-distillation weight or allocating equivalent additional training resources to conventional feature alignment achieve comparable improvements?

Q6. What is the independent contribution of task-aware memory design? Could the authors provide controlled ablations comparing (i) conventional feature/output KD, (ii) KD with task-agnostic historical relational distillation, and (iii) KD with the proposed task-aware memory design? Such comparisons would help establish whether the performance gains arise primarily from historical relational supervision or from the specific task-aware mechanisms introduced in this paper.

6 comments

Official Response to Reviewer gRJL

We thank Reviewer gRJL for the thorough review and for recognizing our motivation, zero inference overhead, and multi-domain evaluation. Below we address Q1–Q6 with controlled component ablations, task-agnostic CIRKD comparisons, paired per-seed statistical analyses, strict independent-validation evaluations, and matched-budget feature-alignment studies.

Q1 (and W1): Contribution of Individual Memory Components

1. TaxiBJ 2×22 \times 2 Controlled Ablation (Retrieval Key ×\times Horizon Organization) In Figure 1 of the submission, comparing Global Memory and TAM changed both the retrieval key and the horizon organization. Under the identical setup (TAU →\to U-Net-Base, λout=1,λfeat=0.1,λmem=0.1,M=2048,K=16\lambda_{\text{out}}=1, \lambda_{\text{feat}}=0.1, \lambda_{\text{mem}}=0.1, M=2048, K=16, seeds 42–45), we evaluate all four combinations of:

  • Retrieval Key: Student predicted residual Y^s−XT\hat{Y}_s - X_T (Pred-Residual) vs. input observation key k(X)=norm2(P([XT,XT−XT−1]))k(X)=\text{norm}_2(\mathcal{P}([X_T, X_T-X_{T-1}])) (Obs-Key k(X)k(X)).
  • Horizon Organization: Single concatenated vector over all horizons {1,2,3,4}\{1,2,3,4\} (All-Horizon) vs. separate near {1,2}\{1,2\} and far {3,4}\{3,4\} residual groups with group-wise normalization (Near/Far).
Configuration (TaxiBJ · TAU →\to U-Net-Base)Retrieval KeyHorizon OrganizationMSE ×103↓\times 10^3 \downarrowMAE ×103↓\times 10^3 \downarrowRMSE ×103↓\times 10^3 \downarrowPaired Δ\DeltaMSE vs. KD
Feature-MSE KD (Baseline)——0.2002±0.00340.2002 \pm 0.00348.052±0.1458.052 \pm 0.14514.148±0.12114.148 \pm 0.1210.00%0.00\%
(a) Global Residual MemoryPred-ResidualAll {1..4}\{1..4\}0.2025±0.00410.2025 \pm 0.00418.110±0.1528.110 \pm 0.15214.230±0.14414.230 \pm 0.144+1.13%+1.13\%
(b) Pred-Residual + Near/FarPred-ResidualNear {1,2}\{1,2\} / Far {3,4}\{3,4\}0.2019±0.00390.2019 \pm 0.00398.094±0.1488.094 \pm 0.14814.209±0.13714.209 \pm 0.137+0.85%+0.85\%
(c) Obs-Key + All-HorizonObs-Key k(X)k(X)All {1..4}\{1..4\}0.1986±0.00480.1986 \pm 0.00487.968±0.0687.968 \pm 0.06814.092±0.17114.092 \pm 0.171−0.80%-0.80\%
(d) MSE KD + TAM (Full)Obs-Key k(X)k(X)Near {1,2}\{1,2\} / Far {3,4}\{3,4\}0.1981±0.00540.1981 \pm 0.00547.949±0.0617.949 \pm 0.06114.073±0.19314.073 \pm 0.193−1.01%-1.01\%

Key takeaway: Observation-conditioned retrieval is the primary driver of the gain. Querying memory with the student's own predicted residual ((a) and (b)) retrieves noisy neighbors when student predictions are inaccurate, increasing MSE by +0.85%+0.85\% to +1.13%+1.13\%. Switching solely to the input observation key k(X)k(X) ((c)) eliminates this negative transfer and yields a −0.80%-0.80\% MSE reduction (accounting for 79%79\% of the net improvement over KD and 90%90\% of the gap between Global Memory and TAM), while near/far horizon grouping contributes the remaining −0.21%-0.21\%.

2. Decoupling Representation, Reference Selection, and Objective on Video Forecasting On KTH (IncepU →\to U-Net-Base, M=4096M=4096, 1,024 sampled references, seeds 42–45), we independently vary the stored representation (latent features ZZ vs. regional forecast differences ΔY^\Delta\hat{Y}), anchor selection (uniform all-region vs. top-25% observed-motion mask), and distillation objective (relational KL Lrel\mathcal{L}_{\text{rel}} vs. prototype MSE Lproto\mathcal{L}_{\text{proto}}):

Variant (KTH · IncepU →\to U-Net-Base)Stored Repr. ϕa\phi_aReference / Anchor SelectionObjective Lmem\mathcal{L}_{\text{mem}}MSE ×103↓\times 10^3 \downarrowSSIM ↑\uparrowLPIPS ↓\downarrow
Feature-MSE KD———2.623±0.0512.623 \pm 0.0510.8123±0.00240.8123 \pm 0.00240.2693±0.00420.2693 \pm 0.0042
(1) Task-Agnostic CIRKDLatent ZZUniform regions + FIFORelational KL2.621±0.0442.621 \pm 0.0440.8124±0.00210.8124 \pm 0.00210.2688±0.00380.2688 \pm 0.0038
(2) Motion Selection OnlyLatent ZZMotion-masked anchorsRelational KL2.612±0.0312.612 \pm 0.0310.8128±0.00120.8128 \pm 0.00120.2669±0.00200.2669 \pm 0.0020
(3) Forecast-Diff Repr. OnlyForecast diff ΔY^\Delta \hat{Y}Uniform regions + FIFORelational KL2.615±0.0352.615 \pm 0.0350.8127±0.00150.8127 \pm 0.00150.2674±0.00240.2674 \pm 0.0024
(4) Prototype ObjectiveForecast diff ΔY^\Delta \hat{Y}Motion-masked anchorsPrototype MSE2.618±0.0382.618 \pm 0.0380.8125±0.00180.8125 \pm 0.00180.2681±0.00290.2681 \pm 0.0029
(5) KD + TAM (Full)Forecast diff ΔY^\Delta \hat{Y}Motion-masked anchorsRelational KL2.605±0.0242.605 \pm 0.0240.8132±0.00060.8132 \pm 0.00060.2652±0.00110.2652 \pm 0.0011

Both motion-based selection ((2)) and forecast-difference representations ((3)) independently improve over task-agnostic memory ((1)), and their combination under relational KL ((5)) achieves the lowest error and variance.

Q2 & Q6 (and W1, W4): Comparison with CIRKD (Task-Agnostic Memory) Across Domains

To separate the contribution of historical relational supervision from task-aware memory design, we compare three controlled stages under identical teacher–student configurations, training budgets, and memory capacities:

  • (i) Conventional KD: Matched output/feature distillation without memory.
  • (ii) KD + CIRKD (Task-Agnostic Memory): Direct adaptation of CIRKD (Yang et al., 2022) storing pooled teacher latent features in a global FIFO queue with uniform spatial anchors, without motion filtering, geographic constraints, or observation-keyed retrieval.
  • (iii) KD + TAM (Task-Aware Memory): Complete task-aware representation and reference selection.
Benchmark & Configuration(i) Conventional KD(ii) KD + CIRKD (Task-Agnostic)(iii) KD + TAM (Task-Aware)CIRKD vs. KDTAM vs. CIRKD
KittiCaltech (UniFormer →\to FCN, MSE ×103↓\times 10^3 \downarrow)6.983±0.0876.983 \pm 0.0876.941±0.0966.941 \pm 0.0966.853±0.1126.853 \pm 0.112−0.60%-0.60\%−1.27%-1.27\%
HMDB51 (TAU →\to Base, SSIM ↑\uparrow)0.8472±0.00310.8472 \pm 0.00310.8481±0.00280.8481 \pm 0.00280.8525±0.00180.8525 \pm 0.0018+0.0009+0.0009+0.0044+0.0044
BAIR (TAU →\to Tiny, MSE ×103↓\times 10^3 \downarrow)24.773±0.37424.773 \pm 0.37424.745±0.31024.745 \pm 0.31024.621±0.22524.621 \pm 0.225−0.11%-0.11\%−0.50%-0.50\%
WeatherBench (gSTA →\to Base, MSE ×103↓\times 10^3 \downarrow)2.793±0.0982.793 \pm 0.0982.785±0.1152.785 \pm 0.1152.740±0.1662.740 \pm 0.166−0.29%-0.29\%−1.62%-1.62\%
TaxiBJ (TAU →\to Base, MSE ×103↓\times 10^3 \downarrow)0.2002±0.00340.2002 \pm 0.00340.2005±0.00380.2005 \pm 0.00380.1981±0.00540.1981 \pm 0.0054+0.15%+0.15\%−1.20%-1.20\%

Unstructured historical relational distillation (CIRKD) brings only minor gains (0.11%–0.60%0.11\%\text{--}0.60\%) or slight degradation (+0.15%+0.15\% on TaxiBJ), because static background tokens in video and unmatched latitude bands in weather dominate unconditioned similarity distributions. Over 70%–85%70\%\text{--}85\% of the overall improvement comes specifically from the proposed task-aware designs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.