acceptodds
Withdrawn from ICLR 2027

Adaptive Quantile Response-Based Knowledge Distillation for Time-Series Foundation Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Knowledge distillation has been widely used to transfer knowledge from large foundation models to compact students, yet remains underexplored for time-series foundation models (TSFMs), particularly in multi-teacher probabilistic forecasting. To address this gap, we propose a multi-teacher response-based knowledge distillation framework that consolidates complementary forecasting knowledge from multiple pretrained TSFMs into a single compact student TSFM. Our method introduces Order-Huber Relative KD (OH-RKD) for teacher-guided cross-quantile distillation and Sample-wise Relative Adaptive Weighting (SRAW), which adaptively adjusts the distillation strength of the selected teacher according to its forecasting quality relative to the student. Extensive experiments on FEV-Bench and GIFT-Eval show that our method reduces SQL on FEV-Bench by 9.03% and WQL on GIFT-Eval by 10.96% relative to the matched GT-only student, demonstrating the effectiveness of multi-teacher distillation for TSFMs.

Primary area:
learning on time series and dynamical systems
Keywords:
Time-Series Foundation Models, Knowledge Distillation, Multi-Teacher Distillation, Probabilistic Forecasting

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_NfZBOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies an important problem of time series forecasting. The authors note that existing related work pays less attention to effectively distilling knowledge from foundation models for time series forecasting. A multi-teacher response-based knowledge distillation framework is proposed, which consolidates complementary forecasting from multiple pretrained TSFMs into a single compact student TSFM. However, knowledge distillation has been widely used in existing time series modeling methods. Experiments show the effectiveness of the proposals to some extent. However, more experiments are required.

Critical strengths
  1. The paper is well-organized and easy to follow. The structure is clear.
  2. A multi-teacher response-based knowledge distillation framework is proposed for time series forecasting, which can effectively distill knowledge from multiple teachers. Experiments show the effectiveness of the proposals to some extent.
Critical weaknesses
  1. Large language models (LLMs) are family members of foundation models. However, knowledge distillation has been widely used in LLM-based time series forecasting methods [1]. Thus, the main argument is not very interesting and exciting.

[1]. Distilling time series foundation models for efficient forecasting, ICASSP 2026.

  1. The baselines are very weak. It is suggested to include more comprehensive baselines, which may consist of statistics-based methods [2], classic deep learning based methods, and LLM-based methods. Please refer to [3, 4] for deep learning based methods and LLM-based methods.

[2]. A review of ARIMA vs. machine learning approaches for time series forecasting in data driven networks. Future Internet 2023.

[3]. TFB: Towards comprehensive and fair benchmarking of time series forecasting methods, PVLDB 2024.

[4]. Foundation models for time series analysis: A tutorial and survey, KDD 2024.

In addition, It would be better to further investigate more recent related work that is published in top venues (e.g., NeurIPS, ICLR, and ICML) to make it more comprehensive.

Questions for authors

Please see the weaknesses.

Reviewer_mPWsOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a multi-teacher response-based knowledge distillation framework for probabilistic time-series foundation models (TSFMs). The framework aims to transfer forecasting knowledge from multiple heterogeneous pretrained teachers into a compact student model. It comprises three components: (1) sample-level teacher selection, which selects the teacher with the lowest ground-truth quantile loss for each training window; (2) Order-Huber Relative Knowledge Distillation (OH-RKD), which combines corresponding-quantile alignment with teacher-guided cross-quantile constraints using a range-adaptive Huber penalty; and (3) Sample-wise Relative Adaptive Weighting (SRAW), which modulates the distillation strength according to the forecasting-loss difference between the selected teacher and the student.

The proposed framework is evaluated on FEV-Bench and GIFT-Eval using Chronos-2, TiRex-2, and Toto-2 as teachers and a Chronos-2 Small student. The reported results show improvements over ground-truth-only training and several response-based distillation baselines. In particular, the proposed method reduces SQL on FEV-Bench by 9.03% and WQL on GIFT-Eval by 10.96% relative to the matched ground-truth-only student. Ablation experiments further suggest that teacher selection, OH-RKD, and SRAW contribute to the overall performance. Nevertheless, the extent to which these gains arise from the proposed methodological innovations rather than established teacher-selection and adaptive-weighting mechanisms remains insufficiently established.

Critical strengths

1.The paper addresses an important problem: consolidating the probabilistic forecasting capabilities of heterogeneous time-series foundation models into a compact student. The response-based formulation is particularly useful because it does not require access to teacher architectures or intermediate representations. The evaluations cover 100 FEV-Bench tasks and 97 GIFT-Eval configurations, and Table 1 reports improvements across all eight aggregate metrics relative to the GT-only baseline. Figure 3 further provides dataset-balanced pairwise win rates, offering additional evidence that the gains are not exclusively driven by a small number of configurations. These results make the proposed framework practically relevant.

2.The combination of teacher selection, cross-quantile distillation, and adaptive weighting is conceptually coherent. Unlike conventional pointwise response matching, OH-RKD attempts to exploit relationships between different forecast quantiles, while SRAW explicitly accounts for the relative forecasting quality of the teacher and student. Table 2 provides controlled component substitutions, and Figures 4 and 5 offer useful descriptive analyses of teacher-selection frequencies and effective distillation weights. Although these experiments do not fully resolve the attribution concerns discussed below, they provide preliminary evidence that the complete framework is more effective than its evaluated variants.

Critical weaknesses
  1. The methodological novelty and its distinction from closely related adaptive multi-teacher distillation approaches are not sufficiently established. A central concern is that two of the three proposed components closely resemble mechanisms already explored in the knowledge-distillation literature. In Section 4.1, Eqs. (6)–(7), teacher selection is performed by choosing the teacher with the lowest ground-truth quantile loss for each training window. In Section 4.3, Eqs. (13)–(18), SRAW determines the distillation weight from the teacher–student forecasting-loss difference through a sigmoid transformation and a global schedule. These are reasonable engineering choices, but their underlying principles—instance-dependent teacher reliability and adaptive distillation strength—are not new.

More importantly, the paper itself cites Dey et al. (2026), When to Trust, How to Distill: Multi-Foundation Model Guidance for Lightweight, Robust Scientific Time Series Forecasting, which already studies instance-dependent teacher selection and uncertainty-gated distillation in the TSFM setting. Zhang et al. (2022), Confidence-Aware Multi-Teacher Knowledge Distillation, also investigates ground-truth-guided sample-wise teacher reliability. Although the present work differs in using oracle training-window selection, relative teacher–student quantile losses, and general-purpose probabilistic forecasting pretraining, the manuscript does not sufficiently explain why these distinctions constitute a substantial methodological advance rather than an adaptation of established principles.

OH-RKD is potentially the most distinctive component. However, its contribution is also not fully isolated. Eqs. (10)–(11) primarily impose one-sided constraints that place each student quantile relative to the teacher's quantiles. The paper does not establish whether these constraints provide benefits beyond conventional quantile alignment combined with a robust loss and standard quantile-ordering regularization. Therefore, the incremental novelty of OH-RKD remains unclear.

This issue is consequential because the paper's main contribution is presented as a new distillation methodology rather than solely as an empirical study of existing techniques. A more explicit comparison with the mechanisms of GUARD and related adaptive multi-teacher methods, together with targeted controls separating the novel aspects of OH-RKD, is needed to substantiate the claimed methodological contribution.

  1. The experimental evidence does not sufficiently isolate the proposed mechanisms or establish the robustness of the reported improvements.

Although the aggregate results are encouraging, several limitations prevent a clear attribution of the improvements. First, Table 2 compares OH-RKD against independent quantile-wise distillation but does not separately evaluate the effect of the Huber penalty, the range-adaptive threshold in Eq. (9), and the off-diagonal cross-quantile constraints in Eq. (10). Thus, it is unclear whether the gain comes from modeling cross-quantile relationships or simply from using a more robust teacher–student matching loss. Furthermore, the cross-quantile objective constrains student predictions relative to teacher quantiles rather than directly enforcing monotonicity among the student's own predicted quantiles. The paper provides no quantile-crossing or calibration analysis to verify the proposed probabilistic advantages.

Second, the SRAW ablation does not establish that sample-wise adaptation itself accounts for the observed gains. Table 2 compares SRAW with a global linear schedule, but the manuscript explicitly acknowledges in Section 5.3 that these strategies also differ in their effective distillation weights. Figure 5 shows a substantial reduction in the mean effective weight during early training. Without a control that matches the effective weight magnitude or schedule, the performance difference cannot be uniquely attributed to sample-wise relative weighting.

Third, the single-teacher ablation is insufficient to characterize the contribution of multi-teacher selection. The manuscript does not report separate results for each fixed teacher, making it difficult to determine whether the improvement is due to complementary teacher knowledge or to the particular fixed-teacher reference. Figure 4 reports selection frequencies but does not establish the magnitude of the performance advantage obtained from selecting different teachers.

Finally, Appendix C reports a single random seed (123), while the main results provide aggregate scores without repeated-training variability or confidence intervals for the proposed method's improvements. Given the relatively small differences among some methods and the computational scale of the experiments, the absence of such uncertainty estimates weakens confidence in their robustness. The paper also motivates compact deployment but does not provide a quantitative comparison of teacher and student parameter counts, inference costs, or offline teacher-forecast-generation costs.

Collectively, these limitations leave important uncertainty regarding which mechanisms actually drive the reported performance gains and whether the improvements are sufficiently stable and practically efficient. More carefully controlled experiments would materially strengthen the paper's conclusions.

Questions for authors

Q1. Novelty relative to prior work (clarification). What are the fundamental methodological differences between the proposed teacher-selection/SRAW mechanisms and the instance-dependent teacher routing and adaptive distillation mechanisms in Dey et al. (2026)? Beyond the quantile forecasting setting, which proposed mechanism is conceptually new, and which is an adaptation of existing approaches?

Q2. Isolating the contribution of OH-RKD (additional experiment). Could the authors compare (a) independent quantile alignment with the same range-adaptive Huber penalty, (b) OH-RKD without off-diagonal cross-quantile terms, (c) OH-RKD with a fixed Huber threshold, and (d) a standard quantile-crossing regularizer? Reporting quantile-crossing rates and a probabilistic calibration measure would also help establish whether the proposed constraints improve the quality of forecast distributions beyond the aggregate quantile losses.

Q3. Controlling for distillation-weight magnitude (additional experiment). Could the authors compare SRAW with a global schedule calibrated to match SRAW's average effective distillation weight over training? This would help distinguish the contribution of sample-wise adaptivity from the effect of simply reducing the overall strength of teacher supervision.

Q4. Establishing the benefit of multi-teacher selection (clarification/additional experiment). Which teacher is used in the single-teacher ablation in Table 2? Could the authors report results for each individual teacher, along with an oracle-selection baseline using conventional quantile alignment? This would clarify whether gains arise primarily from teacher complementarity, OH-RKD, or their interaction.

Q5. Robustness and statistical reliability (clarification/additional experiment). Are the reported results obtained from a single training seed? Could the authors provide variability across independent training runs, or otherwise quantify uncertainty in the main improvements using appropriate benchmark-level statistical analysis? This would be particularly helpful for interpreting the smaller performance differences.

Q6. Efficiency and cost (clarification). What are the total offline teacher-forecast-generation costs and the resulting student inference costs relative to the teacher models? Since computational efficiency is a central motivation for distillation, a quantitative comparison would help determine the practical value of the proposed approach.

Reviewer_VhQFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work focuses on responsive knowledge distillation for Time Series Foundation Models (TSFMs). Unlike LLM distillation, which primarily aligns discrete token distributions or textual responses, TSFM distillation requires transferring continuous probabilistic forecasts—typically represented as multiple quantiles. The authors propose a multi-teacher distillation framework that transfers predictions from Chronos-2, TiRex-2, and Toto-2 to a "Chronos-2 Small" student model. The framework consists of three components: sample-level teacher selection, Order-Huber Relative KD (OH-RKD), and Sample-wise Relative Adaptive Weighting (SRAW). On the FEV-Bench and GIFT-Eval benchmarks, this approach achieves a 9.03% reduction in SQL and a 10.96% reduction in WQL compared to a student model trained solely on ground truth (GT-only), while also outperforming other knowledge distillation baselines.

Critical strengths

1- The paper presents a clear conceptual framework by decomposing the distillation process into three distinct components: which teacher to learn from, what to learn, and to what extent to learn.

2- The sample-level teacher selection mechanism acknowledges that the optimal teacher can vary across different time windows. OH-RKD goes beyond simple quantile-to-quantile alignment by leveraging the ordinal relationships between quantiles to impose cross-quantile constraints. SRAW adjusts distillation weights based on the loss difference between the teacher and the student, preventing excessive distillation when a teacher performs significantly worse than the student. These design choices are better suited for probabilistic forecasting than standard, direct quantile alignment.

3- The experimental scale is highly compelling: the evaluation covers 100 tasks on FEV-Bench and 97 dataset-frequency-horizon configurations on GIFT-Eval, all in a zero-shot setting. The metrics include SQL, MASE, WAPE, WQL, ND, and sMAPE, supplemented by a pairwise win-rate matrix.

4- The paper provides thorough details regarding training data composition, student model architecture, SRAW hyperparameters, and the adaptation of baseline methods.

Critical weaknesses

1- There is a discrepancy between the adaptive coefficient for SRAW as defined in Eq. (16) and its representation in Figure 5.

2- Key ablation studies are missing; specifically, the individual performance contributions of the sample-level teacher selection, OH-RKD, and SRAW components are not quantified. There is no comparison with single-teacher distillation. Thus, it cannot be concluded that multi-teacher selection is necessarily superior to using a single teacher or an aggregated approach.

3- There is no explanation as to why only a single set of results is provided.

4- The analysis of teacher selection and SRAW is insufficient. The teacher selection process utilizes the quantile loss calculated against the ground truth (GT) for the future: a method that is applicable during training but not during inference. While this approach is technically valid, it tends to favor teachers that have already achieved a good fit within the given window, creating a degree of redundancy with the GT-based supervision.

5- A few typos: In Equation (10), the diagonal terms appear to fall under both indicator functions; it is advisable to clarify that these diagonal terms are derived by combining two one-sided components to form a symmetric Huber loss, ensuring readers do not mistakenly assume they are being double-counted. Additionally, there is a gap in the equation numbering between (16) and (18).

Questions for authors

refer to weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.