acceptodds
Under review as a conference paper at ICLR 2027

ORCHESTRA: When the Best Teacher Is Not the Best Guide in Multi-Teacher Time Series Distillation

Abstract

Knowledge distillation (KD) can transfer the complementary strengths of time-series forecasting models to a compact student. However, temporal patterns differ across forecasting tasks, so the best-performing teacher may change with context, and even its guidance may be difficult for the current student to imitate. Choosing or combining multiple teachers does not by itself determine how strongly the student should learn each teacher’s temporal signals. To address these challenges, we propose ORCHESTRA, a multi-teacher distillation framework that separates learning into credit, trust, and execution. A reinforcement-learned Teacher-Credit Agent uses the Adaptive Credit Policy (ACP) to learn context-dependent teacher credit. The Multi-Scale Trust Estimator (MSTE) calibrates point, scale, and first-difference supervision based on student–teacher discrepancies, while the Sparse Execution Policy (SEP) converts learned credit into deterministic sparse distillation weights. Extensive experiments on seven forecasting datasets and four prediction lengths show that ORCHESTRA consistently improves the compact TSMixer student, achieves the best overall MSE and MAE among the evaluated direct forecasters, and retains efficient student-only deployment.

Primary area:
learning on time series and dynamical systems
Keywords:
time series forecasting, knowledge distillation, multi-teacher distillation, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DK3LOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ORCHESTRA, a multi-teacher distillation framework for time series forecasting. The framework first employs an Adaptive Credit Policy (ACP) to learn teacher credits based on teacher behavior and student–teacher compatibility, and then adopts a Multi-Scale Trust Estimator (MSTE) to calibrate the relative importance of the point, multi-scale, and first-difference distillation signals. It therefore addresses two key questions: which teachers the current student should learn from and how different temporal signals should contribute to student learning. Experiments on seven benchmark datasets demonstrate that ORCHESTRA achieves lower forecasting errors than individual teachers and other distillation strategies in most evaluated settings.

Critical strengths
  1. The perspective of adaptive teacher credit assignment and temporal signal calibration is well-motivated and interesting in the context of multi-teacher distillation. The experimental results provide empirical evidence supporting the effectiveness of the proposed framework and its individual components.
Critical weaknesses
  1. Training efficiency remains a concern. Although Figure 3 reports the inference efficiency of the distilled student model, the computational overhead of the ORCHESTRA training process remains unclear. Given that ORCHESTRA requires computing teacher and student predictions, estimating student–teacher discrepancies, and updating an online reinforcement learning policy, it is unclear whether the framework maintains an efficiency advantage over simpler distillation baselines. A comparison of training time and computational costs would help clarify its practical efficiency.

  2. Limited evaluation across student architectures. The experiments use TSMixer as the sole student architecture. Evaluating ORCHESTRA with other types of forecasting architectures would strengthen the evidence for its generalizability and applicability to different student models.

Questions for authors
  1. Why are the relative importance weights of temporal signals, (c_d), fixed at ([0.5, 0.25, 0.25]) in Equation (17)? How sensitive is the performance to these predefined weights?

  2. Why is reinforcement learning adopted instead of directly assigning teacher weights based on the calculated student–teacher discrepancies? A comparison with a simpler discrepancy-based weighting strategy would help justify the necessity of the RL-based approach.

3 comments
AuthorsOpenReview ↗

Response part 1

Thank you for your thoughtful feedback and for finding our credit assignment and temporal-signal calibration well motivated. We have carefully considered each concern and conducted additional experiments to address them. All new results and explanations have been added to the revised manuscript, and the corresponding changes are highlighted in blue. We devide the response into two parts.


W1. Training cost of ORCHESTRA

Thank you for raising this point. We profiled the complete training path on one H200 for all 28 dataset–horizon settings. After a warm-up epoch, we timed 200 batches that include data loading, teacher forwards, student updates and the scheduled policy updates. The table reports medians across settings.

MethodTeachersms/batchPeak memory, GiB
NoKD09.240.121
Single-teacher KD115.770.340
Uniform KD432.650.874
Direct Softmax432.730.874
Direct Sparsemax431.020.874
ORCHESTRA433.300.881
  • The main cost comes from the teachers, not from ORCHESTRA. Any four-teacher method must run four teacher forwards per batch, which explains the gap to NoKD and single-teacher KD. Against Uniform KD, which uses the same four teachers, ORCHESTRA adds only 2.9% median batch time and 0.007 GiB of peak memory, while lowering MSE by 4.63% and MAE by 3.28%.
  • Each step you mentioned is lightweight. The discrepancy matrix is already needed for the KD loss. The RL policy is an MLP with about 5K parameters, updated once every 50 mini-batches. Compared with the direct rules, which share everything except policy learning, it adds only 1.1–1.8%.
  • The cost is paid once. At inference only the student is deployed, so latency and memory are those of a compact TSMixer, below 1 ms p50 latency up to Solar, as shown in Figure 3.

The timing excludes the one-time validation pass that computes the accuracy prior. We have added these measurements and their analysis to Appendix S6.4.

W2. Generality across student architectures

Thank you for encouraging this evaluation. We added two students whose inductive biases differ from TSMixer: DLinear, a linear decomposition model, and a plain MLP. Each is compared with its own NoKD and Uniform KD controls. Entries are mean MSE/MAE over seven datasets and four horizons.

MethodTSMixerDLinearMLP
NoKD0.3029/0.35030.3137/0.35410.3361/0.3760
Uniform KD0.2938/0.34430.2924/0.34380.2997/0.3503
ORCHESTRA0.2802/0.33300.2840/0.33480.2903/0.3415
  • ORCHESTRA is best for every student, lowering MSE relative to Uniform KD by 4.63%, 2.87% and 3.12%, and MAE by 3.28%, 2.61% and 2.52%.
  • Following your suggestion, we also found that the distilled students become strong forecasters in their own right. DLinear with ORCHESTRA reaches 0.2840/0.3348, better than TSMixer with Uniform KD and better than every frozen teacher, the best of which is TimeMixer at 0.2968/0.3403. Even the MLP student, with about 0.05M parameters, has a lower mean MSE than every teacher.
  • Simpler students benefit more from distillation. The gain over NoKD grows from 7.49% for TSMixer to 9.47% for DLinear and 13.62% for MLP; a student with less built-in temporal structure has more to learn from the teachers.
  • TSMixer gains the most over Uniform KD, consistent with a more expressive student making better use of supervision that differs across teachers and temporal signals.

Because ORCHESTRA operates only on forecasts, it needs no matching of hidden layers and applies to each student unchanged. We have summarized this comparison in Section 4.3 and added the full results and analysis to Appendix S4.2.

AuthorsOpenReview ↗

Response part 2

Q1. Why are the signal coefficients fixed, and how sensitive are they?

The default gives half of the base weight to the point signal, since matching forecast values is the primary target, and splits the rest between scale and delta as complementary signals for trend and step-to-step movement. We keep c fixed for two reasons:

  • It is only a base allocation. The effective weight of each teacher–signal pair is w_k · c_d · Ψ_kd, and MSTE adapts Ψ within [0.5, 2] for every teacher and signal during training. The relative emphasis between signals therefore already changes with the student's state.
  • Learning c directly would be degenerate. Minimizing the KD loss over c would simply shift weight to whichever signal currently has the smallest discrepancy.

We tested four allocations over seven datasets and four horizons:

Point / Scale / DeltaMean MSEMean MAE
0.50 / 0.25 / 0.25, default0.28020.3330
1/3 / 1/3 / 1/30.28000.3329
0.25 / 0.50 / 0.250.28050.3331
0.25 / 0.25 / 0.500.27910.3321
  • Performance is insensitive to the ratio. Mean MSE changes by only −0.41% to +0.10%. The equal and scale-heavy settings stay within 0.0003 MSE of the default, the same as the seed standard deviation of ORCHESTRA. This is consistent with MSTE absorbing much of the change in base weights.
  • Emphasizing movement helps further. The delta-heavy setting is best, improving MSE by 0.41% and MAE by 0.26%. This confirms that supervising how the forecast moves between adjacent steps is valuable, which is the motivation for including the delta signal.
  • Although the default is not the best tested ratio, we did not tune c on test results and keep the default throughout the paper. The delta-heavy result therefore shows further headroom for ORCHESTRA rather than a dependence on a carefully chosen ratio.

We have added this analysis to Appendix S5.3.

Q2. Why RL instead of direct discrepancy-based weighting?

Thank you for suggesting this comparison. Direct Softmax and Direct Sparsemax map each teacher's current discrepancy straight to a weight, so a closer teacher gets a larger weight. Both keep MSTE and the same three-signal objective, so only the allocation rule differs. We also include the static prior, which assigns credit from validation accuracy alone.

VariantCredit informationMean MSEMean MAE
Static priorAccuracy only0.28480.3391
Direct SoftmaxDiscrepancy only0.28410.3352
Direct SparsemaxDiscrepancy only0.28470.3361
ORCHESTRABoth, learned by ACP0.28020.3330
  • The learned policy is consistently better. It lowers mean MSE by 1.36% and 1.56% relative to the two direct rules. Per dataset, it beats Direct Sparsemax on all seven in both metrics and Direct Softmax in MSE on six; on Solar they differ by 0.0001.
  • Neither signal is enough alone. Discrepancy tells whether the student can follow a teacher, not whether it should: two equally close teachers receive the same weight even if one is close only because it shares the student's errors. Accuracy alone ignores what the student can currently absorb. ACP combines both with inter-teacher disagreement and temporal volatility, and learns from compatibility feedback how to weigh them.
  • The benefit is inexpensive. As shown in W1, policy learning adds only 1.1–1.8% training time over the direct rules.

We have added the per-dataset comparison and its analysis to Section 4.4, and the summary above to Appendix S4.1.


We hope these responses address your concerns. We would welcome any further questions or suggestions and would be glad to discuss them.

Thank you to the authors for the detailed response and additional experiments. The new results on training efficiency, different student architectures, signal-weight sensitivity, and discrepancy-based weighting have adequately addressed my main concerns. I appreciate the authors' efforts to strengthen the experimental evaluation and am happy to increase my score in support of acceptance.

Reviewer_mSYHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies multi-teacher knowledge distillation for time-series forecasting, transferring the complementary strengths of heterogeneous teachers to a compact student. It addresses three challenges: the best-performing teacher varies across tasks, may not be the best guide for the student, and teacher-level weighting overlooks differences among temporal signals. ORCHESTRA combines a Teacher-Credit Agent with an Adaptive Credit Policy (ACP), a Multi-Scale Trust Estimator (MSTE), and a Sparse Execution Policy (SEP). Experiments on seven datasets and four prediction lengths show the best overall MSE and MAE among evaluated direct forecasters, outperforming NoKD, Uniform KD, and Best-Teacher-KD. Ablations and efficiency analyses provide further validation.

Critical strengths
  1. The paper is well motivated. It uses concrete examples to show why existing multi-teacher distillation methods can fall short, and the experimental results are encouraging.

  2. The paper considers the temporal characteristics of forecasting by separating distillation supervision into point, scale, and delta signals. This provides a clearer interpretation of what the student learns from each teacher.

Critical weaknesses
  1. The reward in ACP is based on the idea that a student more easily absorbs supervision that is close to its own predictions (Cho & Hariharan, 2019; Mirzadeh et al., 2020). However, a teacher that is easy to imitate does not necessarily provide more useful guidance, nor does it necessarily lead to higher forecasting accuracy. The paper only shows through ablation that ACP performs better than the static prior, without studying this issue in depth. In addition, the need for RL is questionable. At each update, the student–teacher discrepancies of all teachers are already computed, so the reward is immediate and fully observable. Weighting teachers directly by these discrepancies could achieve a similar allocation.

  2. The paper discusses existing multi-teacher distillation methods in its related work. However, for a paper that focuses specifically on multi-teacher distillation, the only multi-teacher baseline in the experiments is uniform weighting. It does not compare against these main approaches, such as confidence-aware weighting (Zhang et al., 2022), meta-learning-based weighting (Zhang et al., 2023), and RL-based teacher selection (Yuan et al., 2021).

Questions for authors
  1. What does RL add over weighting teachers directly by their student–teacher discrepancies? Could you report this non-RL variant?

  2. Beyond the overall MSE/MAE improvements, what evidence shows that a teacher closer to the student actually provides more useful guidance?

  3. How does ORCHESTRA perform compared with existing multi-teacher distillation methods, such as confidence-aware weighting, meta-learning-based weighting, and RL-based teacher selection?

2 comments
AuthorsOpenReview ↗

Response part 1

Thank you for your thoughtful feedback and for recognizing our motivation and the point/scale/delta design. We have carefully considered each concern and conducted additional experiments to address them. All new results and explanations have been added to the revised manuscript, and the corresponding changes are highlighted in blue. We divide the response into two parts.


W1 (first part) & Q2. Does a closer teacher provide more useful guidance?

We agree that being easy to imitate does not by itself make a teacher useful, and ORCHESTRA does not assume so. Proximity in the ACP reward measures whether the student can currently follow a teacher; whether the teacher is worth following is handled by the ground-truth loss and by the validation accuracy in ACP's state. To check that proximity is informative, we tested it directly rather than through final MSE.

On all seven datasets at H=96, at epochs 1, 5 and 10, four identical student copies each received one update with ground truth plus KD from a different teacher, under the same batch and optimizer. We computed the Spearman correlation across teachers between proximity, the negative total point/scale/delta discrepancy, and each copy's MSE reduction on a validation batch.

DatasetEpoch 1Epoch 5Epoch 10Mean
ETTh10.200.400.400.333
ETTh2-0.400.400.400.133
ETTm10.600.600.800.667
ETTm20.800.000.800.533
Weather0.800.800.800.800
ECL-0.200.20-0.20-0.067
Solar0.200.200.400.267
Mean0.2860.3710.4860.381
  • Closer teachers tend to give more useful updates. The mean correlation is 0.381; 17 of 21 checkpoints are positive, 3 negative and 1 zero, and a sign test gives p ≈ 0.003.
  • The link strengthens during training, from 0.286 to 0.486, which is when ACP is updated.
  • Proximity is informative but not sufficient. It is weak on ECL, which is why ORCHESTRA never weights teachers by proximity directly but lets ACP interpret it together with accuracy and teacher behavior.

We have added this diagnostic and its analysis to Appendix S5.2.

W1 (second part) & Q1. What does RL add over direct discrepancy weighting?

We acknowledge that the full discrepancy vector is observed at every update. We therefore tested two questions. To see whether a learned policy is needed, Direct Softmax and Direct Sparsemax, the non-RL variants you requested, map each teacher's current discrepancy straight to a weight. To see whether sampling is needed under full observability, Full-information keeps ACP but replaces the sampled update with its expectation over all four teachers' rewards. All variants keep MSTE and the three-signal objective.

DatasetORCHESTRADirect SoftmaxDirect SparsemaxFull-information
ETTh10.4321/0.45010.4379/0.45220.4386/0.45300.4376/0.4534
ETTh20.3501/0.40020.3536/0.40230.3530/0.40190.3547/0.4021
ETTm10.3546/0.38690.3575/0.38570.3600/0.39030.3594/0.3865
ETTm20.2586/0.32260.2604/0.32500.2604/0.32320.2613/0.3244
Weather0.2239/0.26310.2359/0.27480.2347/0.27440.2323/0.2633
ECL0.1553/0.25710.1565/0.25800.1586/0.25740.1683/0.2649
Solar0.1871/0.25090.1870/0.24860.1875/0.25260.1967/0.2582
Mean0.2802/0.33300.2841/0.33520.2847/0.33610.2872/0.3361
  • A learned policy beats direct weighting. Mean MSE drops by 1.36% and 1.56%; ORCHESTRA beats Direct Sparsemax on every dataset in both metrics and Direct Softmax in MSE on six of seven, with Solar differing by 0.0001. Discrepancy tells whether the student can follow a teacher, not whether it should, so two equally close teachers get the same weight even if one merely shares the student's errors. Accuracy alone is not enough either: the static prior reaches only 0.2848. ACP learns how to combine both.
  • Sampled updates beat full-information updates. Despite using every teacher's feedback, Full-information is 2.42% worse in MSE and, on average, worse than both direct rules. What is fully observed is an immediate compatibility signal, not the eventual forecasting benefit, and it drifts as the student changes. Following its exact expectation chases a moving target; sampling adjusts credit more gradually, consistent with a regularizing effect. It affects only the policy, so the student still receives deterministic sparse supervision.

We have added both comparisons to Section 4.4, with further analysis in Appendix S4.1.

AuthorsOpenReview ↗

Response part 2

W2 & Q3. Comparison with existing multi-teacher methods

We agree that comparison with established multi-teacher methods is important, and we have added one representative of each family you mentioned. These methods were designed for classification or for foundation-model teachers, and no public implementation fits our setting of heterogeneous point-forecasting teachers. We therefore re-implemented each under our regression setting, following its original weighting rule as closely as possible:

  • Confidence-aware weighting → GUARD (Dey et al., 2026). CA-MKD (Zhang et al., 2022) defines confidence through classification cross-entropy, which has no direct regression counterpart. GUARD is its closest forecasting analogue, weighting teachers by predictive uncertainty, here estimated by each teacher's training-residual RMS.
  • Meta-learning weighting → MMKD (Zhang et al., 2023), the method you cited, with MSE replacing the classification losses.
  • RL-based teacher selection → MTKD-RL (Yang et al., AAAI 2025). Like Yuan et al. (2021), it sets teacher weights with an RL policy driven by teacher–student feedback, and it is the more recent representative of this family.

All methods use the same four frozen teachers, TSMixer student, data splits, learning rate and epoch budget.

DatasetORCHESTRAGUARDMMKDMTKD-RL
ETTh10.4321/0.45010.4735/0.47600.4667/0.47150.4594/0.4674
ETTh20.3501/0.40020.3728/0.41650.3726/0.41620.3695/0.4138
ETTm10.3546/0.38690.3669/0.39470.3637/0.39190.3598/0.3887
ETTm20.2586/0.32260.2741/0.33730.2720/0.33310.2655/0.3284
Weather0.2239/0.26310.2302/0.27190.2315/0.27310.2270/0.2683
ECL0.1553/0.25710.1735/0.27720.1610/0.26290.1580/0.2585
Solar0.1871/0.25090.1942/0.25990.1940/0.26100.1886/0.2540
Mean0.2802/0.33300.2979/0.34770.2945/0.34420.2897/0.3398
  • ORCHESTRA is best on every dataset in both metrics, reducing mean MSE by 5.93%, 4.84% and 3.27% relative to GUARD, MMKD and MTKD-RL.
  • The ranking is consistent with our premise. MTKD-RL, whose policy explicitly takes student–teacher gaps as input, is the strongest baseline. GUARD, which weights teachers by their own uncertainty, is the weakest. Knowing how the student relates to each teacher appears to matter more than knowing how good each teacher is.
  • Both of our components contribute. These methods adapt how much to learn from each teacher; ORCHESTRA also calibrates how much to learn from each teacher's level, trend and movement. The ablations separate the two: under the same three-signal objective, learned credit still beats direct weighting and the static prior, and removing MSTE alone raises MSE by 1.28%.

The per-dataset comparison is in Section 4.3, and the adaptation details are in Appendix S6.1.


We hope these responses address your concerns. We would welcome any further questions or suggestions and would be glad to discuss them.

Reviewer_5258OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes ORCHESTRA, a method for distilling knowledge from multiple time-series forecasting models into a compact student model. Motivated by the observation that the most accurate teacher is not necessarily the most useful for the student, ORCHESTRA evaluates student–teacher compatibility to adaptively determine each teacher’s contribution and the temporal signals emphasized during training. Experiments on multiple time-series forecasting benchmarks show that ORCHESTRA improves forecasting performance over several alternative methods.

Critical strengths
  • Across multiple benchmarks, ORCHESTRA produces compact models that outperform the alternative methods considered in the experiments.
  • By accounting for temporal variation during distillation, the method is well suited to time-series forecasting.
Critical weaknesses
  • The use of similarity to the student’s predictions as an indicator of a teacher’s usefulness would benefit from further justification. A teacher that makes errors similar to those of the student may receive a high weight and potentially hinder learning, even if the method improves overall performance.
  • The specific benefit of policy learning is not sufficiently demonstrated. A simpler alternative could compute adaptive weights directly from student–teacher prediction discrepancies, but this comparison is not included.
  • The best teacher may vary across forecast horizons. However, Best-Teacher-KD uses the same teacher across different horizons. The current results therefore leave it unclear whether ORCHESTRA would outperform a baseline that selects the best teacher separately for each dataset and forecast horizon.
Questions for authors
  • How does ACP compare with a simpler adaptive weighting method based directly on student–teacher prediction discrepancies? This comparison would help clarify whether the additional improvement justifies the implementation and training costs of policy learning.
  • How was the teacher for Best-Teacher-KD selected? It would be helpful to clarify the data split, selection metric, and aggregation across forecast horizons.
  • What batch size, number of training epochs, and number of GPUs were used in the main experiments?
3 comments
AuthorsOpenReview ↗

Response part 1

Thank you for your thoughtful feedback. We have carefully considered each concern and conducted additional experiments to address them. All new results and explanations have been added to the revised manuscript, and the corresponding changes are highlighted in blue. We divide the response into two parts. Below, we indicate where each change can be found.


W1. Why proximity is a reasonable learning signal

We agree that this choice needs direct justification. In ORCHESTRA, proximity measures whether the student can currently follow a teacher, not whether the teacher is accurate. Accuracy enters separately: the ground-truth loss is always active, and ACP conditions its credit on each teacher's validation error. This division of roles guards against a teacher that is close only because it shares the student's errors:

  • Ground truth stays in the loop. The student is always pulled toward the true targets, and proximity is recomputed against the current student at every batch. Once the student corrects an error it shared with a teacher, that teacher becomes less proximate and receives less trust.
  • Credit does not depend on proximity alone. ACP also sees validation accuracy, inter-teacher disagreement and temporal volatility. W2 shows that weighting by proximity alone performs worse.
  • No single teacher can dominate. MSTE trust is relative and bounded in [0.5, 2], and sparse execution keeps 3.11 of 4 teachers on average.

To test the premise directly, we probed all seven datasets at H=96. At epochs 1, 5 and 10, four identical student copies each received one update with ground truth plus KD from a different teacher, under the same batch and optimizer. We report the Spearman correlation across teachers between proximity, the negative total point/scale/delta discrepancy, and each copy's MSE reduction on the same validation batch.

DatasetEpoch 1Epoch 5Epoch 10Mean
ETTh10.200.400.400.333
ETTh2-0.400.400.400.133
ETTm10.600.600.800.667
ETTm20.800.000.800.533
Weather0.800.800.800.800
ECL-0.200.20-0.20-0.067
Solar0.200.200.400.267
Mean0.2860.3710.4860.381
  • The mean correlation is 0.381, positive at 17 of 21 checkpoints and on 6 of 7 datasets.
  • It strengthens during training, from 0.286 to 0.486, the phase in which ACP is actively updated.
  • Where it is weak, as on ECL, the safeguards above take effect: ORCHESTRA still gives the best ECL student, 0.1553 MSE versus 0.1578 for Best-Teacher-KD and 0.1625 for Uniform KD.

Proximity is thus informative but not the only signal, which is exactly how ORCHESTRA uses it. We have added this diagnostic and the three safeguards above to Appendix S5.2.

W2 & Q1. Does policy learning help beyond direct discrepancy weighting?

We implemented the suggested alternative. All new runs share the paper's frozen teachers, TSMixer student and data splits; tables report four-horizon MSE/MAE. Direct Softmax and Direct Sparsemax map each teacher's current discrepancy score straight to a weight. Both keep MSTE and the same point/scale/delta objective, so the allocation rule is the only difference.

DatasetORCHESTRADirect SoftmaxDirect Sparsemax
ETTh10.4321/0.45010.4379/0.45220.4386/0.4530
ETTh20.3501/0.40020.3536/0.40230.3530/0.4019
ETTm10.3546/0.38690.3575/0.38570.3600/0.3903
ETTm20.2586/0.32260.2604/0.32500.2604/0.3232
Weather0.2239/0.26310.2359/0.27480.2347/0.2744
ECL0.1553/0.25710.1565/0.25800.1586/0.2574
Solar0.1871/0.25090.1870/0.24860.1875/0.2526
Mean0.2802/0.33300.2841/0.33520.2847/0.3361
  • ORCHESTRA has the lowest mean MSE and MAE, reducing MSE by 1.36% and 1.56%.
  • It beats Direct Sparsemax on all seven datasets in both metrics, and Direct Softmax in MSE on six of seven; Solar differs by only 0.0001.
  • The largest gap is on Weather: 5.1% lower MSE than Direct Softmax.

A direct rule sees only discrepancy, so it cannot tell a teacher that is close because it is reliable from one that shares the student's errors. ACP reads discrepancy together with accuracy, disagreement and volatility, and learns this mapping across batches. Combining this with the static-prior ablation gives a clear picture:

Credit signalMean MSE
Accuracy only, static validation prior0.2848
Proximity only, Direct Softmax0.2841
Both, learned by ACP0.2802

Neither signal suffices alone; the gain comes from learning how to combine them. We have added the per-dataset comparison and its analysis to Section 4.4, and this summary to Appendix S4.1. Training cost is reported below with Q3.

AuthorsOpenReview ↗

Response part 2

W3 & Q2. Teacher selection for Best-Teacher-KD

Original protocol. For each dataset, one teacher was fixed for all four horizons, namely the teacher with the lowest test MSE averaged over H = 96, 192, 336 and 720. Selecting on test data favors this baseline, so it serves as an optimistic reference for fixed-source distillation. ORCHESTRA still beats it on 24 of 28 settings.

New horizon-specific baseline. As suggested, we now select a teacher separately for every dataset–horizon pair, choosing the teacher with the lowest MSE on the full validation split, so no test data is used. The student is then distilled with the same objective and KD coefficient as the original single-teacher control. The selected teacher changes across horizons on 6 of 7 datasets, which confirms your intuition.

DatasetORCHESTRAHorizon-specific Best-Teacher-KD
ETTh10.4321/0.45010.4457/0.4573
ETTh20.3501/0.40020.3580/0.4051
ETTm10.3546/0.38690.3624/0.3889
ETTm20.2586/0.32260.2679/0.3307
Weather0.2239/0.26310.2239/0.2632
ECL0.1553/0.25710.1584/0.2579
Solar0.1871/0.25090.1847/0.2468
Mean0.2802/0.33300.2858/0.3357
  • ORCHESTRA lowers mean MSE by 1.95% and MAE by 0.81%. It wins on five datasets and ties on Weather.
  • Solar is the one dataset where validation picks the same teacher, PatchTST, at all four horizons. A single stable source is the most favorable case for single-teacher KD, and the gap there is still only 1.3% MSE.
  • The finer selection improves the baseline from 0.2922 to 0.2858 but does not close the gap. Even when chosen per horizon, the most accurate teacher does not guarantee the best student, which is the central claim of the paper. ORCHESTRA achieves this without any per-horizon selection step.

We have clarified the original selection protocol in Section 4.1 and added the horizon-specific comparison to Appendix S3.2, including the teacher selected for each of the 28 settings.

Q1 & Q3. Training cost and configuration

Cost of policy learning. ACP is a small 12→64→64→4 MLP updated every 50 mini-batches. On one H200, we timed 200 training batches after a warm-up epoch across all 28 settings. The timed window includes teacher forwards, student updates and policy updates.

MethodMedian ms/batchPeak memory, GiB
Uniform KD32.650.874
Direct Softmax32.730.874
Direct Sparsemax31.020.874
ORCHESTRA33.300.881

Policy learning adds a median paired overhead of only 1.1% over Direct Softmax and 1.8% over Direct Sparsemax, and only 0.007 GiB more peak memory. Inference cost is unchanged because only the student is deployed. We have added these measurements and their protocol to Appendix S6.4.

Configuration of the main experiments. Thank you for pointing out these missing details. We have added the batch size, training budget and GPU allocation to the training-configuration table in Appendix S6.1:

  • Batch size: 32 for ETTh1, ETTh2, ETTm1, ETTm2 and Weather, and 16 for ECL and Solar.
  • Epochs: at most 10, with early stopping at patience 5.
  • Optimization: learning rate 5e-5, zero weight decay, look-back window 720.
  • GPUs: each run uses a single NVIDIA A100. Independent runs were executed in parallel on four A100s.

We hope these responses address your concerns. We would welcome any further questions or suggestions and would be glad to discuss them.

Thank you for the detailed response and additional experiments. However, the theoretical justification for using student-teacher proximity remains unclear, and the supporting evidence is limited to a specific experimental setting and is not consistent across datasets. I therefore retain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.