- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents ORCHESTRA, a multi-teacher distillation framework for time series forecasting. The framework first employs an Adaptive Credit Policy (ACP) to learn teacher credits based on teacher behavior and student–teacher compatibility, and then adopts a Multi-Scale Trust Estimator (MSTE) to calibrate the relative importance of the point, multi-scale, and first-difference distillation signals. It therefore addresses two key questions: which teachers the current student should learn from and how different temporal signals should contribute to student learning. Experiments on seven benchmark datasets demonstrate that ORCHESTRA achieves lower forecasting errors than individual teachers and other distillation strategies in most evaluated settings.
Critical strengths
- The perspective of adaptive teacher credit assignment and temporal signal calibration is well-motivated and interesting in the context of multi-teacher distillation. The experimental results provide empirical evidence supporting the effectiveness of the proposed framework and its individual components.
Critical weaknesses
-
Training efficiency remains a concern. Although Figure 3 reports the inference efficiency of the distilled student model, the computational overhead of the ORCHESTRA training process remains unclear. Given that ORCHESTRA requires computing teacher and student predictions, estimating student–teacher discrepancies, and updating an online reinforcement learning policy, it is unclear whether the framework maintains an efficiency advantage over simpler distillation baselines. A comparison of training time and computational costs would help clarify its practical efficiency.
-
Limited evaluation across student architectures. The experiments use TSMixer as the sole student architecture. Evaluating ORCHESTRA with other types of forecasting architectures would strengthen the evidence for its generalizability and applicability to different student models.
Questions for authors
-
Why are the relative importance weights of temporal signals, (c_d), fixed at ([0.5, 0.25, 0.25]) in Equation (17)? How sensitive is the performance to these predefined weights?
-
Why is reinforcement learning adopted instead of directly assigning teacher weights based on the calculated student–teacher discrepancies? A comparison with a simpler discrepancy-based weighting strategy would help justify the necessity of the RL-based approach.
3 comments
Response part 2
Q1. Why are the signal coefficients fixed, and how sensitive are they?
The default gives half of the base weight to the point signal, since matching forecast values is the primary target, and splits the rest between scale and delta as complementary signals for trend and step-to-step movement. We keep c fixed for two reasons:
- It is only a base allocation. The effective weight of each teacher–signal pair is w_k · c_d · Ψ_kd, and MSTE adapts Ψ within [0.5, 2] for every teacher and signal during training. The relative emphasis between signals therefore already changes with the student's state.
- Learning c directly would be degenerate. Minimizing the KD loss over c would simply shift weight to whichever signal currently has the smallest discrepancy.
We tested four allocations over seven datasets and four horizons:
| Point / Scale / Delta | Mean MSE | Mean MAE |
|---|---|---|
| 0.50 / 0.25 / 0.25, default | 0.2802 | 0.3330 |
| 1/3 / 1/3 / 1/3 | 0.2800 | 0.3329 |
| 0.25 / 0.50 / 0.25 | 0.2805 | 0.3331 |
| 0.25 / 0.25 / 0.50 | 0.2791 | 0.3321 |
- Performance is insensitive to the ratio. Mean MSE changes by only −0.41% to +0.10%. The equal and scale-heavy settings stay within 0.0003 MSE of the default, the same as the seed standard deviation of ORCHESTRA. This is consistent with MSTE absorbing much of the change in base weights.
- Emphasizing movement helps further. The delta-heavy setting is best, improving MSE by 0.41% and MAE by 0.26%. This confirms that supervising how the forecast moves between adjacent steps is valuable, which is the motivation for including the delta signal.
- Although the default is not the best tested ratio, we did not tune c on test results and keep the default throughout the paper. The delta-heavy result therefore shows further headroom for ORCHESTRA rather than a dependence on a carefully chosen ratio.
We have added this analysis to Appendix S5.3.
Q2. Why RL instead of direct discrepancy-based weighting?
Thank you for suggesting this comparison. Direct Softmax and Direct Sparsemax map each teacher's current discrepancy straight to a weight, so a closer teacher gets a larger weight. Both keep MSTE and the same three-signal objective, so only the allocation rule differs. We also include the static prior, which assigns credit from validation accuracy alone.
| Variant | Credit information | Mean MSE | Mean MAE |
|---|---|---|---|
| Static prior | Accuracy only | 0.2848 | 0.3391 |
| Direct Softmax | Discrepancy only | 0.2841 | 0.3352 |
| Direct Sparsemax | Discrepancy only | 0.2847 | 0.3361 |
| ORCHESTRA | Both, learned by ACP | 0.2802 | 0.3330 |
- The learned policy is consistently better. It lowers mean MSE by 1.36% and 1.56% relative to the two direct rules. Per dataset, it beats Direct Sparsemax on all seven in both metrics and Direct Softmax in MSE on six; on Solar they differ by 0.0001.
- Neither signal is enough alone. Discrepancy tells whether the student can follow a teacher, not whether it should: two equally close teachers receive the same weight even if one is close only because it shares the student's errors. Accuracy alone ignores what the student can currently absorb. ACP combines both with inter-teacher disagreement and temporal volatility, and learns from compatibility feedback how to weigh them.
- The benefit is inexpensive. As shown in W1, policy learning adds only 1.1–1.8% training time over the direct rules.
We have added the per-dataset comparison and its analysis to Section 4.4, and the summary above to Appendix S4.1.
We hope these responses address your concerns. We would welcome any further questions or suggestions and would be glad to discuss them.
Thank you to the authors for the detailed response and additional experiments. The new results on training efficiency, different student architectures, signal-weight sensitivity, and discrepancy-based weighting have adequately addressed my main concerns. I appreciate the authors' efforts to strengthen the experimental evaluation and am happy to increase my score in support of acceptance.
Response part 1
Thank you for your thoughtful feedback and for finding our credit assignment and temporal-signal calibration well motivated. We have carefully considered each concern and conducted additional experiments to address them. All new results and explanations have been added to the revised manuscript, and the corresponding changes are highlighted in blue. We devide the response into two parts.
W1. Training cost of ORCHESTRA
Thank you for raising this point. We profiled the complete training path on one H200 for all 28 dataset–horizon settings. After a warm-up epoch, we timed 200 batches that include data loading, teacher forwards, student updates and the scheduled policy updates. The table reports medians across settings.
The timing excludes the one-time validation pass that computes the accuracy prior. We have added these measurements and their analysis to Appendix S6.4.
W2. Generality across student architectures
Thank you for encouraging this evaluation. We added two students whose inductive biases differ from TSMixer: DLinear, a linear decomposition model, and a plain MLP. Each is compared with its own NoKD and Uniform KD controls. Entries are mean MSE/MAE over seven datasets and four horizons.
Because ORCHESTRA operates only on forecasts, it needs no matching of hidden layers and applies to each student unchanged. We have summarized this comparison in Section 4.3 and added the full results and analysis to Appendix S4.2.