acceptodds
Under review as a conference paper at ICLR 2027

AnchorMix: Distilling Heterogeneous Forecasters into a Single Routed Time-Series Model

Abstract

Heterogeneous time-series forecasters capture complementary inductive biases, but exploiting them through runtime ensembles multiplies latency, memory, and deployment dependencies. We introduce AnchorMix, a two-stage framework that converts this diversity into one student checkpoint. Stage 1 learns a routed temporal-query anchor from labels and cached Chronos-2 forecasts. Stage 2 warm-starts the same architecture and trains it against ground truth plus a convex target mixing the frozen anchor with an architecture-heterogeneous teacher. The anchor preserves established student behavior while the external teacher supplies complementary corrections; every teacher is removed after training. Across five benchmarks, four horizons, and three seeds, the single student improves 29 of 40 MSE/MAE cells relative to the selected external teachers and records lower values in 39 of 40 cells against four recent published comparison tables. A full-array diagnostic finds a positive convex-combination advantage on all 20 dataset–horizon tasks, and target-composition ablations on ETTh2 and ETTm2 identify the roles of anchor preservation and heterogeneous correction. The deployed model contains 4.06–5.36M parameters and averages 1.19 ms per batch of 32. On matched RTX 5090 profiles, replacing the corresponding runtime ensemble yields up to a 181.7x speedup and 80.7% lower peak allocated memory. AnchorMix therefore turns heterogeneous forecast diversity from a deployment burden into a training-time resource for one compact forecaster.

Primary area:
learning on time series and dynamical systems
Keywords:
time-series forecasting, knowledge distillation, heterogeneous teachers, mixture of experts, model compression, efficient inference, time-series foundation models

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_zpCoOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes AnchorMix, a two-stage distillation framework that transfers knowledge from heterogeneous time-series forecasters into a single anchor model. They introduce CycleRoutedTQNet, a student architecture comprising five internal forecasting branches and a sample-adaptive router. In stage 1, this architecture is trained as an anchor using ground-truth labels and forecasts from Chronos-2. In stage 2, the final CycleRoutedTQNet is initialized from the anchor and trained using ground truth together with a convex target that combines predictions from the frozen anchor and a task-specific external teacher. The method is evaluated on five datasets and generally improves performance over the selected external teachers and recently published baselines.

Critical strengths

S1. Practical compression of complementary forecasters. AnchorMix provides an effective way to transfer complementary knowledge from heterogeneous forecasters into a single deployable student while removing all teacher models at inference. This directly addresses the latency, memory, and deployment overhead of runtime ensembles.

S2. Strong empirical performance. Across five datasets and four forecasting horizons, the single student improves most of the MSE/MAE cells relative to the selected external teachers and recent published baselines, including ensemble-based methods.

Critical weaknesses

W1. Reliance on dataset-specific teacher selection. AnchorMix requires selecting an external teacher that provides useful signals for each dataset. However, the paper assigns different teachers to different datasets without sufficiently explaining or validating the teacher-selection procedure. This creates a practical bottleneck when applying the method to a new dataset and makes it difficult to determine how strongly the reported performance depends on the selected teacher. Similarly, the Stage-1 anchor relies on cached Chronos-2 predictions, but the effect of using this potentially imperfect teacher signal is not sufficiently examined.

W2. Marginal improvements over the selected teachers. The performance improvements over the selected external teachers are generally very small and are not consistent across all datasets and metrics. This raises the question of whether distilling complementary forecasts provides a meaningful benefit over simply selecting and deploying a strong individual forecaster.

Questions for authors

Q1. What exact procedure was used to select the external teacher, and the other task-specific optimization settings? Please specify the detailed selection process used for each decision.

Q2. How dependent are the final results on Chronos-2 supervision in Stage 1? Could you report results when the anchor is trained without the Chronos-2 term, as well as with another time-series forecasting foundation model in its place?

Q3. How sensitive is the method to the choice of heterogeneous teacher? Could you apply both MoHETS and PMDformer—and, if possible, another reasonable teacher—to the same dataset–horizon tasks under a common train/validation-only selection protocol?

Q4. Could you provide comparisons with recent strong single-model forecasters and a simple validation-selected best-single-model baseline?

Q5. Could you evaluate the method on more diverse multivariate datasets beyond the four closely related ETT benchmarks using a standardized evaluation framework such as the TSlib?

Reviewer_PYiLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This manuscript introduces AnchorMix, a two-stage framework that converts this diversity into one CycleRoutedTQNet checkpoint. Stage 1 learns a routed temporal-query anchor from labels and cached Chronos-2 forecasts. Stage 2 warm-starts the same architecture and trains it against ground truth plus a convex target mixing the frozen anchor with an architecture-heterogeneous teacher. The experimental results show that the proposed method can improve the forecasting accuracy on some public time series datasets while maintaining reasonable computational efficiency.

Critical strengths

Honestly, I find no clear merit in this immature manuscript. It reads more like the experimental output of a fully automated research tool rather than a rigorous scientific paper. While the direction is sensible, it is not a fresh idea.

Critical weaknesses

I have no specific comments on this manuscript, as it appears to be entirely generated by an automated research tool. The figures are poorly produced, and even basic elements like the arrows have not been properly adjusted.

Questions for authors

See Weaknesses.

Reviewer_cWAHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces AnchorMix, a two-stage distillation framework for multivariate time-series forecasting. It transfers knowledge from different forecasters into a compact student model with five routed forecasting branches.

The student, CycleRoutedTQNet, combines TQNet’s cycle-indexed temporal queries and channel attention with CycleNet’s residual-cycle forecasting. The authors add scale-enhanced, multi-cycle, and forecast-refinement branches, along with a sample-adaptive router. The base, scale, and multi-cycle branches share attention and forecasting parameters. This design aims to combine complementary forecasting capabilities without retaining external teachers at inference time.

In Stage 1, the student learns from ground truth and cached Chronos-2 forecasts. In Stage 2, it starts from the resulting checkpoint and learns from ground truth and a mixture of predictions from the frozen Stage-1 student and a selected external teacher. The teacher is either MoHETS, which combines convolutional and Fourier experts, or PMDformer, which separates patch means from residual shapes and models recent cross-variable relationships. The frozen student serves as an anchor for behavior learned during Stage 1, including guidance from Chronos-2.

The authors evaluate AnchorMix on four ETT datasets and Weather across four forecast horizons. The three-seed means improve on the selected teachers in 29 of 40 horizon-level MSE/MAE comparisons and seven of ten dataset-average metric comparisons. Both average metrics improve on ETTh2, ETTm1, and ETTm2. On ETTh1, only MAE improves; Weather is slightly worse on both metrics. Single-seed ablations on ETTh2 and ETTm2 favor the mixed target over anchor-only or external-teacher-only distillation. A separate diagnostic shows that the student retains much of the improvement available from combining the anchor and teacher forecasts.

The deployed student has 4.06–5.36 million parameters. In the matched deployment profiles, Chronos-2 has approximately 119.5 million parameters, MoHETS 2.4 million, and PMDformer 0.57 million. Relative to running the student and external teacher together, deploying only the distilled student yields a 181.7-fold speedup and 80.7% lower peak allocated memory for ETTh1–MoHETS. For ETTh2–PMDformer, the speedup is 1.54-fold and the memory reduction is 8.6%. These gains apply to the corresponding runtime ensembles; PMDformer alone is smaller and faster than the student.

Critical strengths

The key strength of the authors' framework proposition is its two-stage distillation process. First, the authors train a student model using ground-truth data and Chronos-2 forecasts. They then freeze the student and use it as an anchor while incorporating guidance from a selected heterogeneous teacher, either MoHETS or PMDformer. Starting the final student from this anchor and keeping its predictions as the distillation target helps preserve what it has already learned while letting the external teacher add useful corrections.

Critical weaknesses

The paper needs to better justify its architectural choices and evaluation scope. The authors explain the intended roles of the five forecasting branches, but do not make a strong case for why a routed architecture based on TQNet and CycleNet is preferable to simpler student models or other branch designs. MoHETS and PMDformer have distinct inductive biases, which provides a reasonable rationale for choosing them as teachers. However, the experiments do not show how much the distillation gains depend on the teachers selected or on the number of teachers used. The evaluation is also limited to four ETT datasets and Weather, despite the teacher papers reporting results on additional datasets. Testing those omitted datasets (or suitable tasks from broader benchmarks such as GIFT-Eval and TIME, with clearly specified training protocols) would help determine whether the gains extend beyond the settings studied here. These gaps make it difficult to assess how broadly the proposed design applies.

Questions for authors
  • Student architecture: What guided the choice of TQNet and CycleNet and the five-branch architecture? Did you consider alternative components, such as multiscale mixing from TimeMixer, adaptive multiscale pathways from Pathformer, or frequency decomposition from FreqMoE? Do comparisons with simpler students or alternative branches, under comparable parameter and training budgets, support the selected design?

  • Regime changes: How well do the existing branches and router handle distribution shifts and structural breaks? Did you consider an explicit mechanism for regime adaptation, drawing on drift-aware routing in Dynamic TMoE or trend and change-point modeling in TCDformer? Such a mechanism could modify the router rather than require a separate branch. Evidence on tasks containing regime changes would help establish whether the current design is sufficient.

  • Stage-1 teacher: What motivated the choice of Chronos-2 to supervise the initial student? If available when the experiments were conducted, were more recent models such as TimesFM-3 or TiRex-2 considered? How sensitive are the final gains to the Stage-1 teacher?

  • External teacher selection: Did you consider TimeMixer or FreqMoE as alternative sources of multiscale or frequency-based guidance, or iTransformer for cross-variable modeling? These models overlap with aspects of MoHETS and PMDformer without reproducing their full designs. What supports restricting Stage 2 to one selected external teacher per task, drawn from these two models? Would alternative teachers, multiple external teachers, or another time-series foundation model change the gains?

  • Need for two-stage training: Have you compared the proposed method with a single-stage student trained on ground truth and selectively weighted predictions from frozen teachers? For example, let AA be a pretrained reference teacher such as Chronos-2, T∗T_* the external teacher with the lowest error for a training example, and gg indicate whether it outperforms AA by a chosen margin. One possible objective is L=E ⁣[ℓ(S(X),Y)+λ{[1−g(1−ρ)] ℓ(S(X),A(X))+g(1−ρ) ℓ(S(X),T∗(X))}]\mathcal L = \mathbb E\!\left[ \ell(S(X),Y) +\lambda\left\{ [1-g(1-\rho)]\,\ell(S(X),A(X)) +g(1-\rho)\,\ell(S(X),T_*(X)) \right\}\right] where 0<ρ<10<\rho<1 retains reference-teacher guidance when an external teacher performs better. Teacher predictions and selection weights would be detached, selection would use training examples only, and hyperparameters would be chosen on chronological validation data. This would anchor predictions directly to a pretrained teacher, without first constructing a student anchor. Under matched student capacity, teacher access, and total training budget, does the two-stage method provide a measurable advantage?

  • Evaluation scope: Why was evaluation restricted to four ETT datasets and Weather when the teacher papers also evaluate ECL and Traffic, and PMDformer additionally evaluates Solar? Are results available on these omitted datasets? Evaluation on suitable tasks from GIFT-Eval or TIME could further test performance across domains and temporal patterns. Any such comparison should specify the training protocol and distinguish task-trained results from strict zero-shot evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.