acceptodds
Under review as a conference paper at ICLR 2027

MoHETS: Long-term Time Series Forecasting with Mixture-of-Heterogeneous-Experts

Abstract

Real-world multivariate time series exhibit heterogeneous temporal behavior, including recurring structure, local variation, and non-stationary regimes, making long-horizon forecasting challenging. Sparse Mixture-of-Experts (MoE) approaches improve scalability and specialization, but they typically rely on homogeneous MLP experts that poorly capture the diverse temporal dynamics of time series data. We introduce MoHETS, an encoder-only Transformer that integrates sparse Mixture-of-Heterogeneous-Experts (MoHE) layers. MoHE combines an always-active shared depthwise-convolution path for sequence-level continuity with routed Fourier-based experts applied independently to latent patch tokens. The resulting layer is heterogeneous along three axes: operator class, receptive field, and activation role. MoHETS further improves robustness to non-stationary dynamics by incorporating exogenous information through a cross-attention pathway. We also replace parameter-heavy linear projection heads with a lightweight convolutional decoder, thus improving parameter efficiency and training stability. Finally, a single trained model autoregressively produces forecasts over arbitrary horizons. We evaluated MoHETS on seven established multivariate benchmarks across multiple horizons and consistently achieved superior performance, indicating that structural heterogeneity is important for long-term forecasting.

Primary area:
learning on time series and dynamical systems
Keywords:
Time series, long-term forecasting, transformer, mixture-of-experts

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ErRDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MoHETS, a patch-based, encoder-only Transformer for long-term multivariate time series forecasting. Its main component is a Mixture-of-Heterogeneous-Experts (MoHE) layer that combines one always-active depthwise-convolution shared expert with top-2 routed Fourier-based (FAN) experts. The authors argue this makes the layer heterogeneous in operator class, receptive field, and activation role. The model also adds a cross-attention path that injects timestamp-derived calendar covariates, a lightweight convolutional output decoder, and chunked autoregressive rollout so that one model serves all horizons. It is trained with a Huber loss over a shifted full-length window plus a load-balancing loss. On seven standard benchmarks (ETT, Weather, ECL, Traffic), the paper reports the best average MSE/MAE (0.295/0.316), compared with baseline numbers taken from the literature.

Critical strengths

The paper has a motivated idea: replacing identical MLP experts with structurally different experts for time series. It supports this with ablations and detailed per-layer router diagnostics. It also reports hyperparameters, per-dataset configurations, and FLOP/memory, which is useful to the community. This suggests that structural heterogeneity— not just more parameters— can matter for time-series MoE.

Critical weaknesses

The output chunk size is set to Ho = 24 because of the ablation in Table 11, which says it led us to set Ho = 24. The Ho = 24 row of that table (ETTh1 0.377/0.411, ETTh2 0.343/0.379) is identical to the main test results in Tables 1, 6, and 9. The ablation therefore appears to report test metrics, and the paper does not state which split the ablations use. This conflicts with the claim that all selection was done on the validation split. If Ho or other configurations were chosen with test metrics, the main results are optimistically biased, which matters given the small margins. The authors should clarify which data split Tables 2, 3, 10, and 11 report, and how Ho = 24 was chosen.

Baseline numbers are copied from several papers. The paper says splits, horizons, and metrics are aligned, but never states the look-back length, covariates, or tuning budget of the baselines. MoHETS uses L = 672 with per-dataset tuning of model size, patch length, learning rate, batch size, and epochs, and it uses calendar covariates that most baselines do not. The reported 11.7% MSE reduction over TimeXer/SOFTS therefore cannot be attributed to the architecture alone. The authors should state the look-back, covariates, and tuning budget behind each baseline number, or provide matched-protocol comparisons.

Average MSE is 0.295 vs. 0.298 for xLSTM-M (MAE 0.316 vs. 0.318), and MoHETS loses to xLSTM-M on ETTm2 and ECL (on ECL-720, 0.200 vs. 0.183 MSE). Main results appear to be single runs; seed variability is reported only in the scaling study. The claim that MoHETS 'consistently achieved superior performance' is overstated. The authors should report the main results as mean ± std over several seeds, and qualify the claims.

In Table 2, most of the gain comes from using FAN experts (an existing component) instead of MLP experts. Adding the shared convolution expert changes the average MSE only from 0.315 to 0.310, with no seeds. DwConv + MLP (0.350) is worse than MLP alone (0.343), so the shared expert is not a robust benefit on its own. The all-MLP variant reaches 0.267 MSE on ECL, far worse than DLinear (0.212), which suggests the variants share hyperparameters tuned for MoHETS. There are no parameter- or FLOP-matched comparisons, no dense FAN-FFN control, and no sweeps over expert count or K. The authors should provide parameter- and FLOP-matched controls, a dense FAN-FFN baseline, and sweeps over N and K.

Eq. 11 supervises all L positions of a window shifted by Ho using a bidirectional encoder. With L = 672 and Ho = 24, only 24 of 672 supervised positions (about 3.6%) are true forecasts; the rest overlap the observed input. Only the last Ho outputs are used at inference, so I am not claiming test leakage. But the objective may dilute the forecasting signal or act as an auxiliary reconstruction task. It may also explain why accuracy improves sharply from Ho = 8 to Ho = 24 in Table 11, since the share of true-forecast supervision grows. The authors should provide a control (loss on the last Ho positions only, or a causal variant).

The ablation study is not sufficiently detailed to support the central claim. Specifically, there is no isolation of the periodic branches in the FA-FFN.The Fourier expert combines cosine, sine, and GELU branches. Without an ablation removing the periodic branches (e.g., setting W_per = 0), it is unclear whether the Fourier component is doing the work or whether a standard MLP-like pathway would suffice. The paper does not ablate the placement of MoHE layers—whether every block uses MoHE or only a subset—nor does it vary the number of MoHE layers to test whether all are needed. If only a subset of layers use MoHE, the contribution is smaller than the paper implies. The paper claims experts specialize in different temporal patterns, but provides no evidence of what they specialize in. The paper acknowledges in Appendix D that -semantic analysis of routing assignments is set as future work, but this is precisely the analysis needed to support the central claim. Correlating routing decisions with seasonality strength, trend, or frequency content of patches would directly test the claim. The paper claims improved robustness to non-stationarity but does not evaluate on any distribution-shift benchmark or synthetic non-stationary setting.Without these controls, the heterogeneity claim is not isolated.

MoHETS uses timestamp-derived calendar features, but the comparison does not adequately isolate the contribution of these features from that of the proposed architecture. Improvements from calendar features alone do not establish robustness to distribution shift, which would require explicit evaluation. The reported first-place counts appear inconsistent with the number of dataset–metric–horizon cells in Tables 6 and 7; the counting convention should be clarified. Fig. 4's text reports 4.7M total parameters for MoHETS-small on Traffic (conv decoder), while Table 4 lists 2.5M total (1.3M activated); a 2.2M difference seems larger than D-dependent layers would explain. The any-variate claim appears to conflict with Eq. 8, whose linear layers depend on D. Section 4.1 attributes the largest gain on ETTh1 to strong seasonality, yet Table 5 gives ETTh1 the lowest forecastability (0.38), which the paper defines as 1 minus the entropy of the Fourier decomposition. The authors should clarify how "strong seasonality" is being defined here and why the Fourier-based experts help most on the dataset with the least forecastable (and, by this definition, least periodic) structure.

Questions for authors
  1. Which data split do the ablations in Tables 2, 3, 10, and 11 report? How exactly was Ho = 24 chosen, and did any test metric influence the choice of Ho or the per-dataset configurations in Table 8?

  2. What are the baseline results under L = 672, or MoHETS under L = 96? Please state the look-back, covariates, and tuning budget behind each baseline number.

  3. Please report the main results (Table 1) for MoHETS with and without calendar covariates, as mean ± std over several seeds, so the covariate effect can be separated from the architectural effect when comparing against baselines without timestamp features.

  4. What happens if the loss is restricted to the last Ho positions, or if the encoder is causal?

  5. Does a dense, parameter- and FLOP-matched FAN-FFN, and a parameter-matched MLP-expert MoE, match MoHE? What do sweeps over N and K show?

  6. Fig.4's text reports 4.7M total parameters for MoHETS-small on Traffic (conv decoder), while Table 4 lists 2.5M total (1.3M activated). Please clarify what each number counts.

  7. Minor clarifications: the first-place counts appear inconsistent with the number of cells in Tables 6 and 7; what is the counting convention? Section 4.1 attributes the largest gain on ETTh1 to strong seasonality, yet Table 5 gives ETTh1 the lowest forecastability (0.38), which the paper defines as 1 minus the entropy of the Fourier decomposition. The authors should clarify how 'strong seasonality' is being defined here and why the Fourier-based experts help most on the dataset with the least forecastable (and, by this definition, least periodic) structure. The any-variate claim seems to conflict with Eq. 8, whose linear layers depend on D.

  8. Does removing the periodic branches in the FA-FFN (e.g., setting W_per = 0) degrade performance? How many MoHE layers are used in the best configuration, and does varying this number affect results?

  9. Is there a distribution-shift experiment that tests the claimed robustness to non-stationarity? Is there evidence that routing decisions correlate with temporal patterns (e.g., seasonality, trend, frequency)?

4 comments
AuthorsOpenReview ↗

Part 1 of 2

We thank the reviewer for the careful and constructive review, which led to substantial changes. Answers follow the order of the questions. Table and section numbers refer to the revised PDF.

Q1. Tables 2, 3, 10, and 12 (Table 11 in the submission) report test-split metrics. Ho=24 was chosen by grid search over {8,16,24,32} on the validation split (validation loss normalized by the chunk size). No test metric influenced any choice, as we stated in both the reproducibility statement and Appendix B.1. Checkpoints were also chosen on validation loss. The Ho=24 row of Table 12 matches the main results because it is the same configuration on the same test split. The sentence the reviewer quotes ("led us to set Ho=24") now states the validation-based selection, and the captions name the split.

Q2. Baselines at L=672: new Table 11 (Appendix C.4) reports eight baselines trained with a 672-step context under the same one-for-all rolling protocol (Liu et al., 2024). On four shared datasets, MoHETS has the lowest average MSE/MAE. Previous works on long-term forecasting typically fix the look-back window to L = 96 for "fair comparisons" in benchmark tasks. We follow Kraus et al. (2026) in the argument that such a practice is a source of distorted comparisons. Protocol behind each number: Table 1 now takes all baselines from one source (Kraus et al., 2026), which reports each model with its own optimized hyperparameters, including the look-back; tuning budgets are as reported.

Q3. Table 1 is single-run, which we note as a practice (e.g., PatchTST paper, page 18, "The results reported in the main text and appendix above are run with the fixed random seed 2021," i.e., single seed). Two seed studies give consistent estimates of run-to-run variability. Appendix C.1 (Figure 5, with error bars) uses five seeds on four datasets and four model sizes. New Table 13 uses three seeds on all seven datasets with the current implementation: the standard deviation is 0.002-0.009 in MSE on seven datasets. Consequence for the claims: the margin over xLSTM-Mixer is within this variability; we removed "consistently achieved superior performance" from the abstract and the state-of-the-art wording from the conclusion. The calendar covariates pathway is part of the proposed model, so Table 1 reports the full system; its effect is isolated in Table 3 under the fixed ablation configuration. This is why Section 4.1 is framed as a system-level comparison, and why we include baselines that rely on covariates (e.g., Timer-XL, AutoTimes).

Q4. Both are less accurate. Loss on the last Ho positions (new Appendix C.6, Table 13: three seeds, seven datasets, identical settings, each variant selected on its own validation loss): the future-only objective is worse on seven datasets in both MSE and MAE; the average rises from 0.312/0.327 to 0.340/0.351. Dense-trained models also reach a 3-10% lower validation loss on the last Ho positions, so the overlapping positions do not dilute the forecast signal. Causal variant: Appendix C.2, Table 10. A decoder-only MoHETS trained with the same objective. Trend in Table 12: the reviewer suggests that the gain from Ho=8 to 24 reflects the growing share of true-forecast supervision. However, the trend is not monotone: at Ho=32, that share is largest, yet accuracy drops. The trend is better explained by the rollout: a larger chunk requires fewer autoregressive steps, limiting error accumulation, but each step must predict further ahead.

Q5. Sweeps over N and K dense FAN-FFN were not run. We agree that these are useful design questions, but they are not entirely unstudied. Time-MoE compares sparse MoE models against dense variants matched to the activated parameter count under the same training data and setup, and also evaluates K ∈\in {1,2,4,6,8}, finding only marginal accuracy changes beyond K=2 while inference cost increases. Independently, the FAN paper (Dong et al., 2025) directly compares a standard dense Transformer with a dense Transformer that uses FAN in place of the MLP and reports lower error. Our ablations are designed to address the complementary question not isolated in the literature. For the MLP-expert MoE, the new Table 15 shows that the comparison is unfavorable to MoHETS: the MLP-expert variants have about 22% more parameters, more FLOPs, and higher error rates. On the reviewer's observations: (i) DwConv + MLP is 0.335, not 0.350 (Weather MSE in Table 2 was mistyped; 0.368 instead of 0.268); with the shared expert fixed, FAN experts beat MLP experts on all seven datasets. (ii) We agree that the shared-convolution gain is small; the revised text now names the routed experts as the more influential choice and the shared operator as a contributor. (iii) The variants do share one protocol (Sec 4.2), so these numbers are comparable with each other and not with tuned literature results such as DLinear. MoHETS is 0.165.

AuthorsOpenReview ↗

Part 2 of 2

Q6. Both numbers are correct and count different instantiations. 4.7M is the complete MoHETS-small for Traffic (D=862). Table 4 reported a low-dimensional reference instantiation without saying so. Its caption now states the reference (D=1, P=8; for MoHETS-small, 2.39M total and 1.21M activated), and no longer implies that the counts vary only slightly. Table 8 now reports dataset-instantiated counts.

Q7. The counting convention was reviewed, is now defined in the text (overall-average row excluded, ties shared), and follows prior work (e.g., Time-MoE). We agree that attributing the ETTh1 margin to "strong seasonality" was unsupported. The sentence is removed. Any-variate claim: the reviewer is right; Eq. 8 depends on D. The claim is narrowed to the channel-independent Transformer backbone (Sec. 3.2).

Q8. Every Transformer block of every reported model uses a MoHE layer in place of the FFN. Sec 3.5 now states this. We did not run W_per=0. That setting does not yield a standard MLP expert: the cosine and sine coordinates become constants, and all input-dependent variation is carried by the GELU branch at half the width, so it removes periodic features and capacity simultaneously. The control that maintains width while replacing periodic units with GELU units is the MLP-FFN expert, and it is in Table 2, with the same shared expert (DwConv + MLP), and is worse than MoHETS on all seven datasets despite having 21.9% more parameters (Q5). We agree that this is a broader control than a branch-level ablation.

Q9. We have no distribution-shift experiment, so the claim is withdrawn. The only remaining mention is the standard motivation for input normalization (Sec 3.2). Appendix D reports expert-utilization diagnostics, which show that the routed pool does not collapse, but not what the experts specialize in. Statements that experts specialize in identifiable temporal patterns are removed, and the semantic analysis of routing remains future work.

I thank the authors for their detailed rebuttal. The new experiments (especially the validation-based selection of Ho=24, the new Table 11 with L=672 baselines, the 3-seed standard deviations in Table 13, and the future-only loss control) provide clarity. Authors correcting the typos (0.335 instead of 0.350, 0.368 instead of 0.268) and removing the unsupported strong seasonality and distribution-shift claims.

However, I will keep my rating of 2 (Weak Rejection ). My primary concerns remain partially unresolved --

Q1 I appreciate the clarification that Ho=24 was chosen on validation and the captions now state the split. However, because the Ho=24 row exactly matches the main test results, and the original text lacked this clarity, the possibility of test-metric influence in the initial submission cannot be fully erased. The authors should ensure the final version strictly separates validation ablations from test reporting.

Q5 The authors admit they did not run the requested dense FAN-FFN control or the N/K sweeps. While I understand the computational constraints and their references to Time-MoE and FAN, the lack of these matched controls means the claim that heterogeneity (rather than just the FAN expert) matters is still not fully isolated.

Q8 The authors argue that setting W_per=0 changes capacity, and they use the MLP-FFN expert as the correct control. I accept this reasoning, but the fact remains that the specific periodic branches themselves are not ablated.

Q9 The authors have responsibly withdrawn the distribution-shift claim and removed the statements about expert specialization. This is the right action, but it means the core motivation (that experts learn distinct temporal patterns) remains a hypothesis rather than a demonstrated fact

Dear reviewer ErRD, thank you for the constructive feedback.

Q1. We understand that the wording of the original submission was unclear. What we can add is the selection evidence and the separation that the reviewer asks for. The loss on the validation of the four candidates using MoHETS tiny. As in Table 12, Ho=24 is the lowest:

Ho8162432
ETTh11.27641.09151.05611.1023
ETTh20.40010.34980.34520.3481

Table 12 will report these losses in the final version, together with the inference time, in place of the test reporting. Selection is on validation data only, and test metrics appear for the selected configuration in the main tables, which strictly separates validation-based selection from test reporting, as the reviewer requests. The text will be updated accordingly.

Q5 and Q8. We ran the requested controls under the ablation protocol of Table 2 on the five smaller datasets. Entries are MSE; the last column is the average MSE/MAE.

ETTh1ETTh2ETTm1ETTm2WeatherAvg. MSE/MAE
MoHETS (Table 2)0.3830.3620.3380.2700.2440.319/0.350
W_per=00.3920.3670.3510.2780.2390.325/0.352
Dense FA-FFN0.3940.3550.3690.2840.2620.333/0.359
MLP experts, matched0.4040.3500.3610.2700.2410.325/0.354
N=4, K=10.3930.3560.3440.2750.2600.326/0.351
N=4, K=20.3910.3590.3480.2770.2490.325/0.352
N=8, K=10.3840.3530.3480.2710.2450.320/0.351
N=8, K=40.3940.3700.3410.2730.2720.330/0.355
N=16, K=20.3880.3660.3540.2700.2620.328/0.354

MoHETS has the lowest average MSE and MAE of the nine variants.

Q8. With the periodic projection of every FA layer set to zero and frozen, the error is higher on four of the five datasets (average 0.325/0.352 vs. 0.319/0.350). Removing the periodic branch modestly degrades accuracy at this scale.

Q5. What each control isolates:

  • Dense FA-FFN, activated parameters matched (sparse routing): the weakest variant (0.333/0.359), worse on four of the five datasets and most on ETTm1 (0.369 vs. 0.338). A dense FA-FFN of equal capacity does not match MoHE.
  • MLP experts with the parameter count of an FA expert (d_ff=192; operator at equal capacity): higher average error (0.325/0.354), most on ETTh1 and ETTm1.
  • N and K (sparsity setting): N=8, K=2 gives the lowest average. K=1 is close, while fewer or more experts and K=4 bring no gain, and K=4 is 32% slower at inference (46.1 s vs. 34.9 s for K=1 on ETTh1). This result agrees with the top-K study of Time-MoE (Table 7), which reports equal accuracy for K=2 and K=4 at a lower cost for K=2.

The controls are consistent with each design choice: sparse routing over a dense FFN, FA over capacity-matched MLP experts, and N=8 with K=2. The margins are modest on these datasets; the largest effects of the composition are on ECL and Traffic (Table 2: 0.164 and 0.406 for MoHE, 0.174 and 0.428 for homogeneous FA, at least 0.197 and 0.420 with MLP experts). Those two datasets take several hours per run and could not be completed within the discussion period. The final version will include these controls on them.

Q9. We agree: which temporal characteristics drive individual routing decisions is an open question, and the revision does not read the utilization diagnostics as evidence of trend-, seasonality-, or frequency-specific experts. We would separate this interpretability question from the design motivation. Input-dependent specialization is the standard motivation for sparse MoE layers (Shazeer et al., 2017), shared-plus-routed designs (DeepSeekMoE; Dai et al., 2024), and their use in forecasting (Time-MoE; Shi et al., 2024). This literature explains why an MoE is designed to permit specialization. MoHE adds structural heterogeneity to that design space. The paper's claims rest on what is explicit in the architecture: a sequence-mixing shared path and token-wise FA experts, and on its measured effect in the composition ablation. Appendix D shows that routing is non-degenerate and layer-dependent. Whether individual experts acquire interpretable temporal roles remains a separate empirical question, now stated as future work.

Reviewer_2oHPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MoHETS, an encoder-only Transformer for long-term time series forecasting with a mixture of heterogeneous experts. It combines a shared depthwise-convolutional expert with sparsely routed Fourier-based experts, alongside exogenous covariate conditioning and a lightweight convolutional decoder. The model supports autoregressive forecasting across different horizons using a single checkpoint. Experiments on seven multivariate forecasting benchmarks demonstrate competitive performance, supported by architectural ablations and efficiency analyses.

Critical strengths
  1. MoHETS combines sequence-level convolutional processing with token-wise Fourier-based experts, providing complementary inductive biases for modeling diverse temporal patterns.

  2. The model achieves strong results across seven established benchmarks, particularly on ETTh1, ETTm1, Weather, and Traffic. The component ablations, scaling experiments, and efficiency analyses further support the proposed architecture.

  3. The article is written with clear expressions and charts.

Critical weaknesses
  1. The novelty of the method is limited. The innovation of MoHETS mainly lies in combining them into a unified prediction framework, including sparse MoE, convolution FAN、 Cross attention is an existing technology. The novelty of this combination has not been fully demonstrated compared to previous frequency-aware MoE methods such as FreqMoE.

  2. The characters in Figure 1 are too small, which is not conducive to reading and understanding

  3. On page 17, it is clearly stated that FAN experts also include non-periodic branches. Appendix D mainly verifies whether experts are used, but does not directly analyze the differences in response of different experts to trends, frequencies, or local changes. Thus, the paper currently demonstrates structural heterogeneity, but the evidence for its claimed expert specialization mechanism is still insufficient.

  4. The references are somewhat outdated and lack literature from 2026.

  5. The article seems to rely too heavily on AI, with "therefore" appearing too many times (a total of 17 times).

Questions for authors

Please see Weaknesses.

2 comments

We thank the reviewer for the positive assessment of the complementary design, the results, and the ablations. Answers follow the weakness numbers; section and table numbers refer to the revised PDF.

W1. The reviewer identifies precisely where our contribution lies. We do not propose a new operation: sparse MoE, depthwise convolution, FAN layers, and cross-attention are existing components, and the paper attributes each of them. The contribution is how the experts of a sparse MoE layer are composed. In MoE forecasters such as Time-MoE and Moirai-MoE, every expert is a copy of the same point-wise MLP, so experts can differ only through learned weights and routing. In MoHE, the experts differ by construction along three axes:

  • operator class: a depthwise convolution vs. Fourier-analysis transformations;
  • receptive field: the convolution mixes neighboring patch positions, while the routed experts act on single tokens;
  • activation role: one path is always active, and the others are sparsely routed, with separate learned gates.

MoHE differs from FreqMoE. FreqMoE transforms the input window with an FFT, splits the spectrum into bands, gives each band to one expert, weights the experts with a gate on the magnitude spectrum, and returns to the time domain before a stack of residual prediction blocks. So its experts differ in the frequency band they receive. MoHE performs no spectral decomposition. Its experts are different operators that act on latent patch tokens inside every Transformer block, and the router dispatches individual tokens. The revised Section 2.3 states this distinction more clearly

Evidence that the composition matters. Table 2 presents six expert compositions under a single protocol. The three compositions with Fourier-routed experts rank first to third in average MSE (0.310, 0.315, 0.315), and the three with MLP-routed experts rank fourth to sixth (0.319, 0.335, 0.343). New Table 15 shows that this is not a capacity effect: an FA-FFN has 0.75x the parameters of an MLP-FFN of equal width, so the MLP-expert variants are the larger models. With the shared expert of MoHETS held fixed, FA experts are better than MLP experts on all seven datasets in both metrics.

W2. Figure 1 is enlarged by 20%, so its characters are larger

W3. The reviewer is right that Appendix D verifies whether experts are used, not how they respond to trends, frequencies, or local changes, and that FA-FFN experts also contain a non-periodic branch. We removed the text saying that experts are shown to specialize in such patterns. The revision claims what is established: MoHE is structurally heterogeneous, and the routing diagnostics show that the routed pool does not collapse and that utilization depends on the layer. Appendix D now states that a semantic analysis relating routing decisions to trends, periodicities, and frequency content remains an important direction for future analysis

W4. We added recent work where it is directly relevant: xLSTM-Mixer and TimeMixer++ in Section 2.1, and Super-Linear (TMLR 2026), M2FMoE (AAAI 2026), and Dynamic TMoE (2026) in Section 2.3. In addition, all baselines of Table 1 now come from a 2026 consolidated evaluation (Kraus et al.), which includes recent models such as CycleNet and TimeMixer

W5. As disclosed in the paper, AI assistance was used only to polish the writing. We acknowledge that such polishing can leave repetitive phrasing. The manuscript was proofread again, and repeated connectives were reduced.

AuthorsOpenReview ↗

Kind Follow-up on Our Response

Dear Reviewer 2oHP,

Thank you again for your thoughtful review. As the discussion period nears its close, we wanted to follow up and ask whether you have had a chance to consider our response, in particular the distinction between MoHE and FreqMoE now stated in Section 2.3, the evidence that the expert composition matters beyond capacity (Table 2 together with the new Table 15), and the revisions concerning Figure 1, expert specialization, and recent references.

We would welcome any remaining questions. If our response has changed your assessment, we would appreciate any update you consider appropriate.

Best regards, The Authors

Reviewer_R8fHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MoHETS, a patch-based encoder-only Transformer for long-term multivariate forecasting that replaces the FFN with a Mixture-of-Heterogeneous-Experts layer combining an always-active shared depthwise-convolution expert and top-2 of 8 sparsely routed Fourier-Analysis-Network experts. It also incorporates cross-attention for timestamp calendar covariates, a convolutional patch decoder, RoPE, GQA, and a shifted-window objective with autoregressive rollout in chunks of 24. Across seven benchmarks, MoHETS reports the best average MSE and MAE of 0.295 and 0.316 against literature baselines, edging out xLSTM-Mixer at 0.298 and 0.318.

Critical strengths

The engineering and analysis are thorough and transparent, with a complete appendix detailing hyperparameters, per-dataset configurations, FLOPs, memory, and router load diagnostics, alongside honest scoping of limitations.

The ablation signal for the routed operator is consistent and practical, as replacing MLP experts with FAN-style experts yields large gains on high-dimensional datasets in Table 2 by reducing ECL MSE from 0.267 to 0.164 and Traffic from 0.498 to 0.406, within a compact model of 0.3M to 21M active parameters where one checkpoint serves all horizons via chunk-24 rollout.

Critical weaknesses

The headline SOTA comparison is uncontrolled and unfair because literature baselines use the standard look-back L=96 while MoHETS uses L=672; moreover, MoHETS injects timestamp covariates that baselines do not see, which accounts for a 4.5% average MSE gain and 9% on ETTh1 from 0.418 to 0.383, and applies per-dataset tuning of model size and hyperparameters, meaning the 11.7% gain over TimeXer and SOFTS reflects protocol differences while the gap to the comparably tuned xLSTM-Mixer is only 1% at 0.295 versus 0.298 and worse on ECL-720 at 0.200 versus 0.183.

Causal attribution to structural heterogeneity is weak since FA-only at 0.315 and Conv+FA at 0.315 in Table 2 are within 0.005 of the full model at 0.310, showing the shared conv expert contributes little and gains come mainly from the Fourier operator, which is confounded by different initialization and unmatched parameter counts without evidence that routing specializes by trend versus periodicity.

The paper claims an any-variate design, yet the linear projection in the covariate path maps the full D-dimensional vector and makes the model D-specific, while key ablations like output chunk size and patch length report test-set metrics.

Questions for authors

Can you rerun the main baselines, including PatchTST, iTransformer, TimeXer, SOFTS, and TimeMixer, under a fair protocol using the same look-back L=672 and the same calendar covariates to verify whether the architectural claim holds?

Please provide parameter- and FLOP-matched ablations comparing FA versus MLP experts under identical initialization and learned versus random routing, and show whether routing correlates with temporal properties like periodicity or trend.
Since Table 11 and the patch-length ablations report test-set metrics, were the output chunk size of 24, patch length, and model size selected strictly on validation data, and how is the any-variate claim reconciled with the D-dimensional covariate fusion?

4 comments

We thank the reviewer for the precise diagnosis, and the revisions have changed the paper accordingly. Table and section numbers refer to the revised PDF (the submission's Tab 11 is now Tab 12).

Q1. A fair protocol. We agree that the submitted Tab 1 mixed protocols. Tab 1 now takes all 10 baselines from a single consolidated evaluation (Kraus et al., 2026; NeurIPS), which reports every model with optimized hyperparameters, including the look-back, so neither the look-back nor per-dataset tuning is one-sided. TimeMixer is included at its tuned look-back. TimeXer and SOFTS are not in that source, so they are dropped; the 11.7% statement is removed

Baselines at L=672. Instead of retraining, new Tab 11 (Appendix C.4) uses a published benchmark that matches our protocol (Liu et al., 2024; NeurIPS): each method is trained once per dataset with a 672-step context and rolled out to all horizons. MoHETS is best in 31 of 40 cells, with average MSE/MAE of 0.284/0.290, compared to 0.299/0.310 for PatchTST and 0.309/0.317 for iTransformer.

Calendar covariates. They are not exclusive to MoHETS: AutoTimes (Tab 11) takes timestamps as input, and Timer-XL (Tab 7) supports covariates by design. Adding identical calendar inputs to the other baselines is not a drop-in change: several lack a covariate interface, and adding one would modify their architectures

We agree with the reviewer's reading. The margin over the tuned xLSTM-Mixer is about 1%, and xLSTM-Mixer is better on ECL, including ECL-720; the revision states both (Sec 4.1, Appendix C), and the headline claim of "consistently superior performance" is removed. What the benchmark tables support is that a model with 0.3M-21M activated parameters and one checkpoint for all horizons is competitive with tuned state-of-the-art baselines

Q2. Attribution to the heterogeneous experts (W2). We agree: 0.310 vs. 0.315 is small. The two variants differ by 1,536 parameters (under 0.1%), so the comparison is fair, but it does not establish a robust benefit. Sec 4.2 now says so, naming the routed Fourier experts as the main factor, with a contribution from the shared expert.

Parameter and FLOP matching. New Tab 15 (Appendix C.7) reports parameters and FLOPs for the six variants. The comparison is not fully matched, but the imbalance does not favor MoHETS: the MLP-expert variants have more parameters and FLOPs, since an FA-FFN has 0.75x the parameters of an MLP-FFN of equal width. With the shared expert held fixed, FA experts beat MLP experts on all seven datasets in both metrics, although the MLP variant has 21.9% more parameters

The reviewer is correct that FAN and MLP use different initialization schemes (Appendix B.2). However, forcing identical initialization across different nonlinear operators is not, by itself, a fairer control. The choice is tied to the operator: with small initial weights, sine and cosine start in their near-linear region, and the periodic branch is degenerate. Imposing one scheme on both operators would handicap one of them.

Routing. We did not run a random-routing control. Appendix D shows that the routed pool is used without collapse. The revision removes the statements that experts specialize in identifiable temporal patterns and lists such an analysis as a limitation. The claim that remains is the one the reviewer credits as a strength: replacing MLP experts with Fourier experts reduces error, and Tab 15 shows that this does not stem from capacity

Q3. Validation-based selection; any-variate (W3).

Selection. Yes, Ho, the patch length, model size, and optimization hyperparameters were selected on the validation split, and checkpoints were chosen based on validation loss. The test split was used only for reporting, as stated in both the reproducibility statement and Appendix B.1. Ho was searched over {8, 16, 24, 32}, and P over {8, 12, 16}. Tab 12 and the tables behind the patch-length discussion do report test metrics, as the reviewer notes; they describe the selected and neighboring configurations and were not used for selection. The revision names the split in the captions and clarifies in Appendix C.5 that Ho was selected on validation

Any-variate. The reviewer is right. The temporal backbone is channel-independent, but the covariate fusion of Eq. 8 projects the full D-dimensional vector. The claim is narrowed to the channel-independent backbone (Sec 3.2).

AuthorsOpenReview ↗

Kind Follow-up on Our Response

Dear Reviewer R8fH,

Thank you again for your thoughtful review. As the discussion period nears its close, we wanted to follow up and ask whether you have had a chance to consider our response, in particular the comparison at a matched 672-step look-back (new Table 11, which includes PatchTST and iTransformer), the parameter and FLOP accounting for the expert ablation (new Table 15), and the narrowed any-variate claim. We will update Table 12 to report the validation loss of each candidate chunk size, which is lowest at Ho = 24 on both datasets.

We would welcome any remaining questions. If our response has changed your assessment, we would appreciate any update you consider appropriate.

Best regards, The Authors

Thanks for the revisions. I still have concerns about the fairness of the comparison, and it’s unclear how much of the improvement comes from using heterogeneous experts. For those reasons, I’m inclined to keep my original score.

Dear Reviewer R8fH, thank you for the reply. We have two short clarifications.

How much comes from the heterogeneous experts. Table 2 gives the direct comparison with homogeneous experts under one protocol: the heterogeneous layer lowers the seven-dataset MSE by 1.6% against all-FA experts (0.310 vs. 0.315) and by 9.7% against all-MLP experts, and the effect is largest on the high-dimensional datasets (against all-FA: 0.164 vs. 0.174 on ECL and 0.406 vs. 0.428 on Traffic).

The controls requested in Q2 were run after our first response and are included in our reply to Reviewer ErRD: on five datasets, capacity-matched MLP experts reach 0.325/0.354, and a dense FA-FFN of matched capacity reaches 0.333/0.359, against 0.319/0.350 for MoHETS. We invite the reviewer to look at these new controls.

Fairness. Table 11 compares all methods using the same 672-step context with a single checkpoint across all horizons, and Table 1 reports each baseline at its own tuned look-back. We agree that neither adds our calendar covariates to the baselines (AutoTimes, in Table 11, already takes timestamps as input). For this reason, the paper now claims that MoHETS is competitive rather than superior.

We are glad to clarify any remaining specific points.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Comments

We thank all reviewers for their careful and constructive reviews. We have thoroughly revised the manuscript in response, and the revised PDF is now available. Our individual responses point to the relevant sections and tables for each comment. We believe the paper is clearer and better supported as a result, and we are grateful for the feedback that made this possible.

Main changes: all baselines in Table 1 now come from one consolidated evaluation; a comparison at matched look-back (Table 11); a three-seed ablation of the training objective (Appendix C.6); parameter and FLOP accounting for the expert ablation (Table 15); one corrected cell in Table 2; and narrower claims on non-stationarity, any-variate use, and expert specialization.

Since our previous summary, Table 12 reports the validation loss of each candidate output chunk size in place of test metrics, and the controls requested during the discussion (dense FA-FFN, capacity-matched MLP experts, sweeps over N and K, and removal of the periodic branch) are reported in our reply to Reviewer ErRD.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.