- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes MoHETS, a patch-based, encoder-only Transformer for long-term multivariate time series forecasting. Its main component is a Mixture-of-Heterogeneous-Experts (MoHE) layer that combines one always-active depthwise-convolution shared expert with top-2 routed Fourier-based (FAN) experts. The authors argue this makes the layer heterogeneous in operator class, receptive field, and activation role. The model also adds a cross-attention path that injects timestamp-derived calendar covariates, a lightweight convolutional output decoder, and chunked autoregressive rollout so that one model serves all horizons. It is trained with a Huber loss over a shifted full-length window plus a load-balancing loss. On seven standard benchmarks (ETT, Weather, ECL, Traffic), the paper reports the best average MSE/MAE (0.295/0.316), compared with baseline numbers taken from the literature.
Critical strengths
The paper has a motivated idea: replacing identical MLP experts with structurally different experts for time series. It supports this with ablations and detailed per-layer router diagnostics. It also reports hyperparameters, per-dataset configurations, and FLOP/memory, which is useful to the community. This suggests that structural heterogeneity— not just more parameters— can matter for time-series MoE.
Critical weaknesses
The output chunk size is set to Ho = 24 because of the ablation in Table 11, which says it led us to set Ho = 24. The Ho = 24 row of that table (ETTh1 0.377/0.411, ETTh2 0.343/0.379) is identical to the main test results in Tables 1, 6, and 9. The ablation therefore appears to report test metrics, and the paper does not state which split the ablations use. This conflicts with the claim that all selection was done on the validation split. If Ho or other configurations were chosen with test metrics, the main results are optimistically biased, which matters given the small margins. The authors should clarify which data split Tables 2, 3, 10, and 11 report, and how Ho = 24 was chosen.
Baseline numbers are copied from several papers. The paper says splits, horizons, and metrics are aligned, but never states the look-back length, covariates, or tuning budget of the baselines. MoHETS uses L = 672 with per-dataset tuning of model size, patch length, learning rate, batch size, and epochs, and it uses calendar covariates that most baselines do not. The reported 11.7% MSE reduction over TimeXer/SOFTS therefore cannot be attributed to the architecture alone. The authors should state the look-back, covariates, and tuning budget behind each baseline number, or provide matched-protocol comparisons.
Average MSE is 0.295 vs. 0.298 for xLSTM-M (MAE 0.316 vs. 0.318), and MoHETS loses to xLSTM-M on ETTm2 and ECL (on ECL-720, 0.200 vs. 0.183 MSE). Main results appear to be single runs; seed variability is reported only in the scaling study. The claim that MoHETS 'consistently achieved superior performance' is overstated. The authors should report the main results as mean ± std over several seeds, and qualify the claims.
In Table 2, most of the gain comes from using FAN experts (an existing component) instead of MLP experts. Adding the shared convolution expert changes the average MSE only from 0.315 to 0.310, with no seeds. DwConv + MLP (0.350) is worse than MLP alone (0.343), so the shared expert is not a robust benefit on its own. The all-MLP variant reaches 0.267 MSE on ECL, far worse than DLinear (0.212), which suggests the variants share hyperparameters tuned for MoHETS. There are no parameter- or FLOP-matched comparisons, no dense FAN-FFN control, and no sweeps over expert count or K. The authors should provide parameter- and FLOP-matched controls, a dense FAN-FFN baseline, and sweeps over N and K.
Eq. 11 supervises all L positions of a window shifted by Ho using a bidirectional encoder. With L = 672 and Ho = 24, only 24 of 672 supervised positions (about 3.6%) are true forecasts; the rest overlap the observed input. Only the last Ho outputs are used at inference, so I am not claiming test leakage. But the objective may dilute the forecasting signal or act as an auxiliary reconstruction task. It may also explain why accuracy improves sharply from Ho = 8 to Ho = 24 in Table 11, since the share of true-forecast supervision grows. The authors should provide a control (loss on the last Ho positions only, or a causal variant).
The ablation study is not sufficiently detailed to support the central claim. Specifically, there is no isolation of the periodic branches in the FA-FFN.The Fourier expert combines cosine, sine, and GELU branches. Without an ablation removing the periodic branches (e.g., setting W_per = 0), it is unclear whether the Fourier component is doing the work or whether a standard MLP-like pathway would suffice. The paper does not ablate the placement of MoHE layers—whether every block uses MoHE or only a subset—nor does it vary the number of MoHE layers to test whether all are needed. If only a subset of layers use MoHE, the contribution is smaller than the paper implies. The paper claims experts specialize in different temporal patterns, but provides no evidence of what they specialize in. The paper acknowledges in Appendix D that -semantic analysis of routing assignments is set as future work, but this is precisely the analysis needed to support the central claim. Correlating routing decisions with seasonality strength, trend, or frequency content of patches would directly test the claim. The paper claims improved robustness to non-stationarity but does not evaluate on any distribution-shift benchmark or synthetic non-stationary setting.Without these controls, the heterogeneity claim is not isolated.
MoHETS uses timestamp-derived calendar features, but the comparison does not adequately isolate the contribution of these features from that of the proposed architecture. Improvements from calendar features alone do not establish robustness to distribution shift, which would require explicit evaluation. The reported first-place counts appear inconsistent with the number of dataset–metric–horizon cells in Tables 6 and 7; the counting convention should be clarified. Fig. 4's text reports 4.7M total parameters for MoHETS-small on Traffic (conv decoder), while Table 4 lists 2.5M total (1.3M activated); a 2.2M difference seems larger than D-dependent layers would explain. The any-variate claim appears to conflict with Eq. 8, whose linear layers depend on D. Section 4.1 attributes the largest gain on ETTh1 to strong seasonality, yet Table 5 gives ETTh1 the lowest forecastability (0.38), which the paper defines as 1 minus the entropy of the Fourier decomposition. The authors should clarify how "strong seasonality" is being defined here and why the Fourier-based experts help most on the dataset with the least forecastable (and, by this definition, least periodic) structure.
Questions for authors
-
Which data split do the ablations in Tables 2, 3, 10, and 11 report? How exactly was Ho = 24 chosen, and did any test metric influence the choice of Ho or the per-dataset configurations in Table 8?
-
What are the baseline results under L = 672, or MoHETS under L = 96? Please state the look-back, covariates, and tuning budget behind each baseline number.
-
Please report the main results (Table 1) for MoHETS with and without calendar covariates, as mean ± std over several seeds, so the covariate effect can be separated from the architectural effect when comparing against baselines without timestamp features.
-
What happens if the loss is restricted to the last Ho positions, or if the encoder is causal?
-
Does a dense, parameter- and FLOP-matched FAN-FFN, and a parameter-matched MLP-expert MoE, match MoHE? What do sweeps over N and K show?
-
Fig.4's text reports 4.7M total parameters for MoHETS-small on Traffic (conv decoder), while Table 4 lists 2.5M total (1.3M activated). Please clarify what each number counts.
-
Minor clarifications: the first-place counts appear inconsistent with the number of cells in Tables 6 and 7; what is the counting convention? Section 4.1 attributes the largest gain on ETTh1 to strong seasonality, yet Table 5 gives ETTh1 the lowest forecastability (0.38), which the paper defines as 1 minus the entropy of the Fourier decomposition. The authors should clarify how 'strong seasonality' is being defined here and why the Fourier-based experts help most on the dataset with the least forecastable (and, by this definition, least periodic) structure. The any-variate claim seems to conflict with Eq. 8, whose linear layers depend on D.
-
Does removing the periodic branches in the FA-FFN (e.g., setting W_per = 0) degrade performance? How many MoHE layers are used in the best configuration, and does varying this number affect results?
-
Is there a distribution-shift experiment that tests the claimed robustness to non-stationarity? Is there evidence that routing decisions correlate with temporal patterns (e.g., seasonality, trend, frequency)?
4 comments
Part 2 of 2
Q6. Both numbers are correct and count different instantiations. 4.7M is the complete MoHETS-small for Traffic (D=862). Table 4 reported a low-dimensional reference instantiation without saying so. Its caption now states the reference (D=1, P=8; for MoHETS-small, 2.39M total and 1.21M activated), and no longer implies that the counts vary only slightly. Table 8 now reports dataset-instantiated counts.
Q7. The counting convention was reviewed, is now defined in the text (overall-average row excluded, ties shared), and follows prior work (e.g., Time-MoE). We agree that attributing the ETTh1 margin to "strong seasonality" was unsupported. The sentence is removed. Any-variate claim: the reviewer is right; Eq. 8 depends on D. The claim is narrowed to the channel-independent Transformer backbone (Sec. 3.2).
Q8. Every Transformer block of every reported model uses a MoHE layer in place of the FFN. Sec 3.5 now states this. We did not run W_per=0. That setting does not yield a standard MLP expert: the cosine and sine coordinates become constants, and all input-dependent variation is carried by the GELU branch at half the width, so it removes periodic features and capacity simultaneously. The control that maintains width while replacing periodic units with GELU units is the MLP-FFN expert, and it is in Table 2, with the same shared expert (DwConv + MLP), and is worse than MoHETS on all seven datasets despite having 21.9% more parameters (Q5). We agree that this is a broader control than a branch-level ablation.
Q9. We have no distribution-shift experiment, so the claim is withdrawn. The only remaining mention is the standard motivation for input normalization (Sec 3.2). Appendix D reports expert-utilization diagnostics, which show that the routed pool does not collapse, but not what the experts specialize in. Statements that experts specialize in identifiable temporal patterns are removed, and the semantic analysis of routing remains future work.
I thank the authors for their detailed rebuttal. The new experiments (especially the validation-based selection of Ho=24, the new Table 11 with L=672 baselines, the 3-seed standard deviations in Table 13, and the future-only loss control) provide clarity. Authors correcting the typos (0.335 instead of 0.350, 0.368 instead of 0.268) and removing the unsupported strong seasonality and distribution-shift claims.
However, I will keep my rating of 2 (Weak Rejection ). My primary concerns remain partially unresolved --
Q1 I appreciate the clarification that Ho=24 was chosen on validation and the captions now state the split. However, because the Ho=24 row exactly matches the main test results, and the original text lacked this clarity, the possibility of test-metric influence in the initial submission cannot be fully erased. The authors should ensure the final version strictly separates validation ablations from test reporting.
Q5 The authors admit they did not run the requested dense FAN-FFN control or the N/K sweeps. While I understand the computational constraints and their references to Time-MoE and FAN, the lack of these matched controls means the claim that heterogeneity (rather than just the FAN expert) matters is still not fully isolated.
Q8 The authors argue that setting W_per=0 changes capacity, and they use the MLP-FFN expert as the correct control. I accept this reasoning, but the fact remains that the specific periodic branches themselves are not ablated.
Q9 The authors have responsibly withdrawn the distribution-shift claim and removed the statements about expert specialization. This is the right action, but it means the core motivation (that experts learn distinct temporal patterns) remains a hypothesis rather than a demonstrated fact
Dear reviewer ErRD, thank you for the constructive feedback.
Q1. We understand that the wording of the original submission was unclear. What we can add is the selection evidence and the separation that the reviewer asks for. The loss on the validation of the four candidates using MoHETS tiny. As in Table 12, Ho=24 is the lowest:
| Ho | 8 | 16 | 24 | 32 |
|---|---|---|---|---|
| ETTh1 | 1.2764 | 1.0915 | 1.0561 | 1.1023 |
| ETTh2 | 0.4001 | 0.3498 | 0.3452 | 0.3481 |
Table 12 will report these losses in the final version, together with the inference time, in place of the test reporting. Selection is on validation data only, and test metrics appear for the selected configuration in the main tables, which strictly separates validation-based selection from test reporting, as the reviewer requests. The text will be updated accordingly.
Q5 and Q8. We ran the requested controls under the ablation protocol of Table 2 on the five smaller datasets. Entries are MSE; the last column is the average MSE/MAE.
| ETTh1 | ETTh2 | ETTm1 | ETTm2 | Weather | Avg. MSE/MAE | |
|---|---|---|---|---|---|---|
| MoHETS (Table 2) | 0.383 | 0.362 | 0.338 | 0.270 | 0.244 | 0.319/0.350 |
| W_per=0 | 0.392 | 0.367 | 0.351 | 0.278 | 0.239 | 0.325/0.352 |
| Dense FA-FFN | 0.394 | 0.355 | 0.369 | 0.284 | 0.262 | 0.333/0.359 |
| MLP experts, matched | 0.404 | 0.350 | 0.361 | 0.270 | 0.241 | 0.325/0.354 |
| N=4, K=1 | 0.393 | 0.356 | 0.344 | 0.275 | 0.260 | 0.326/0.351 |
| N=4, K=2 | 0.391 | 0.359 | 0.348 | 0.277 | 0.249 | 0.325/0.352 |
| N=8, K=1 | 0.384 | 0.353 | 0.348 | 0.271 | 0.245 | 0.320/0.351 |
| N=8, K=4 | 0.394 | 0.370 | 0.341 | 0.273 | 0.272 | 0.330/0.355 |
| N=16, K=2 | 0.388 | 0.366 | 0.354 | 0.270 | 0.262 | 0.328/0.354 |
MoHETS has the lowest average MSE and MAE of the nine variants.
Q8. With the periodic projection of every FA layer set to zero and frozen, the error is higher on four of the five datasets (average 0.325/0.352 vs. 0.319/0.350). Removing the periodic branch modestly degrades accuracy at this scale.
Q5. What each control isolates:
- Dense FA-FFN, activated parameters matched (sparse routing): the weakest variant (0.333/0.359), worse on four of the five datasets and most on ETTm1 (0.369 vs. 0.338). A dense FA-FFN of equal capacity does not match MoHE.
- MLP experts with the parameter count of an FA expert (d_ff=192; operator at equal capacity): higher average error (0.325/0.354), most on ETTh1 and ETTm1.
- N and K (sparsity setting): N=8, K=2 gives the lowest average. K=1 is close, while fewer or more experts and K=4 bring no gain, and K=4 is 32% slower at inference (46.1 s vs. 34.9 s for K=1 on ETTh1). This result agrees with the top-K study of Time-MoE (Table 7), which reports equal accuracy for K=2 and K=4 at a lower cost for K=2.
The controls are consistent with each design choice: sparse routing over a dense FFN, FA over capacity-matched MLP experts, and N=8 with K=2. The margins are modest on these datasets; the largest effects of the composition are on ECL and Traffic (Table 2: 0.164 and 0.406 for MoHE, 0.174 and 0.428 for homogeneous FA, at least 0.197 and 0.420 with MLP experts). Those two datasets take several hours per run and could not be completed within the discussion period. The final version will include these controls on them.
Q9. We agree: which temporal characteristics drive individual routing decisions is an open question, and the revision does not read the utilization diagnostics as evidence of trend-, seasonality-, or frequency-specific experts. We would separate this interpretability question from the design motivation. Input-dependent specialization is the standard motivation for sparse MoE layers (Shazeer et al., 2017), shared-plus-routed designs (DeepSeekMoE; Dai et al., 2024), and their use in forecasting (Time-MoE; Shi et al., 2024). This literature explains why an MoE is designed to permit specialization. MoHE adds structural heterogeneity to that design space. The paper's claims rest on what is explicit in the architecture: a sequence-mixing shared path and token-wise FA experts, and on its measured effect in the composition ablation. Appendix D shows that routing is non-degenerate and layer-dependent. Whether individual experts acquire interpretable temporal roles remains a separate empirical question, now stated as future work.
Part 1 of 2
We thank the reviewer for the careful and constructive review, which led to substantial changes. Answers follow the order of the questions. Table and section numbers refer to the revised PDF.
Q1. Tables 2, 3, 10, and 12 (Table 11 in the submission) report test-split metrics. Ho=24 was chosen by grid search over {8,16,24,32} on the validation split (validation loss normalized by the chunk size). No test metric influenced any choice, as we stated in both the reproducibility statement and Appendix B.1. Checkpoints were also chosen on validation loss. The Ho=24 row of Table 12 matches the main results because it is the same configuration on the same test split. The sentence the reviewer quotes ("led us to set Ho=24") now states the validation-based selection, and the captions name the split.
Q2. Baselines at L=672: new Table 11 (Appendix C.4) reports eight baselines trained with a 672-step context under the same one-for-all rolling protocol (Liu et al., 2024). On four shared datasets, MoHETS has the lowest average MSE/MAE. Previous works on long-term forecasting typically fix the look-back window to L = 96 for "fair comparisons" in benchmark tasks. We follow Kraus et al. (2026) in the argument that such a practice is a source of distorted comparisons. Protocol behind each number: Table 1 now takes all baselines from one source (Kraus et al., 2026), which reports each model with its own optimized hyperparameters, including the look-back; tuning budgets are as reported.
Q3. Table 1 is single-run, which we note as a practice (e.g., PatchTST paper, page 18, "The results reported in the main text and appendix above are run with the fixed random seed 2021," i.e., single seed). Two seed studies give consistent estimates of run-to-run variability. Appendix C.1 (Figure 5, with error bars) uses five seeds on four datasets and four model sizes. New Table 13 uses three seeds on all seven datasets with the current implementation: the standard deviation is 0.002-0.009 in MSE on seven datasets. Consequence for the claims: the margin over xLSTM-Mixer is within this variability; we removed "consistently achieved superior performance" from the abstract and the state-of-the-art wording from the conclusion. The calendar covariates pathway is part of the proposed model, so Table 1 reports the full system; its effect is isolated in Table 3 under the fixed ablation configuration. This is why Section 4.1 is framed as a system-level comparison, and why we include baselines that rely on covariates (e.g., Timer-XL, AutoTimes).
Q4. Both are less accurate. Loss on the last Ho positions (new Appendix C.6, Table 13: three seeds, seven datasets, identical settings, each variant selected on its own validation loss): the future-only objective is worse on seven datasets in both MSE and MAE; the average rises from 0.312/0.327 to 0.340/0.351. Dense-trained models also reach a 3-10% lower validation loss on the last Ho positions, so the overlapping positions do not dilute the forecast signal. Causal variant: Appendix C.2, Table 10. A decoder-only MoHETS trained with the same objective. Trend in Table 12: the reviewer suggests that the gain from Ho=8 to 24 reflects the growing share of true-forecast supervision. However, the trend is not monotone: at Ho=32, that share is largest, yet accuracy drops. The trend is better explained by the rollout: a larger chunk requires fewer autoregressive steps, limiting error accumulation, but each step must predict further ahead.
Q5. Sweeps over N and K dense FAN-FFN were not run. We agree that these are useful design questions, but they are not entirely unstudied. Time-MoE compares sparse MoE models against dense variants matched to the activated parameter count under the same training data and setup, and also evaluates K ∈ {1,2,4,6,8}, finding only marginal accuracy changes beyond K=2 while inference cost increases. Independently, the FAN paper (Dong et al., 2025) directly compares a standard dense Transformer with a dense Transformer that uses FAN in place of the MLP and reports lower error. Our ablations are designed to address the complementary question not isolated in the literature. For the MLP-expert MoE, the new Table 15 shows that the comparison is unfavorable to MoHETS: the MLP-expert variants have about 22% more parameters, more FLOPs, and higher error rates. On the reviewer's observations: (i) DwConv + MLP is 0.335, not 0.350 (Weather MSE in Table 2 was mistyped; 0.368 instead of 0.268); with the shared expert fixed, FAN experts beat MLP experts on all seven datasets. (ii) We agree that the shared-convolution gain is small; the revised text now names the routed experts as the more influential choice and the shared operator as a contributor. (iii) The variants do share one protocol (Sec 4.2), so these numbers are comparable with each other and not with tuned literature results such as DLinear. MoHETS is 0.165.