acceptodds
Under review as a conference paper at ICLR 2027

Fast Training of Mixture-of-Experts for Time Series Forecasting via Expert Loss Integration

Abstract

We propose a novel Mixture-of-Experts framework, referred to as Expert Loss Integration MoE (EliMoE), for time series forecasting. EliMoE addresses the optimization problem arising from small gating weights by incorporating expert-specific losses, which provide each expert with a direct learning signal independent of the gate-assigned weight. Specifically, the overall objective comprises the base forecasting loss and expert-specific losses, allowing individual expert prediction errors to directly influence parameter updates alongside the aggregate forecasting error. The framework also encourages different experts to learn from different temporal segments of the data. The proposed framework is further combined with a partial online learning strategy that enables efficient incremental updates of model parameters. By integrating expert-level loss information with partial online learning, EliMoE improves forecasting performance while retaining computational efficiency. Empirical results across five benchmark datasets from diverse application domains with different sampling frequencies show that the proposed approach generally outperforms state-of-the-art supervised neural forecasting models, including Transformer-based architectures such as PatchTST, as well as zero-shot time-series foundation models such as TimeMoE. Furthermore, ablation studies confirm the effectiveness of the expert-specific loss integration strategy, highlighting its contribution to enhancing predictive performance.

Primary area:
learning on time series and dynamical systems
Keywords:
MoE, Online Learning, Expert Loss, Time Series

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_sYZYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method called Expert Loss Integration MoE (ELIMoE), which is a temporal forecasting method that introduces a loss function that separately evaluates the loss for different experts. This prevents the gradient vanishing problem in MoE models, when some experts are rarely used and therefore do not get significant gradient signal. The authors show on benchmark datasets that their models outperform leading time series models, while often having lower computational costs.

Critical strengths
  • Relevant and important problem
  • The algorithm is extremely simple
  • The empirical results are promising and include multiple baselines and datasets

The idea of this paper was interesting and nice (particularly as the idea is very simple). It was slightly surprising to me that experts should be masked over time, which seems to constrain what they can learn. Your explanation makes sense at a high level, but I’m not convinced that this is the “right” thing to do, particularly since we don’t mask MoE experts in this way in other domains. More explanation, intuition, and experiments on this front would be great.

I would like to thank you sincerely for not submitting an AI slop paper. It was easy to read and understand.

Critical weaknesses
  • The paper does not crisply analyze (empirically) the effects of different algorithmic components and isolate their impact on the two metrics of interest: error and computation
  • It is not clear if the proposed idea extends to all MoE forecasting models, and how it affects computational costs.
  • The ablations do not include error bars

Overall, the paper felt a bit light on results and analysis. That being said, I appreciated the very simple idea, and the surprising message that we should constrain experts manually to operate over different time periods. I think the most significant issue with the paper right now is that I think it is muddling the issues of improved accuracy with computational efficiency, and it’s not clear which gains are coming from which design choices, and why certain design choices are made.

I would like to see a few items added to the paper:

(1) You are analyzing the tradeoff between error/accuracy and computation. It would be very natural (and easier to interpret) if at least some of your results were plotted on a 2D axis, where one axis is computation and the other is error. For instance, I think Figure 2 should be a 2D plot, which would better highlight the tradeoff (and the benefits of EliMoE).

(2) I didn’t fully understand the intuition of why the masking should be temporal. You have done an ablation on with/without masking, which I appreciate, but I would like to see a more detailed ablation on why the masking should be temporally-separated over experts in particular. For instance, this seems like it would not allow different experts to learn different global properties of a signal (e.g. different frequency subcomponents in the time series).

(3) I think it would be good to lean in a bit more to the idea that this loss should be applied to every MoE time series model. Can you extend the ablations to include other MoE models beyond MoLE, and show that it helps for all of them? If your proposed architecture still wins over other MoE models that have been extended with this expert-specific loss, then you should explain what is special about your architecture (e.g. is it the linear skip connection? If this is it, you presumably need an ablation with(out) the linear expert, and showing the effect of the linear skip connector and the expert-specific losses, in isolation)

(4) The claims about lower computational complexity are not adequately explained. What is causing this lower complexity, given that you are adding complexity to the loss and gradient computation? Is it that other models use deeper experts? If you hold an existing MoE model fixed and add your expert-specific loss, it should make computational costs higher, but it should also improve accuracy… is that right? If so, I want to see that story highlighted more clearly. For instance, you could plot existing MoE baselines on a computation vs accuracy plot with and without the expert-specific loss, and show how both change as you add the expert-specific loss.

Please note that you are using \citet{foo} incorrectly throughout, where you should be using \citep{foo}.

Questions for authors

See above

2 comments

We thank the reviewer for their thorough assessment of our work and for recognizing the practical relevance and simplicity of the proposed method, as well as the promising empirical results. We also appreciate the reviewer’s constructive comments and suggestions regarding the justification of the temporal masking strategy, the generalizability of the proposed objective function, and the need for more comprehensive ablation studies. We have addressed these concerns in the revised manuscript and respond to each point in detail below.

You are analyzing the tradeoff between error/accuracy and computation. It would be very natural (and easier to interpret) if at least some of your results were plotted on a 2D axis, where one axis is computation and the other is error. For instance, I think Figure 2 should be a 2D plot, which would better highlight the tradeoff (and the benefits of EliMoE).

We thank the reviewer for this suggestion. Figure 2 of the revised manuscript now consists of two panels, with forecasting error on the y-axis and computational time on the x-axis, allowing for a more direct assessment of the trade-off between forecasting accuracy and computational efficiency. The results show that EliMoE with partial online learning is positioned in the lower-left region of both panels, achieving low forecasting errors while maintaining the lowest computational cost among all evaluated models, including the newly introduced online learning benchmark, ADAPT-Z [1].

[1] Huang, X., Qiu, S., Du, J., & Yang, C. (2026). Online time series prediction using feature adjustment. International Conference on Learning Representations (ICLR).

I didn’t fully understand the intuition of why the masking should be temporal. You have done an ablation on with/without masking, which I appreciate, but I would like to see a more detailed ablation on why the masking should be temporally-separated over experts in particular. For instance, this seems like it would not allow different experts to learn different global properties of a signal (e.g. different frequency subcomponents in the time series).

The proposed temporal masking strategy is motivated by the assumption that recent observations are particularly informative for forecasting. By adopting a nested masking strategy, we allow some experts to access longer historical segments to capture long-term temporal relationships, while others focus on more recent observations to specialize in short-term dynamics. However, we acknowledge that there is no theoretical guarantee that this masking strategy is optimal.

To investigate alternative masking designs, we have extended our ablation analysis by introducing an additional masking strategy based on disjoint temporal partitions, where each expert receives a distinct segment of the historical data through the auxiliary loss. A potential limitation of this strategy is that only one expert is directly trained on the most recent observations through the auxiliary loss. Nevertheless, somewhat unexpectedly, this alternative strategy also yields promising forecasting results for (γ=0.75)(\gamma=0.75).

These findings suggest that the benefits of temporal masking are not necessarily restricted to the proposed nested strategy and that alternative masking designs may also be effective. Further investigation into the optimal design of temporal masking strategies represents an interesting direction for future research.

I think it would be good to lean in a bit more to the idea that this loss should be applied to every MoE time series model. Can you extend the ablations to include other MoE models beyond MoLE, and show that it helps for all of them? If your proposed architecture still wins over other MoE models that have been extended with this expert-specific loss, then you should explain what is special about your architecture (e.g. is it the linear skip connection? If this is it, you presumably need an ablation with(out) the linear expert, and showing the effect of the linear skip connector and the expert-specific losses, in isolation)

To further investigate the generalizability of the proposed objective function across different MoE architectures, we have extended our analysis by including an additional MoE-based model, MoGU[5] with a DLinear[6] backbone. The results, presented in Figure 4 of the revised manuscript, demonstrate that incorporating the proposed auxiliary loss generally improves the forecasting accuracy of both MoLE and MoGU relative to their original objective functions, corresponding to (γ=1)(\gamma=1).

An interesting aspect of this comparison is that the two MoE architectures employ fundamentally different base loss functions. MoLE adopts a conventional MoE objective, similar to EliMoE, in which the loss is evaluated on the weighted combination of expert predictions using the squared error loss. In contrast, MoGU employs a probabilistic objective based on the gating-weighted negative log-likelihood of individual expert predictions.

Despite these differences in the underlying objective functions, incorporating the proposed auxiliary loss yields an improvements in forecasting accuracy, with MAE reductions of up to 25% for MoGU and approximately 12% for MoLE. These findings provide further empirical evidence that the benefits of the proposed auxiliary loss are not restricted to EliMoE or a particular choice of base loss function but extend to other MoE forecasting architectures

It is worth noting that EliMoE retains its advantage in computational efficiency over both MoLE and MoGU, even when these models incorporate the proposed auxiliary loss.

[5] Aviv, G., Goldberger, J., & Shavit, Y. (2025). MoGU: Mixture-of-Gaussians with Uncertainty-based Gating for Time Series Forecasting. arXiv preprint arXiv:2510.07459.

[6] Zeng, A., Chen, M., Zhang, L., & Xu, Q. (2023). Are transformers effective for time series forecasting? Proceedings of the AAAI Conference on Artificial Intelligence, 37(9), 11121–11128.

The claims about lower computational complexity are not adequately explained. What is causing this lower complexity, given that you are adding complexity to the loss and gradient computation? Is it that other models use deeper experts? If you hold an existing MoE model fixed and add your expert-specific loss, it should make computational costs higher, but it should also improve accuracy… is that right? If so, I want to see that story highlighted more clearly. For instance, you could plot existing MoE baselines on a computation vs accuracy plot with and without the expert-specific loss, and show how both change as you add the expert-specific loss

We agree that in the previous version of the manuscript, forecasting accuracy and computational efficiency were discussed jointly without sufficiently distinguishing the contributions of the individual methodological components. We have therefore revised the manuscript to clarify their respective roles. Specifically, the proposed objective function primarily contributes to improving forecasting accuracy, whereas the lightweight architecture and partial online learning strategy contribute to reducing computational costs.

To support the former claim, we have added a new mechanistic analysis investigating the impact of the proposed auxiliary loss on expert optimization under low gating weights. The results, presented in Figure 3 of the revised manuscript, demonstrate that incorporating the auxiliary loss strengthens the gradient signals received by the initially low-weight expert and improves its forecasting accuracy.

To isolate the sources of computational efficiency, we have conducted two additional ablation experiments. First, we compare the proposed lightweight architecture, consisting of a linear expert and MLP experts, with an alternative architecture employing Transformer-based experts. Second, we compare EliMoE with and without partial online learning. The results, presented in Figure 6 of the revised manuscript, demonstrate that both the lightweight architecture and the partial online learning strategy contribute to the computational efficiency of EliMoE.

Reviewer_hVAFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EliMoE combines a linear expert with MLP experts and a soft gating network for forecasting from lagged observations. Its objective mixes the aggregate forecast MAE with directly supervised expert losses. Nested masks restrict the auxiliary losses of different MLPs to progressively more recent training examples, while all experts contribute to the full-window aggregate loss. The method is combined with partial online learning. Evaluation includes five fixed-scheme benchmark collections with ten-seed results, Saugeen rolling-window experiments studying loss weights, masking, and transfer of the auxiliary objective to MoLE, and a fixed-scheme analysis of the number of experts.

Critical strengths
  1. The auxiliary objective addresses an explicit optimization issue in a simple, reusable way. The gradient calculation explains why an expert with a small gate weight receives weak supervision from the mixture loss, and how a direct expert loss adds an unscaled learning signal. The nested masks provide a concrete recency-based division of auxiliary supervision. The objective is inexpensive to implement and is not tied to a particular large forecasting backbone, which gives the idea practical value.

  2. The experiments support accuracy benefits within the tested settings. The fixed-scheme study reports ten-seed results over multiple domains and frequencies. On Saugeen, Table 2 compares the full objective with both the base-loss-only setting and an unmasked auxiliary-loss setting. Table 3 also tests the loss in MoLE. These are relevant controls supporting the utility of expert supervision, even though they do not by themselves demonstrate faster optimization or broad online robustness.

Critical weaknesses
  1. The timing evidence does not isolate the claimed fast-training mechanism. Appendix C uses different implementations and batching for EliMoE and FFNN: PyTorch versus GluonTS, batch sizes 256 versus 32, and different definitions of batches per epoch. The rolling study also introduces warm starts and separately tuned update-window settings. The resulting speed differences can reflect implementation and optimization-budget choices, rather than expert-loss integration. Since efficiency is a central contribution, evidence that the loss reaches comparable accuracy faster under a matched implementation/update budget is needed to support the strongest claim.

  2. The optimization and specialization interpretation is supported by a narrow online study. The main loss/masking and MoLE-transfer analyses use one river-flow series, with thirty test observations and thirty preceding validation observations. Repeating random seeds measures optimization variability, but does not establish robustness across forecasting regimes. The partial-update schedule is also not clearly connected to the architecture: the Introduction describes updating the gating network and a subset of expert parameters while refreshing shared representational components less frequently, but Section 4.2 describes only warm starting and separate hyperparameters for the initial and subsequent windows, and the architecture in Section 3 contains no identified shared representation. Finally, at γ=0 the stated auxiliary objective contains no gate weights, so it supplies no training gradient to the router; the interpretation of the γ=0 ablation therefore needs explanation. These limitations leave uncertainty about whether the gains primarily reflect stronger expert optimization, recency regularization, or the online-update procedure.

Questions for authors
  1. Clarification and controlled timing evidence: Which parameters are updated or frozen at each online step, and how many optimizer updates are performed for each model? Is there a matched-framework comparison with the same warm-start and update budget that isolates the timing effect of expert-loss integration? If not, which narrower efficiency claim is supported by the reported experiment?

  2. Clarification of the objective and ablations: At γ=0, how are router parameters initialized, trained, and used for prediction, given that L_aux does not depend on them? The same question applies to MoLE in Table 3, where γ=0 yields an MAE of 7.33, better than the standard γ=1 formulation (8.16), apparently without router training. If an untrained router performs this well, what role does learned gating play in the reported gains, as opposed to stronger individual experts? Please also distinguish what the Saugeen ablations establish about accuracy from any evidence they provide about convergence speed or expert specialization.

2 comments

We thank the reviewer for their constructive feedback and for recognizing the practical value and accuracy benefits of the proposed objective. We have revised the manuscript, clarified methodological details, and conducted additional experiments to address the concerns raised. We respond to each point below.

Which parameters are updated or frozen at each online step, and how many optimizer updates are performed for each model?

We first clarify that the only two hyperparameters that differ between the initial and subsequent training windows of EliMoE are the learning rate and the window size. All remaining hyperparameters, including the number of neurons, are kept fixed across training windows.

Regarding the number of optimizer updates, we acknowledge that it may differ across models. As the reviewer correctly points out, differences in batch sizes between EliMoE and FFNN may affect computational time. However, the batch size alone does not determine the number of optimizer updates, particularly because the two implementations employ different definitions of an epoch. In our PyTorch implementation, the model processes the entire training dataset in each epoch, whereas the GluonTS implementation of FFNN relies on randomly sampled training examples with a predefined number of batches per epoch.

For instance, in the fixed forecasting scheme, the Saugeen dataset comprises 23,711 training observations. With an input length of 9 and a forecasting horizon of 30, this results in 23,673 input–output training pairs. With a batch size of 256, EliMoE performs 93 optimizer updates per epoch, whereas FFNN, with a batch size of 32 and a predefined number of 50 batches per epoch, performs 50 optimizer updates. Thus, despite employing a larger batch size, EliMoE performs more optimizer updates and processes substantially more training examples per epoch than FFNN.

Is there a matched-framework comparison with the same warm-start and update budget that isolates the timing effect of expert-loss integration? If not, which narrower efficiency claim is supported by the reported experiment?

In the original manuscript, we discussed forecasting accuracy and computational efficiency jointly, making it difficult for readers to distinguish which components of the proposed methodology contribute to improved forecasting accuracy and which are responsible for reduced computational cost.

We have revised the manuscript to explicitly distinguish these contributions. In particular, we clarify that the proposed objective function is designed to improve forecasting accuracy. To support this claim, we provide ablation and mechanistic studies investigating the contributions of its individual components:

  • Figure 3: Monitoring the gradient magnitudes of experts with small gating weights and demonstrating how the incorporation of the auxiliary loss improves their predictive accuracy.
  • Table 5: Investigating the impact of the weighting parameter γ\gamma and different temporal masking strategies on forecasting accuracy.

Similarly, we explicitly identify the components responsible for reducing computational cost, namely the lightweight expert architecture and the adoption of partial online learning. The individual contributions of these two components to computational efficiency are investigated in Figure 6 of the revised manuscript.

These revisions provide a clearer distinction between the mechanisms underlying forecasting accuracy and computational efficiency. In particular, we no longer attribute computational efficiency to expert-loss integration but rather to the lightweight expert architecture and partial online learning strategy.

At γ=0\gamma=0, how are router parameters initialized, trained, and used for prediction, given that L_aux does not depend on them? The same question applies to MoLE in Table 3, where γ=0 yields an MAE of 7.33, better than the standard γ=1 formulation (8.16), apparently without router training. If an untrained router performs this well, what role does learned gating play in the reported gains, as opposed to stronger individual experts?

When γ=0\gamma=0, the gating network is initialized randomly in the same manner as for the other values of γ\gamma. However, since the objective reduces to the auxiliary loss, which is independent of the gating network parameters, the router receives no training gradient from the forecasting objective. During prediction, the gating network is still used to aggregate the expert outputs, but its weights are determined by its initial, rather than learned, parameters.

The improvement observed at γ=0\gamma=0 relative to γ=1\gamma=1 suggests that direct expert supervision alone can improve forecasting accuracy, even without learned gating. However, this does not imply that learned gating is redundant.

To further investigate this observation, we have included MoGU [5] alongside MoLE in the revised manuscript. As shown in Figure 4, both models outperform their respective standard formulations (γ=1\gamma=1) when γ=0\gamma=0. Nevertheless, MoLE and MoGU achieve their lowest MAE at γ=0.5\gamma=0.5 and γ=0.75\gamma=0.75, respectively, when both the aggregate and auxiliary losses are incorporated.

Finally, we note that the numerical results for MoLE differ slightly from those reported in the original submission due to updates to the implementation. Nevertheless, the finding that γ=0\gamma=0 outperforms γ=1\gamma=1 remains consistent across both versions.

Please also distinguish what the Saugeen ablations establish about accuracy from any evidence they provide about convergence speed or expert specialization.

The Saugeen ablations are primarily designed to investigate the impact of the proposed objective function including the weighting parameter γ\gamma, and temporal masking strategies on forecasting accuracy rather than convergence speed. While the results demonstrate the accuracy benefits of direct expert supervision and temporal masking, they do not provide direct empirical evidence supporting the central motivation of the proposed objective, namely, mitigating the weak-gradient problem experienced by experts with small gating weights.

To address this limitation, we have included an additional mechanistic experiment in Figure 3 of the revised manuscript, where we monitor the gradient magnitude and predictive accuracy of an expert receiving small gating weights. The results demonstrate that auxiliary supervision strengthens the learning signal and improves individual expert accuracy, thereby providing empirical evidence supporting the proposed optimization mechanism.

[5] Aviv, G., Goldberger, J., & Shavit, Y. (2025). MoGU: Mixture-of-Gaussians with Uncertainty-based Gating for Time Series Forecasting. arXiv preprint arXiv:2510.07459.

Reviewer_ZWjdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EliMoE, a Mixture-of-Experts (MoE) framework designed to address the issue of weak learning signals received by experts with low gating weights in time-series forecasting. EliMoE comprises one linear expert, multiple Multilayer Perceptron (MLP) experts, and a gating network. Its training objective combines the global mixture forecasting loss with auxiliary expert-specific losses. Additionally, a nested temporal masking strategy encourages different experts to specialize in training segments of varying lengths. Experiments across five benchmarks demonstrate strong forecasting performance and computational efficiency, supplemented by ablation studies validating individual components.

Critical strengths
  1. Lightweight and Unscaled Optimization Design: The introduction of expert-specific losses that are unscaled by current gating weights offers a simple, architecture-agnostic mechanism to improve gradient flow and mitigate the under-optimization of low-weight experts in MoE models.
  2. Effective Multi-Scale Temporal Adaptation: The nested temporal masking strategy provides an intuitive and effective way to guide different experts toward capturing patterns across diverse temporal horizons (from full history to recent segments).
Critical weaknesses
  1. Lack of Mechanistic Verification: While the paper demonstrates improvements in final forecasting accuracy, it provides limited empirical evidence regarding the core underlying mechanism. It remains unverified whether experts with low gating weights are genuinely undertrained in standard settings, or whether the proposed auxiliary losses successfully drive true semantic/functional specialization among experts.
  2. Limited Baseline Scope: The empirical comparisons are primarily restricted to FFNN and MoLE. The evaluation would be significantly strengthened by including more recent online adaptation baselines (such as ADAPT-Z). Moreover, although Moirai-MoE is cited in the literature review, it is omitted from the comparative experiments, leaving the framework's competitiveness against contemporary state-of-the-art MoE models open to question.
Questions for authors
  1. Experimental Fairness in Table 7: Why does Table 7 permit EliMoE to use a significantly shorter historical window for subsequent updates, while baseline models such as FFNN and MoLE are constrained to at least 730 observations? Do all compared methods operate under identical warm-start treatments, update step budgets, and window-size constraints?
  2. Evaluation Protocol Consistency: Do the fixed-scheme and online forecasting experiments evaluate predictions over the exact same future intervals? Additionally, do all compared methods have access to identical information sets at each forecasting step?
2 comments

We thank the reviewer for their careful reading of our manuscript and for their constructive comments and suggestions, particularly regarding the mechanistic verification and comparisons with stronger baselines. We appreciate the reviewer’s recognition of the lightweight nature of the proposed method, as well as the usefulness of the masking mechanism. Below, we address each concern in detail and describe the corresponding revisions and clarifications.

Why does Table 7 permit EliMoE to use a significantly shorter historical window for subsequent updates, while baseline models such as FFNN and MoLE are constrained to at least 730 observations?

The different window size ranges reflect the distinct learning strategies. In EliMoE, the model is initially trained from scratch using a relatively long historical window. At each subsequent forecasting step, partial online learning allows the model to reuse previously learned parameters. Consequently, shorter training windows may be sufficient to adapt the model to newly available information without requiring complete retraining.

In contrast, FFNN is retrained from scratch at each forecasting step, meaning that shorter training windows may not provide sufficient historical information to effectively estimate the model parameters. This motivates the choice of a longer window size range for FFNN, consistent with the range used for the initial training of EliMoE.

Importantly, we do not impose a specific window size on either model. Instead, the optimal window size within each search range is selected using the Tree-structured Parzen Estimator (TPE). Thus, the different search ranges are motivated by the requirements of the respective learning strategies rather than an intentional restriction on the benchmark model.

Do all compared methods operate under identical warm-start treatments, update step budgets, and window-size constraints?

In the rolling-window forecasting setup, only EliMoE incorporates the proposed partial online learning strategy, which combines warm-start initialization with window-specific hyperparameters. Extending this strategy to the benchmark models would require modifying several components of their original training procedures, potentially compromising consistency with their original implementations. We acknowledge that this difference in training strategies introduces an asymmetry in terms of computational cost. To address this concern and provide a more comprehensive comparison, we have expanded our benchmark analysis to include ADAPT-Z[1] with iTransformer[2] as its backbone. This additional benchmark allows us to assess the computational efficiency and forecasting accuracy of EliMoE against an existing online learning method, rather than exclusively against models that rely on conventional rolling-window retraining.

The runtime comparison demonstrates that EliMoE is the fastest among all evaluated models, with a runtime of approximately 2 seconds, outperforming even the online learning method ADAPT-Z. Furthermore, the comparison of EliMoE with and without partial online learning reveals that, even under the conventional rolling-window estimation strategy, EliMoE achieves a runtime of 13.79 seconds, which is substantially lower than that of the fastest benchmark model (27.82 seconds). These findings indicate that the computational efficiency of EliMoE is not solely attributable to the partial online learning strategy but also to its lightweight architecture. The corresponding results are presented in Figures 2 and 6 of the revised manuscript.

[1] Huang, X., Qiu, S., Du, J., & Yang, C. (2026). Online time series prediction using feature adjustment. International Conference on Learning Representations (ICLR).

[2] Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., & Long, M. (2024). iTransformer: Inverted transformers are effective for time series forecasting. International Conference on Learning Representations (ICLR).

Do the fixed-scheme and online forecasting experiments evaluate predictions over the exact same future intervals? Additionally, do all compared methods have access to identical information sets at each forecasting step?

Yes, all methods use a predetermined set of lags to generate forecasts over a specified horizon. For each dataset, both the number of lags and the forecast horizon strictly follow the specifications in [3]. The exact values are now provided in Table 2 of the revised manuscript.

The rolling-window forecasting experiment is conducted on the Saugeen dataset, for which the forecast horizon is 30 in the fixed-window forecasting scheme. In the rolling-window setup, we perform one-step-ahead forecasting, with the model updated as each new observation becomes available. Accordingly, 30 successive one-step-ahead forecasts are generated over the same evaluation period, with the model updated at each forecasting step.

Additional Analysis

Regarding the concerns raised in the Critical Weaknesses section, we have added a new mechanistic analysis to investigate how the proposed auxiliary loss affects expert optimization under low gating weights. Specifically, we initialize the gating mechanism such that one expert receives a very small gating weight and subsequently monitor the gradient norm and MAE of that expert at each optimization step.

The results, presented in Figure 3 of the revised manuscript, show that the gradient norms for (γ=1)(\gamma=1), which corresponds to the standard MoE objective, range from (3.84×10−5)(3.84\times10^{-5}) to (7.42×10−3)(7.42\times10^{-3}), whereas for (γ=0.5)(\gamma=0.5), they range from (1.56×10−2)(1.56\times10^{-2}) to (1.49)(1.49). These results demonstrate that incorporating the auxiliary loss substantially strengthens the gradient signals received by the initially low-weight expert, mitigating its under-optimization. Furthermore, the expert-specific MAE increases over the course of training for (γ=1)(\gamma=1), whereas it decreases for (γ=0.5)(\gamma=0.5). Taken together, these findings provide direct mechanistic evidence that the proposed auxiliary loss improves gradient flow and facilitates the optimization of experts that initially receive low gating weights.

Additionally, we have expanded our benchmark analysis by including three new models: Moirai-MoE [4] and ADAPT-Z, as kindly suggested by the reviewer, and an additional MoE-based model. The latter, alongside MoLE, is used to further investigate the generalizability of the proposed objective function across different MoE architectures. The results demonstrate that EliMoE remains competitive against the newly introduced benchmarks and that incorporating the proposed auxiliary loss improves the forecasting accuracy of other MoE architectures. These findings provide further evidence that the benefits of the proposed objective function extend beyond the EliMoE architecture.

[3] Godahewa, R., Bergmeir, C., Webb, G. I., Hyndman, R. J., & Montero-Manso, P. (2021). Monash Time Series Forecasting Archive. Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track.

[4] Liu, X., Liu, J., Woo, G., Aksu, T., Liang, Y., Zimmermann, R., Liu, C., Savarese, S., Xiong, C., & Sahoo, D. (2024). Moirai-MoE: Empowering time series foundation models with sparse mixture of experts. arXiv preprint arXiv:2410.10469.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.