- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a method called Expert Loss Integration MoE (ELIMoE), which is a temporal forecasting method that introduces a loss function that separately evaluates the loss for different experts. This prevents the gradient vanishing problem in MoE models, when some experts are rarely used and therefore do not get significant gradient signal. The authors show on benchmark datasets that their models outperform leading time series models, while often having lower computational costs.
Critical strengths
- Relevant and important problem
- The algorithm is extremely simple
- The empirical results are promising and include multiple baselines and datasets
The idea of this paper was interesting and nice (particularly as the idea is very simple). It was slightly surprising to me that experts should be masked over time, which seems to constrain what they can learn. Your explanation makes sense at a high level, but I’m not convinced that this is the “right” thing to do, particularly since we don’t mask MoE experts in this way in other domains. More explanation, intuition, and experiments on this front would be great.
I would like to thank you sincerely for not submitting an AI slop paper. It was easy to read and understand.
Critical weaknesses
- The paper does not crisply analyze (empirically) the effects of different algorithmic components and isolate their impact on the two metrics of interest: error and computation
- It is not clear if the proposed idea extends to all MoE forecasting models, and how it affects computational costs.
- The ablations do not include error bars
Overall, the paper felt a bit light on results and analysis. That being said, I appreciated the very simple idea, and the surprising message that we should constrain experts manually to operate over different time periods. I think the most significant issue with the paper right now is that I think it is muddling the issues of improved accuracy with computational efficiency, and it’s not clear which gains are coming from which design choices, and why certain design choices are made.
I would like to see a few items added to the paper:
(1) You are analyzing the tradeoff between error/accuracy and computation. It would be very natural (and easier to interpret) if at least some of your results were plotted on a 2D axis, where one axis is computation and the other is error. For instance, I think Figure 2 should be a 2D plot, which would better highlight the tradeoff (and the benefits of EliMoE).
(2) I didn’t fully understand the intuition of why the masking should be temporal. You have done an ablation on with/without masking, which I appreciate, but I would like to see a more detailed ablation on why the masking should be temporally-separated over experts in particular. For instance, this seems like it would not allow different experts to learn different global properties of a signal (e.g. different frequency subcomponents in the time series).
(3) I think it would be good to lean in a bit more to the idea that this loss should be applied to every MoE time series model. Can you extend the ablations to include other MoE models beyond MoLE, and show that it helps for all of them? If your proposed architecture still wins over other MoE models that have been extended with this expert-specific loss, then you should explain what is special about your architecture (e.g. is it the linear skip connection? If this is it, you presumably need an ablation with(out) the linear expert, and showing the effect of the linear skip connector and the expert-specific losses, in isolation)
(4) The claims about lower computational complexity are not adequately explained. What is causing this lower complexity, given that you are adding complexity to the loss and gradient computation? Is it that other models use deeper experts? If you hold an existing MoE model fixed and add your expert-specific loss, it should make computational costs higher, but it should also improve accuracy… is that right? If so, I want to see that story highlighted more clearly. For instance, you could plot existing MoE baselines on a computation vs accuracy plot with and without the expert-specific loss, and show how both change as you add the expert-specific loss.
Please note that you are using \citet{foo} incorrectly throughout, where you should be using \citep{foo}.
Questions for authors
See above
2 comments
I think it would be good to lean in a bit more to the idea that this loss should be applied to every MoE time series model. Can you extend the ablations to include other MoE models beyond MoLE, and show that it helps for all of them? If your proposed architecture still wins over other MoE models that have been extended with this expert-specific loss, then you should explain what is special about your architecture (e.g. is it the linear skip connection? If this is it, you presumably need an ablation with(out) the linear expert, and showing the effect of the linear skip connector and the expert-specific losses, in isolation)
To further investigate the generalizability of the proposed objective function across different MoE architectures, we have extended our analysis by including an additional MoE-based model, MoGU[5] with a DLinear[6] backbone. The results, presented in Figure 4 of the revised manuscript, demonstrate that incorporating the proposed auxiliary loss generally improves the forecasting accuracy of both MoLE and MoGU relative to their original objective functions, corresponding to .
An interesting aspect of this comparison is that the two MoE architectures employ fundamentally different base loss functions. MoLE adopts a conventional MoE objective, similar to EliMoE, in which the loss is evaluated on the weighted combination of expert predictions using the squared error loss. In contrast, MoGU employs a probabilistic objective based on the gating-weighted negative log-likelihood of individual expert predictions.
Despite these differences in the underlying objective functions, incorporating the proposed auxiliary loss yields an improvements in forecasting accuracy, with MAE reductions of up to 25% for MoGU and approximately 12% for MoLE. These findings provide further empirical evidence that the benefits of the proposed auxiliary loss are not restricted to EliMoE or a particular choice of base loss function but extend to other MoE forecasting architectures
It is worth noting that EliMoE retains its advantage in computational efficiency over both MoLE and MoGU, even when these models incorporate the proposed auxiliary loss.
[5] Aviv, G., Goldberger, J., & Shavit, Y. (2025). MoGU: Mixture-of-Gaussians with Uncertainty-based Gating for Time Series Forecasting. arXiv preprint arXiv:2510.07459.
[6] Zeng, A., Chen, M., Zhang, L., & Xu, Q. (2023). Are transformers effective for time series forecasting? Proceedings of the AAAI Conference on Artificial Intelligence, 37(9), 11121–11128.
The claims about lower computational complexity are not adequately explained. What is causing this lower complexity, given that you are adding complexity to the loss and gradient computation? Is it that other models use deeper experts? If you hold an existing MoE model fixed and add your expert-specific loss, it should make computational costs higher, but it should also improve accuracy… is that right? If so, I want to see that story highlighted more clearly. For instance, you could plot existing MoE baselines on a computation vs accuracy plot with and without the expert-specific loss, and show how both change as you add the expert-specific loss
We agree that in the previous version of the manuscript, forecasting accuracy and computational efficiency were discussed jointly without sufficiently distinguishing the contributions of the individual methodological components. We have therefore revised the manuscript to clarify their respective roles. Specifically, the proposed objective function primarily contributes to improving forecasting accuracy, whereas the lightweight architecture and partial online learning strategy contribute to reducing computational costs.
To support the former claim, we have added a new mechanistic analysis investigating the impact of the proposed auxiliary loss on expert optimization under low gating weights. The results, presented in Figure 3 of the revised manuscript, demonstrate that incorporating the auxiliary loss strengthens the gradient signals received by the initially low-weight expert and improves its forecasting accuracy.
To isolate the sources of computational efficiency, we have conducted two additional ablation experiments. First, we compare the proposed lightweight architecture, consisting of a linear expert and MLP experts, with an alternative architecture employing Transformer-based experts. Second, we compare EliMoE with and without partial online learning. The results, presented in Figure 6 of the revised manuscript, demonstrate that both the lightweight architecture and the partial online learning strategy contribute to the computational efficiency of EliMoE.
We thank the reviewer for their thorough assessment of our work and for recognizing the practical relevance and simplicity of the proposed method, as well as the promising empirical results. We also appreciate the reviewer’s constructive comments and suggestions regarding the justification of the temporal masking strategy, the generalizability of the proposed objective function, and the need for more comprehensive ablation studies. We have addressed these concerns in the revised manuscript and respond to each point in detail below.
We thank the reviewer for this suggestion. Figure 2 of the revised manuscript now consists of two panels, with forecasting error on the y-axis and computational time on the x-axis, allowing for a more direct assessment of the trade-off between forecasting accuracy and computational efficiency. The results show that EliMoE with partial online learning is positioned in the lower-left region of both panels, achieving low forecasting errors while maintaining the lowest computational cost among all evaluated models, including the newly introduced online learning benchmark, ADAPT-Z [1].
[1] Huang, X., Qiu, S., Du, J., & Yang, C. (2026). Online time series prediction using feature adjustment. International Conference on Learning Representations (ICLR).
The proposed temporal masking strategy is motivated by the assumption that recent observations are particularly informative for forecasting. By adopting a nested masking strategy, we allow some experts to access longer historical segments to capture long-term temporal relationships, while others focus on more recent observations to specialize in short-term dynamics. However, we acknowledge that there is no theoretical guarantee that this masking strategy is optimal.
To investigate alternative masking designs, we have extended our ablation analysis by introducing an additional masking strategy based on disjoint temporal partitions, where each expert receives a distinct segment of the historical data through the auxiliary loss. A potential limitation of this strategy is that only one expert is directly trained on the most recent observations through the auxiliary loss. Nevertheless, somewhat unexpectedly, this alternative strategy also yields promising forecasting results for (γ=0.75).
These findings suggest that the benefits of temporal masking are not necessarily restricted to the proposed nested strategy and that alternative masking designs may also be effective. Further investigation into the optimal design of temporal masking strategies represents an interesting direction for future research.