acceptodds
Withdrawn from ICLR 2027

Multi-Mask Online Ensemble Time Series Forecasting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Time series forecasting is increasingly required to operate on non-stationary data streams, where models must adapt online as temporal patterns evolve. Although Transformers have achieved strong performance in offline forecasting by relying on flexible all-to-all attention, such weak inductive bias can be less reliable in online settings. We identify three core mismatches: Over-extended receptive fields that capture spurious correlations. Single static architectures that fail to decouple heterogeneous temporal patterns. Slow adaptation to abrupt concept drift. To address these challenges, we propose MiMe, a Multi-Mask online ensemble framework for time series forecasting. MiMe introduces Pattern-induced Masking to impose diverse temporal inductive biases on self-attention, thereby constraining receptive fields for heterogeneous temporal patterns. It further employs Shared Backbone Training to integrate multiple masked forecasters within a single Transformer backbone, enabling parameter-efficient decoupling of heterogeneous patterns. Finally, online ensembling dynamically reweights masked forecasters through online convex optimization, enabling rapid strategy-level adaptation to concept drift with an regret guarantee. Extensive experiments demonstrate MiMe's strong forecasting performance and robustness against state-of-the-art baselines. Code is available at \https://github.com/anonymous7538/AnonProj.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series, Forecasting, Online Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uCnjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes MiMe, an online forecasting multi-task ensemble framework. The framework consists of a shared backbone with multiple masked attention components to address the heterogeneous nature of temporal patterns, particularly as the series progresses and introduces new patterns. The different masked attentions include Global, Periodic, adaptive, directional (causal) and local for addressing different receptive to enhance robustness. The usage of the different masks depends on corresponding weights, which are optimized with convex optimization and an interesting guarantee with respect to regret.

Critical strengths
  • Online learning is an interesting and practical challenge. Most real TSF relies on streaming data and adaptation to new examples; creating static models may include additional overhead and may introduce significant changes in the model performance, whereas online models can introduce soft adaptations.

  • From looking at the code, it seems that the implementation avoids the data leakage issue addressed in previous works [1]

  • The paper's flow is coherent and easy to follow; these cases have become rarer with the introduction of AI tools for writing.

  • The utilization of Regret in the framework development

  • Most masks are not hard-coded designs; for instance, periodic masks select a relevant frequency (however, please see my question below about the local mask)

[1] Fast and Slow Streams for Online Time Series Forecasting Without Information Leakage

Critical weaknesses
  • Limited novelty; the majority of the framework relies on various elements that were previously used in TSF, namely, various attention masks, component reweighting/MoE, ensembles etc.

  • Most works in online learning for TSF, show results for the 1 and 48 horizons. In my opinion, the paper would benifit from the completeness of the results.

  • [1] addressed a major issue with data leakage in online learning TSF frameworks. This work compares against methods that suffer from this issue, OneNet and FSNET, yet has not discussed how they mitigated this issue in their evaluation protocol.

  • It seems that much of the setup has been reproduced, which is ok since an adequate setup is required due to the bullet point above, but this at the same time might have been for the methods' benefit. I would suggest the authors at least provide comparisons with different lookback sizes.

  • Time series foundation models (TSFM) have proven themselves to be strong zero-shot forecasters; this approach may challenge any form of supervised training, including online learning. In my opinion, the work would benefit from a comparison to recent TSFMs (e.g., Chronos-2) to prove that online supervised learning is still a relevant alternative. If the authors wish to proceed with such analysis, it is important to note that some TSFMs might include some of the datasets from LTSF (e.g. Electricity/Solar) in their pretraining corpora; thus, this analysis should be presented with caution.

Questions for authors

Questions

  • Table2 suggests that adding more diverse masks is positively correlated with performance; can the authors add insights into what may happen if we add additional different masks? Can the performance become bounded by a certain number?

  • How is the radius parameter for the local mask selected?

Overall, the problem is indeed interesting; the design is relatively simple and intuitive. I would reconsider changing my score if the majority of the weaknesses/questions could be adequately addressed. Most importantly, with respect to data leakage and evaluations.

Reviewer_4sHSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes MiMe, a multi-mask online ensemble framework for time series forecasting under non-stationary data streams. The method introduces five attention masks (global, periodic, contextual, causal, and local) to impose different temporal inductive biases on Transformers and capture heterogeneous temporal dependencies. Multiple masked forecasters share a Transformer backbone while maintaining separate prediction heads to reduce parameter redundancy. During online forecasting, MiMe employs online gradient descent (OGD) to dynamically adjust the ensemble weights and adapt to evolving temporal patterns. Experiments are conducted on seven time series forecasting datasets, including comparisons with existing baselines, ablation studies on mask combinations, expert diversity, online ensemble strategies, backbone efficiency, cross-dataset transfer, and hyperparameter sensitivity. The results demonstrate competitive forecasting performance and parameter efficiency.

Critical strengths
  1. Comprehensive empirical evaluation supporting the effectiveness of the proposed framework. The paper provides a relatively thorough empirical evaluation, including comparisons with a broad range of forecasting baselines and ablation studies on mask combinations, expert diversity, and online ensemble strategies. The appendix further includes analyses of parameter efficiency, ensemble-weight evolution, cross-dataset transfer, and hyperparameter sensitivity. These experiments provide useful evidence for the effectiveness of the proposed framework and help clarify the contributions of its individual components.
  2. Reasonable framework design with practical value. Combining complementary temporal inductive biases through different attention masks and dynamically adjusting their contributions via online ensembling is an intuitive and well-motivated approach. The shared-backbone design reduces parameter redundancy compared with independently trained Transformers, offering a reasonable balance between forecasting performance and parameter efficiency. The cross-dataset transfer experiments also provide encouraging evidence of adaptability under distribution shifts.
Critical weaknesses
  1. Limited methodological novelty. My primary concern is that the methodological contribution is relatively incremental. The three core components of MiMe—structured attention masks, shared-backbone multi-head training, and online gradient-based ensemble weighting—are all built upon established techniques. Although their integration is reasonable and empirically effective, the paper does not appear to introduce a substantially new mechanism for temporal pattern modeling or online adaptation. The experiments demonstrate the benefits of heterogeneous masks and their combination, but the paper does not sufficiently establish the distinctive methodological advantages of the proposed framework over alternative approaches based on heterogeneous forecasters and adaptive online aggregation. Consequently, the overall contribution appears to be primarily an effective integration of existing techniques rather than a significant methodological advance.
  2. Relatively modest improvements compared with the additional methodological complexity. Although MiMe achieves competitive results across multiple datasets, the main experiments report average improvements of approximately 3.83% in MSE and 3.53% in MAE over the second-best baselines. These gains are positive but relatively modest, considering the additional masked forecasting branches, prediction heads, and online ensemble updates introduced by the framework. The parameter-efficiency analysis and cross-dataset transfer experiments demonstrate additional practical benefits, which should be acknowledged. Nevertheless, given the relatively incremental methodological contribution, the modest gains on the main benchmarks leave some uncertainty about the overall practical advantage of the proposed framework, particularly without a direct comparison of online computational costs.
Questions for authors

Q1. Methodological novelty. Could the authors further clarify the main methodological contribution of MiMe beyond combining heterogeneous attention masks, shared-backbone training, and online ensemble optimization? In particular, what distinguishes MiMe conceptually from a straightforward ensemble of differently masked forecasters with adaptive online weighting?

Q2. Choice of forecasting horizons. The experimental evaluation only considers relatively short forecasting horizons, H = {12, 24}, without evaluating commonly adopted longer horizons such as H = {96, 192, 336, 720}. Could the authors explain the motivation for this choice? Given that MiMe aims to capture heterogeneous temporal dependencies through different receptive-field biases, evaluating longer forecasting horizons would help establish the generality of the proposed approach.

Q3. Computational efficiency of strategy-level adaptation. A central motivation of MiMe is to enable rapid strategy-level adaptation through online ensembling. However, the framework evaluates multiple masked forecasters at each time step and performs online parameter updates for these forecasters. Although backbone sharing reduces parameter redundancy, it does not necessarily reduce the computational overhead of executing multiple masked attention branches. Given the relatively modest improvements on the main forecasting benchmarks, could the authors provide comparisons of inference latency, online update time, and computational cost against strong baselines? Such results would help clarify the practical trade-off between forecasting accuracy, adaptation capability, and computational efficiency.

Minor Comments

  1. Inconsistent highlighting in Table 2. The blue underlining used to indicate the second-best results in Table 2 appears incomplete or inconsistent. The authors should carefully check the highlighting and ensure that all second-best values are correctly marked according to the table's formatting convention.
Reviewer_Yed6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes MiMe, an online ensemble method. The inductive bias of the time series prediction model needs to change when dealing with different data. Therefore, the author manually crafted five masks, combined with a unified transformer backbone and the online gradient processing of the prediction head, to assign weights to and ensemble the predictions of the five masks. The method was tested on multiple datasets and showed some improvement.

Critical strengths
  1. Clearly state the problem and motivation. Each layer, from receptive field to structure to adaptation speed, corresponds to a module, ensuring a complete story.

  2. The core idea is concise and the implementation cost is low.

  3. The coverage of additional experiments is entensive.

Critical weaknesses
  1. The paper selected favorable data for calculation. The MAE of 3.53% in Table 1 is the arithmetic mean of six datasets excluding Traffic. I don't know how the value of MSE is derived.

  2. Table 2 and 3 does not reflect the effect of MiMe proposed by the author. It seems that the improvement is only from 1 MG to 5 MG. 1 MG ——> 5 MG: 1.133 ——> 0.895 (a decrease of 21%) 5 MG ——> MiMe proposed by the author: 0.895 ——> 0.890 (a decrease of 0.5%)

  3. The code implementation of the paper does not seem to align with the description in the paper. The periodic mask in the main entry always uses patch_T=2, and after local replication, it appears that the weights of the five masks are all around 20%, without reflecting the role of weight allocation

  4. Different attention masks will undoubtedly introduce varying inductive biases, but I find this approach less elegant. If there are more types of masks, there's a risk of overfitting.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.