- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work proposes MiMe, an online forecasting multi-task ensemble framework. The framework consists of a shared backbone with multiple masked attention components to address the heterogeneous nature of temporal patterns, particularly as the series progresses and introduces new patterns. The different masked attentions include Global, Periodic, adaptive, directional (causal) and local for addressing different receptive to enhance robustness. The usage of the different masks depends on corresponding weights, which are optimized with convex optimization and an interesting guarantee with respect to regret.
Critical strengths
-
Online learning is an interesting and practical challenge. Most real TSF relies on streaming data and adaptation to new examples; creating static models may include additional overhead and may introduce significant changes in the model performance, whereas online models can introduce soft adaptations.
-
From looking at the code, it seems that the implementation avoids the data leakage issue addressed in previous works [1]
-
The paper's flow is coherent and easy to follow; these cases have become rarer with the introduction of AI tools for writing.
-
The utilization of Regret in the framework development
-
Most masks are not hard-coded designs; for instance, periodic masks select a relevant frequency (however, please see my question below about the local mask)
[1] Fast and Slow Streams for Online Time Series Forecasting Without Information Leakage
Critical weaknesses
-
Limited novelty; the majority of the framework relies on various elements that were previously used in TSF, namely, various attention masks, component reweighting/MoE, ensembles etc.
-
Most works in online learning for TSF, show results for the 1 and 48 horizons. In my opinion, the paper would benifit from the completeness of the results.
-
[1] addressed a major issue with data leakage in online learning TSF frameworks. This work compares against methods that suffer from this issue, OneNet and FSNET, yet has not discussed how they mitigated this issue in their evaluation protocol.
-
It seems that much of the setup has been reproduced, which is ok since an adequate setup is required due to the bullet point above, but this at the same time might have been for the methods' benefit. I would suggest the authors at least provide comparisons with different lookback sizes.
-
Time series foundation models (TSFM) have proven themselves to be strong zero-shot forecasters; this approach may challenge any form of supervised training, including online learning. In my opinion, the work would benefit from a comparison to recent TSFMs (e.g., Chronos-2) to prove that online supervised learning is still a relevant alternative. If the authors wish to proceed with such analysis, it is important to note that some TSFMs might include some of the datasets from LTSF (e.g. Electricity/Solar) in their pretraining corpora; thus, this analysis should be presented with caution.
Questions for authors
Questions
-
Table2 suggests that adding more diverse masks is positively correlated with performance; can the authors add insights into what may happen if we add additional different masks? Can the performance become bounded by a certain number?
-
How is the radius parameter for the local mask selected?
Overall, the problem is indeed interesting; the design is relatively simple and intuitive. I would reconsider changing my score if the majority of the weaknesses/questions could be adequately addressed. Most importantly, with respect to data leakage and evaluations.