- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces a two stage framework for time series forecasting named HopFormer where the external covariates are available. They separate the covariates prediction from the residual in the forecasting. The stage 1 fits regression experts and recombines them later through sparse pattern aggregation and softmax weights, stage 2 uses a fixed pretrained transformer in stage 1 with the Low-Rank Adaptation. They test their method on 6 selected datasets with three designed synthetic and three real world datasets. They also provided theoretical analysis, ablations, sensitivity analysis and compared with other time series foundation models.
Critical strengths
1 This paper provides a detailed implementation and evaluation protocol. Besides, they report both favorable and unfavorable empirical outcomes.
2 This paper provided the ablation controls with the similar Chronos backbone and verified the different ways for the LoRA adaptation.
Critical weaknesses
1 The model seems applicable to the datasets with the covariates, which limits the application scope of this method. The motivation for decomposing the forecasting problem into two stages should be further clarified. If the goal is not to build a universal forecasting model, it is also unclear why the backbone needs to be frozen instead of finetune the whole parameters.
2 The main components are familiar like LoRA, sparse aggregation, even the decomposition into predictable and residual components. The paper needs stronger justification against the contributions that currently appear at component-level novelty.
3 The empirical evidence lacks comparisons on the standard benchmark in time series forecasting like traffic or widely-used electricity, raising concerns about whether the evaluation is sufficiently representative. The strongest empirical result is concentrated on a particularly favorable synthetic setting.
4 The significant tests are not performed over the random seeds, and only three random seeds are not enough to establish the uncertainty across model training. Considering some performance differences are marginal, for example, the MASE on EPF improves only from 0.528 to 0.524 compared with direct LoRA adaptation, the significant test on training are more favorable.
5 The theoretical analysis does not fully match with the actual model setting. For example, the theoretical analysis uses the squared-loss while the implemented methods use the quartile loss.
6 The ablation study doesn’t test the other pretrained backbone, making it unclear whether the proposed setup is specific for the selected Chronos-Bolt backbone and LoRA configuration?
Questions for authors
1 What is the application scope of this work when informative covariates are unavailable? Besides, the rationale to decompose based on the covariates and residuals is not fully convincing, since the model can predict without covariates and sometimes can do better without the covariates.
2 Why the experiments do not include widely used benchmarks like ETT datasets or weather datasets or commonly used Electricity, traffic datasets, they also have the covariates.
3 Why are significance tests performed over rolling forecast origins rather than over independent training runs? Given that only three random seeds are used and some matched-backbone differences are small, could the authors provide additional runs or statistical analysis over training randomness?
4 How dependent is the proposed framework on the choice of a pretrained backbone?