- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Halo, a way of turning some pre-existing deep forecaster into a heterothedastic one; it does this by adding a scale output, either in its projection head or as another copy of the same network trained with the corresponding NLL. The main claim it does is that the outputted scale improves the point forecast itself, not just uncertainty. To show this, Halo is applied to TimeXer, GCGNet and CrossLinear on the five EPF electricity price markets, showing improvements in MSE/MAE in 28 of 30 comparisons, with average MSE reductions between 2.6% and 16.5%.
Critical strengths
- The method is neat and simple, seems decently cheap and agnostic to the architecture. It can be added to any existing forecaster.
- The results seem positive for 3 different architectures (transformer, GNN+VAE, CNN) and under both Gaussian and Laplacian losses.
- The protocol is good: future exogenous inputs are excluded to prevent data leakage, the tuning is done only on validation, and the code with the hyperparameters has been released, which is always a plus.
Critical weaknesses
This paper should be rejected for the following reasons: (1) all results come from single runs, and (2) the gains are small enough to be seed noise; (3) the comparison to the baseline is not methodologically correct.
Reasons for rejection:
-
All result come from a single run with one seed, no variance is reported. For TimeXer and CrossLinear the average gains are around 2-4%, which are low enough to suspect that Halo is not actually the reason of improvement, since in these datasets, different runs for the same model can produce these results.
-
The tuning with respect to the baseline is unequal. Halo models get their own hyperparameter search whereas the baselines reuse the original authors' settings, even though everything is retrained under a new setup than what they were tuned for. GCGNet is the worst case. It was originally tuned in a case with access to future exogenous inputs, which this paper removes, yet only the Halo version is retuned. This makes the biggest paper empirical improvement of 16.5% MSE unsurprising. The parallel variant also doubles the parameter count, with the baseline not matching it.
Questions for authors
-
The appendix is quite limited, although it gives the training setup it leaves out reporting several basic things like: lookback, horizon, the hyperparameter search space and budget, the final per-model and per-market configurations, and data split sizes. I appreciate that the code and hyperparameters are in the anonymous repository, but it would be helpful to have these details in the appendix as well. Could you also say if the baselines got a comparable tuning budget in your setting?
-
All results are single runs. Could you report means and standard deviations over more seeds?