- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes TTC4TS, an algorithm that gives better forecasts from some pre-trained time series foundational model without retraining. Instead of just asking the model for one single forecast, it asks for up to 80 slightly different ones, produced by giving the model slightly altered versions of the same data each time (adding noise, stretching it, flipping it, etc.) Then, a second component combines all of them together into a final forecast (either by an average, by a learned weighted average learned from past data or by a small network trained to pick the best combination of the forecasts for each case).
The main thesis is that, as it happens with LLMs, spending more compute at prediction time can lead to better forecasts. The method is tested with 6 pretrained models on 6 datasets. The authors report a reduction of the forecast error (using the fixed weighted average algorithm) of 6.4% on average, with improvements on the original model in 420 of the 432 tested settings.
Critical strengths
- Improving an existing model without retraining it is a useful practical goal, and the method works with any model, which is a plus.
- The authors report negative results openly, including failed components, the cost in time, and the fact that a small subset of candidates matches the full set. This made the paper easy to evaluate.
- The experiments are careful not to use information from the future, a common problem in forecasting papers. I did not find this happening here.
Critical weaknesses
The paper should be rejected because: (1) It does not establish that the proposed method is really worth its cost, which is about 60 times more computation per final forecast; (2) the paper overstates the evidence; and (3) the method reduces to known techniques, with its own results showing that the "test-time compute" idea is not what produces the stated gains in the end.
Reasons for rejection
-
The practical value of the framing is not well stablished. The complete method runs the model 61 times per forecast, this makes the smallest model go from 43 ms to about 2600 ms. It is also never compared against other possible ways of spending that compute, such as a bigger model on its own like Chronos-Bolt-base, which takes 74 ms or an ensemble of different models.
-
The evidence makes the size of the obtained gains and the ranking between model variants hard to trust. For example, on the hourly EET data just 4 non-overlapping test forecasts at the longest horizon and about 30 at the shortest are used. The "420 of 432" claim rests on the reuse the same test data across horizons, seeds and models, and no confidence intervals are reported.
-
Furthermore, the contribution reduces to some good known techniques: The experiments show that, in fact, the proposed test time compute does not really help (for example, the proposed test-time classifier is worse than using the original model alone, giving 23% higher error) and what works best in the end is just a classical fixed weighted average (Bates & Granger, 1969) obtained by an ensemble selection algorithm (Caruana et al. (ICML 2004), not cited). This is probably showing a fundamental shortcoming of the proposed idea: test-time compute works well for LLMs because many of the proposed answers can be checked or compared before it is returned (think on a solution to a math or programing problem), but in forecasting, a prediction can only be checked after the future arrives, which is never available at test-time.
Questions for authors
- If 13 forecasts match 80, what is the case for the full method, and how does it differ from standard forecast combination (Bates & Granger, 1969) and ensemble selection (Caruana et al., 2004)?
- Could you report the actual error values and compare against a larger model, an average of several models, and the models' own probabilistic forecasts, at the same computational cost? This would address the second weakness.
- Could you report confidence intervals over the test forecasts and evaluate on a longer test period?