- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates directional bias in time series foundation models. Through synthetic and financial experiments, it shows that upward bias can resemble forecasting skill in rising markets, examines possible historical memorization, and evaluates mirror correction and augmentation.
Critical strengths
The paper identifies an interesting and relevant phenomenon.
The experimental design is clever, with carefully chosen controls that help distinguish several confounding factors.
Critical weaknesses
The theoretical arguments are largely standard and lack clear novelty.
The findings are not sufficiently translated into concrete guidance for TSFM pretraining and downstream applications.
Multivariate forecasting and the use of covariates receive insufficient discussion.
Questions for authors
Please develop concrete implications for TSFM pretraining or downstream applications, supported by additional analysis or experiments.
Please investigate the role of covariates and discuss whether the findings extend to multivariate forecasting.
Please position the theoretical results as supporting analysis and moderate the emphasis on theoretical novelty.
I would be willing to increase my score if the authors satisfactorily address these three points.
5 comments
Response to Reviewer EsHt (2 of 2): Q2 and W3, covariates and multivariate forecasting
To Q2 and W3, Thank you for asking about covariates and multivariate forecasting. The submission used neither, and the revision adds a study of both (Section 4, Appendix M). A covariate can bring a direction of its own, and more channels do not remove the bias in general.
(a) Four of the eleven models accept further inputs in the interfaces we use. Chronos-2 and Moirai-1.1 take joint targets, past covariates and known future covariates. Moirai-2.0 takes past and known future covariates. TimesFM-2.5 takes known future covariates through a linear regression.
(b) In two of the four models, an unrelated covariate changes the direction of the forecast. We paired each random walk with a covariate that is independent of it, so the last value remains the best forecast, and let the covariate drift up or down by 0.1 standard deviations per step. Replacing a falling covariate by a rising one raises the share of rising forecasts at 128 steps by 0.40 for Chronos-2 and 0.54 for Moirai-2.0 (Table E1). Moirai-1.1 barely changes. TimesFM-2.5, given an independent or a rising known future covariate, stays at 0.50 to 0.52.
(c) Two positive controls check whether a model uses the channel at all. With a known future covariate that drives the target, Chronos-2 reaches the attainable skill of 0.64 and TimesFM-2.5 about half of it. Moirai-2.0 stays behind the last value there (−0.11), although an unrelated covariate moved its forecasts in (b). No model uses a past covariate that leads the target by one step, and with it Moirai-1.1 is far behind the last value (skill −1.90).
(d) With several inputs, what the mirror correction removes depends on which inputs are mirrored. If the target and the covariate are negated together, the shift caused by the covariate changes sign with them and stays in the odd part, which the correction keeps. If the target alone is negated, the shift is the same in both forecasts and falls into the even part, which the correction removes (Table 37). The mirror therefore has to be applied to the inputs whose direction is in question.
(e) For multivariate forecasting, Chronos-2's share of rising forecasts falls from 0.80 with one channel to about one half with five or more independent channels, and it falls less when the channels are correlated. Moirai-1.1's stays between 0.62 and 0.74 (Table E2). This analysis is post hoc and covers the two models that forecast several targets jointly.
(f) On the financial task we gave the models the market as a past covariate or forecast up to 50 series jointly. With none of these inputs does a direction strategy beat holding the market by two standard errors, and none gives a RankIC above zero after a Holm correction (Table 39). With 50 joint series Moirai-1.1's RankIC falls by 0.056 ± 0.022, and Chronos-2's share of rising forecasts falls from 0.79 to 0.43.
(g) With the real covariates of ETTh1 and Weather, the effect depends on the model and the data (Table 40).
Table E1 (new runs, Table 36 of the revision). Random-walk targets, 128 series. Columns 2 and 3 give the share of rising forecasts at 128 steps without a covariate and with an independent past covariate. Column 4 gives the change in that share when a falling past covariate is replaced by a rising one. Columns 5 and 6 are the positive controls, the skill at 16 steps with a known future covariate that drives the target and the skill at 1 step with a covariate that leads it. Both have an attainable skill of 0.64.
| Model | No covariate | Independent covariate | Rising minus falling | Known future | Leading |
|---|---|---|---|---|---|
| Chronos-2 | 0.70 | 0.66 | +0.40 | +0.65 ± 0.04 | −0.06 ± 0.05 |
| Moirai-1.1 | 0.69 | 0.70 | −0.02 | +0.10 ± 0.05 | −1.90 ± 0.67 |
| Moirai-2.0 | 0.59 | 0.52 | +0.54 | −0.11 ± 0.06 | −0.02 ± 0.07 |
| TimesFM-2.5 | 0.52 | - | - | +0.34 ± 0.04 | - |
Table E2 (new runs, Table 38 of the revision, post hoc). Share of rising forecasts at 128 steps when several random walks are forecast jointly, by number of channels. Each experiment draws its own 128 random walks, so the one-channel shares differ from Table E1 by sampling.
| Model | Channels | 1 | 2 | 5 | 10 | 20 | 50 |
|---|---|---|---|---|---|---|---|
| Chronos-2 | independent | 0.80 | 0.58 | 0.49 | 0.50 | 0.51 | 0.51 |
| Chronos-2 | correlated 0.7 | 0.71 | 0.71 | 0.60 | 0.60 | 0.59 | 0.55 |
| Moirai-1.1 | independent | 0.65 | 0.68 | 0.65 | 0.64 | 0.66 | 0.62 |
| Moirai-1.1 | correlated 0.7 | 0.71 | 0.67 | 0.72 | 0.69 | 0.74 | 0.73 |
We welcome further discussion if any concerns remain.
Dear Reviewer EsHt,
We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.
Response to Reviewer EsHt (1 of 2): Q3, W1, Q1 and W2, theory and practical guidance
Thank you for the review and for stating clearly what would change your assessment.
To Q3 and W1, Thank you for asking us to position the theory as supporting analysis. We agree that the arguments are standard, and the revision presents them that way.
To Q1 and W2, Thank you for asking for concrete guidance supported by additional analysis or experiments. The paper closes with guidance for downstream use and for pretraining (Section 7, Appendix I). For downstream use the revision adds two checks, a backtest on years after the training data and a test with an unrelated covariate. New experiments support all four checks. The guidance for pretraining rests on the corpus and fine-tuning experiments from the submission (Figure 4, Tables 15 and 30).
For downstream use, Section 7 lists checks that need only forecasts.
The checks complement each other. In squared error alone, the first check would not flag Chronos-T5, Chronos-Bolt and TiRex, whose point estimates are above both the last value and the fitted drift on raw daily windows at 128 steps (Table 3). Its direction part does flag them, and so does the second check, on which all three depart upward.
For pretraining, Section 7 makes these recommendations.