- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents a diagnostic study of how time series foundation models (TSFMs) use cross-channel information, evaluating frozen Chronos-2, TimesFM-3, and TiRex-2 under controlled synthetic processes with known data-generating mechanisms and on real-world forecasting tasks from fev-bench. The analysis is organized around four research questions addressing temporal relationships (event-response patterns with varying delays), dependence structures (a shared latent factor observed through noisy, duplicate, and distractor channels), conditioning interfaces (multivariate, past-covariate, and known-future), and covariate reliability (degraded and predicted covariates). The core contribution is an alignment-based diagnostic that distinguishes underuse of informative inputs from sensitivity to inputs that add no predictive information. The central finding is that cross-channel information can be available without being effectively used: the models underuse delayed or complementary signals while remaining sensitive to redundant or irrelevant inputs, and statistical baselines fitted to the same observed contexts recover the relationships the models miss, indicating that the shortfalls reflect limited use of information rather than its absence. On real-world tasks, conditioning on each model's own predicted covariate medians increases quantile loss and reduces interval coverage, and propagating covariate uncertainty recovers much but not all of this loss.
Critical strengths
-
A clean, well-controlled diagnostic methodology. By using synthetic data with known generating processes, together with oracle references and statistical baselines, the paper can precisely separate "no information available" from "information available but unused" — a distinction aggregate benchmark accuracy cannot make.
-
Findings that are concrete, actionable, and real-world validated. The result that conditioning on a model's own predicted covariate medians degrades probabilistic forecasts is directly relevant to common forecasting practice, and it is confirmed on three real-world task pairs.
Critical weaknesses
- A possible home-field advantage in one key control. The fitted state-space model used to show the information is "recoverable" is close to the factor-model class that generated the synthetic data, so its strong performance may partly reflect a lucky model-class match rather than a general deficiency of the TSFMs.
Questions for authors
-
Since the three models are very recent and some are not peer-reviewed, could you add a short background section describing their architectures and training, so that readers less familiar with these models can interpret the results?
-
The findings are described as model-dependent and cover only three models. Do you have evidence that the same qualitative patterns, such as underuse of delayed signals and sensitivity to irrelevant channels, extend to other TSFMs?
4 comments
Response to Reviewer NdnY (2/2)
[Q1 : Architecture and pretraining background]
We added a short background section to the revised paper, uploaded with this response (Appendix A.1). It describes the architecture, cross-channel mechanism, and pretraining of each released checkpoint, for the original three models (Chronos-2, TimesFM-3, and TiRex-2) and the two added models. In brief, for the added models: CITRAS-FM [1] is a small decoder-only patch Transformer with cross-time and shifted cross-variate attention, and t0 [2] is a decoder-style patch Transformer alternating temporal and group attention. All five checkpoints natively accept known-future covariates.
[Q2 : Evidence from additional TSFMs]
We agree that additional models are important for assessing the scope of the findings. To address this, we added CITRAS-FM [1] and t0 [2], two recent TSFMs with native support for known-future covariates. We evaluated their frozen checkpoints using the same temporal, structural, and irrelevant-future protocols.
As summarized in Tables 4–5, both models have small response components along the true response template, already at zero lag. For conditionally irrelevant future covariates, t0 reacts more than SparseDirect and CITRAS-FM reacts less. These results establish under-response rather than a common pattern of lag-induced collapse. Their structural behavior remains different: t0 has a mean distractor penalty, while CITRAS-FM’s distractor change is small relative to its SEM and its MAE decreases with duplicated inputs (Tables 2–3). Your question led us to test two more models, and the answer differs by pattern. None of the five tested models uses the delayed relation well: Chronos-2 and TimesFM-3 lose it as the lag grows, while TiRex-2, t0, and CITRAS-FM respond weakly even at zero lag. Sensitivity to irrelevant inputs extends to t0 but not to CITRAS-FM.
Table 4. Temporal response gain: original models, additional TSFMs, and the context-fitted baseline (corresponding to Figure 2 of the paper).
| Model / reference | Gain: lag 0 | Gain: lag 16 | Gain: lag 64 |
|---|---|---|---|
| EventResponseFit | 0.686 ± 0.003 | 0.713 ± 0.004 | 0.670 ± 0.004 |
| Chronos-2 | 0.468 ± 0.003 | 0.342 ± 0.002 | 0.002 ± 0.000 |
| TimesFM-3 | 0.533 ± 0.005 | 0.035 ± 0.001 | 0.005 ± 0.000 |
| TiRex-2 | 0.018 ± 0.001 | -0.005 ± 0.001 | -0.001 ± 0.000 |
| t0 | 0.047 ± 0.001 | 0.001 ± 0.000 | 0.000 ± 0.000 |
| CITRAS-FM | 0.002 ± 0.000 | 0.001 ± 0.000 | 0.000 ± 0.000 |
The setup is the same as in Sec. 2 of the paper. Entries are mean ± SEM across generation seeds. Gain is the signed projection of the mean forecast response onto the true injected response, normalized so that the exact response has gain 1. Zero gain means zero projection onto that template.
Table 5. Sensitivity to conditionally irrelevant future covariates: original models, additional TSFMs, and statistical baselines (corresponding to Table 1 of the paper).
| Model / reference | S_future^50: lag 8 | S_future^50: lag 16 |
|---|---|---|
| SelectedLagOLS | 0.000 ± 0.000 | 0.000 ± 0.000 |
| SparseDirect | 0.030 ± 0.001 | 0.021 ± 0.001 |
| Chronos-2 | 0.162 ± 0.001 | 0.154 ± 0.000 |
| TimesFM-3 | 0.114 ± 0.001 | 0.053 ± 0.001 |
| TiRex-2 | 0.128 ± 0.001 | 0.061 ± 0.001 |
| t0 | 0.074 ± 0.001 | 0.077 ± 0.001 |
| CITRAS-FM | 0.020 ± 0.001 | 0.007 ± 0.000 |
The setup and the sensitivity measure are the same as in Table 1 of the paper (noisy lagged copy, β = 0.8, context 512, horizon 8). Entries are mean ± SEM across five generation seeds.
Thank you again for the careful review. Your comments helped us clarify the role of the statistical references and the scope of our findings.
[1] CITRAS-FM: Tiny Time Series Foundation Model for Covariate-Informed Zero-Shot Forecasting, arXiv 2026
[2] t0: A Time-Series Foundation Model for Forecasting with Context, arXiv 2026
Follow-up on our rebuttal
Thank you again for your suggestions. The uploaded revision includes the requested model background in Appendix A.1. Our rebuttal also includes a statistical baseline for the structural and lagged-copy experiments and evaluations of two additional models. These additions clarify the structural comparisons and extend the evidence for underuse of delayed relations across all five tested models. We would be grateful if you could review our responses when you have a chance.
I thank the authors for their detailed response. The rebuttal addressed some of my concerns; however, after considering the other reviewers' feedback as well, I believe the current score is consistent with my overall assessment of the paper, and I therefore maintain my score.
Response to Reviewer NdnY (1/2)
We sincerely thank the reviewer for the thoughtful feedback. Below we address the concerns on the statistical reference and the evidence beyond the original three models.
[W1 : Model-class assumptions and statistical references]
We agree that inferring cross-channel structure from a finite, noisy context is challenging, and that the fitted SSM benefits from assumptions close to our shared-latent generator. Its performance should therefore be interpreted as an achievable reference under a specified model class. We will revise the wording accordingly.
To address this concern, we added SparseDirect, a direct sparse linear regression that selects its input channels, lag length, and regularization using rolling validation within the observed context. It selects between target-only and all-channel inputs, retains raw duplicates, and receives no latent state, generating parameters, informative-channel labels, or true lag. It therefore provides an additional statistical reference without the SSM’s shared-state and common-persistence assumptions, while retaining its own linear sparsity assumptions.
As summarized in Tables 1–3, the comparison changes our interpretation. At K = 16, every tested TSFM has lower absolute MAE and smaller distractor deterioration than SparseDirect, so the distractor reaction is not excessive by this standard. TimesFM-3 (+0.01 ± 0.16%) shows that it can also be avoided zero-shot. Two observations remain.
The same reference leaves the main result unchanged. In the lagged-copy task, SparseDirect is not told the lag and still recovers 97% of the attainable improvement, while four of the five TSFMs recover at most 2% (details in our response to Reviewer Es7u, Q10).
We will report these narrower observations together with both absolute accuracy and sensitivity.
The structural comparisons use ρ=0.95, λ=0.75, context 512, horizon 8, and five generation seeds with 256 paired windows per seed. Entries are relative MAE changes (%) reported as mean ± SEM across seed-level ratios, with each method’s own Noisy K1 MAE as the denominator.
Table 1. Independent observations: relative MAE reduction (%). Positive values indicate lower MAE.
Table 2. Exact duplicates: relative MAE increase (%). Positive values indicate higher MAE.
Table 3. Distractors: relative MAE increase (%). Positive values indicate higher MAE.