- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces What If TSF (WIT), a benchmark for assessing whether time-series forecasts respond appropriately to future scenarios described in text. Holding the historical series fixed, WIT evaluates factual forecast error, relations across scenarios, and agreement with similar historical cases. The experiments show that gains from factual scenario information do not ensure appropriate responses to alternative scenarios.
Critical strengths
-
Forecasting often requires comparing possible future scenarios. Factual forecast error alone does not assess this ability.
-
The paper explains why accuracy gains from factual text do not show whether a model responds appropriately to other scenarios. This motivates its cross-scenario evaluation.
-
The three evaluation perspectives reveal non-obvious findings. Some forecasts improve under factual scenarios while violating cross-scenario relations. Models with higher relational scores do not necessarily align more closely with similar historical cases.
Critical weaknesses
-
TESS similarly finds that accuracy gains from text do not necessarily indicate effective use of forecast-relevant semantics. CAMEF constructs counterfactual samples by perturbing event text for multimodal time-series forecasting. The paper should discuss both studies, especially how its findings differ from those of TESS.
-
The benchmark contains 114 forecast instances from three domains. It is unclear whether the conclusions generalize to broader forecasting settings.
-
The direction, contrast, and intensity metrics rely mainly on average forecast revisions over the prediction horizon. A model may therefore score well even if it predicts the onset or duration of an effect incorrectly.
Questions for authors
-
How do WIT and its findings differ from TESS and CAMEF, particularly from the findings of TESS?
-
Do the main findings hold across more domains and a larger set of forecast instances?
-
Can the evaluation also assess whether a model predicts the onset and duration of scenario effects correctly?
2 comments
Follow-up on our response and revised manuscript
We are following up on our earlier response to your review. We have now incorporated the following clarifications and additions into the revised manuscript:
-
[W1, Q1] We added TESS and CAMEF to Related Work and clarified how WIT's evaluation of cross-scenario forecast revisions differs from TESS's text-use diagnostics and CAMEF’s contrastive training (Section 2).
-
[W2, Q2] We clarified that the 114 forecasting anchors support 55–68 relational tests each, yielding 6,954 tests in total, and detailed the common construction protocol supporting extension to additional targets, domains, and observation periods (Tables 6–7; Section 3 and Appendix B).
-
[W3, Q3] We clarified Grounded Comparison's complementary assessment through timestep-wise trajectory comparisons. We also discussed extending relational checks to predefined time windows where scenario text and domain knowledge justify temporal constraints, as future work (Section 4.4; Appendices D.4.4 and G).
With approximately three hours remaining in the discussion period, we would be very grateful for even a brief follow-up on whether these revisions address your concerns.
We appreciate your valuable comments and suggestions.
[W1, Q1] Thank you for pointing out the connections between WIT and these two important studies, TESS [1] and CAMEF [2].
We share TESS's observation that forecast accuracy gains alone leave open how effectively a model uses predictive information in text. TESS's diagnostic experiments use text describing statistical features extracted from observed future windows, making it possible to identify which tokens carry predictive information. The analysis shows that models often pay more attention to redundant tokens than to predictive ones, even when the text reduces forecast error.
WIT examines how numerical forecasts respond to different plausible future scenarios, holding the history and domain background fixed and measuring each revision relative to the same model's forecast without a scenario. Within the same anchor, we find that the factual scenario can reduce forecast error while alternative scenarios still elicit revisions that violate the expected direction or intensity ordering.
CAMEF also constructs alternative event descriptions by changing numerical values or sentiment-bearing words in macroeconomic announcements and uses these variants in contrastive training. WIT uses alternative scenarios as evaluation inputs and assesses whether the resulting forecast revisions satisfy predefined relations.
We will discuss both studies and clarify these distinctions in the revised manuscript.
[1] Li, L., Wang, Y., Yan, J., ... & Gong, Y. From Text to Forecasts: Bridging Modality Gap with Temporal Evolution Semantic Space. ICML 2026.
[2] Zhang, Y., Yang, W., ... & Xiong, J. CAMEF: Causal-Augmented Multi-Modality Event-Driven Financial Forecasting by Integrating Time Series Patterns and Salient Macroeconomic Announcements. KDD 2025.
[W2, Q2] The three domains provide distinct real-world forecasting conditions for evaluating scenario responsiveness.
The 114 instances from these domains are forecasting anchors, each representing a distinct forecasting state within its domain. At each anchor, we evaluate multiple controlled scenarios describing plausible future conditions applicable across anchors within the same domain. Using the same model’s neutral forecast as the common reference, we examine whether revisions follow the expected direction, whether opposing scenarios produce contrasting revisions, whether stronger scenarios induce larger revisions, and whether irrelevant text leaves the forecast largely unchanged. This yields 55-68 relational tests per anchor and 6,954 in total across Direction, Contrast, Intensity, and Restraint (Tables 6-7). This design allows us to examine the main findings across different forecasting conditions both within and across domains, with multiple controlled scenario comparisons at each anchor.
Broader coverage of targets and domains would further strengthen the benchmark. To support such extensions, Section 3 and Appendices B-D cover anchor selection, reusable construction framework of controlled scenario sets with validation, and complementary evaluation metrics and sensitivity analyses. This common protocol allows additional targets and domains to be incorporated while preserving the same evaluation principles.
[W3, Q3] We agree that high Direction, Contrast, and Intensity scores do not guarantee correct onset or duration, since these metrics use horizon-averaged revisions.
CiK and CAF-7M include scenarios with explicit temporal information about future effects, allowing models to be evaluated under more precisely specified conditions. In many practical forecasting settings, however, a scenario may describe future events and their timing without specifying precisely when, how strongly, or for how long the target series will respond. WIT's current relational metrics focus on relations justified by the scenario and domain background, without assuming an exact response trajectory for an unobserved alternative future.
For supported anchor-scenario pairs, Grounded Comparison compares full revision trajectories with neutral-relative deviations in matched real cases at each forecast step (Section 4.4, Eq. 8). It can therefore reflect temporal mismatches that horizon averages miss, although matched cases provide empirical references rather than exact onset or duration labels.
Where scenario text and domain knowledge justify temporal constraints, relational checks can be extended to predefined time windows. We will clarify the scope of the current metrics and discuss this extension as future work in the revised manuscript.