acceptodds
Under review as a conference paper at ICLR 2027

What If TSF: Reframing Time Series Forecasting as Scenario-Guided Multimodal Forecasting

Abstract

Recent advances in large language models (LLMs) have enabled time series forecasting to move beyond numerical observations and incorporate external information in multimodal settings. Such information can improve forecasting performance, but accuracy alone may not reveal whether models appropriately respond to it: a model may ignore relevant information, fail to distinguish scenarios with different implications, or overreact to irrelevant or weak signals. We introduce What If TSF (WIT), a benchmark for evaluating whether models effectively respond to future scenarios. WIT constructs controlled scenario sets by fixing the forecasting state while systematically varying future scenarios, enabling three complementary evaluations: Factual Comparison, which measures the predictive utility of factual scenario information; Relational Comparison, which evaluates whether forecasts satisfy expected direction, contrast, intensity, and restraint relations across scenarios; and Grounded Comparison, which assesses whether scenario-induced revision trajectories are empirically plausible relative to comparable real cases. Experiments show that factual accuracy gains do not consistently translate into appropriate responses to alternative scenarios, demonstrating the need to evaluate multimodal forecasting beyond predictive accuracy.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Multimodal Time Series Forecasting, Future Scenarios, Neutral-relative Revisions, Relational and Grounded Comparisons

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Z2agOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces What If TSF (WIT), a benchmark for assessing whether time-series forecasts respond appropriately to future scenarios described in text. Holding the historical series fixed, WIT evaluates factual forecast error, relations across scenarios, and agreement with similar historical cases. The experiments show that gains from factual scenario information do not ensure appropriate responses to alternative scenarios.

Critical strengths
  1. Forecasting often requires comparing possible future scenarios. Factual forecast error alone does not assess this ability.

  2. The paper explains why accuracy gains from factual text do not show whether a model responds appropriately to other scenarios. This motivates its cross-scenario evaluation.

  3. The three evaluation perspectives reveal non-obvious findings. Some forecasts improve under factual scenarios while violating cross-scenario relations. Models with higher relational scores do not necessarily align more closely with similar historical cases.

Critical weaknesses
  1. TESS similarly finds that accuracy gains from text do not necessarily indicate effective use of forecast-relevant semantics. CAMEF constructs counterfactual samples by perturbing event text for multimodal time-series forecasting. The paper should discuss both studies, especially how its findings differ from those of TESS.

  2. The benchmark contains 114 forecast instances from three domains. It is unclear whether the conclusions generalize to broader forecasting settings.

  3. The direction, contrast, and intensity metrics rely mainly on average forecast revisions over the prediction horizon. A model may therefore score well even if it predicts the onset or duration of an effect incorrectly.

Questions for authors
  1. How do WIT and its findings differ from TESS and CAMEF, particularly from the findings of TESS?

  2. Do the main findings hold across more domains and a larger set of forecast instances?

  3. Can the evaluation also assess whether a model predicts the onset and duration of scenario effects correctly?

2 comments

We appreciate your valuable comments and suggestions.


[W1, Q1] Thank you for pointing out the connections between WIT and these two important studies, TESS [1] and CAMEF [2].

We share TESS's observation that forecast accuracy gains alone leave open how effectively a model uses predictive information in text. TESS's diagnostic experiments use text describing statistical features extracted from observed future windows, making it possible to identify which tokens carry predictive information. The analysis shows that models often pay more attention to redundant tokens than to predictive ones, even when the text reduces forecast error.

WIT examines how numerical forecasts respond to different plausible future scenarios, holding the history and domain background fixed and measuring each revision relative to the same model's forecast without a scenario. Within the same anchor, we find that the factual scenario can reduce forecast error while alternative scenarios still elicit revisions that violate the expected direction or intensity ordering.

CAMEF also constructs alternative event descriptions by changing numerical values or sentiment-bearing words in macroeconomic announcements and uses these variants in contrastive training. WIT uses alternative scenarios as evaluation inputs and assesses whether the resulting forecast revisions satisfy predefined relations.

We will discuss both studies and clarify these distinctions in the revised manuscript.

[1] Li, L., Wang, Y., Yan, J., ... & Gong, Y. From Text to Forecasts: Bridging Modality Gap with Temporal Evolution Semantic Space. ICML 2026.

[2] Zhang, Y., Yang, W., ... & Xiong, J. CAMEF: Causal-Augmented Multi-Modality Event-Driven Financial Forecasting by Integrating Time Series Patterns and Salient Macroeconomic Announcements. KDD 2025.


[W2, Q2] The three domains provide distinct real-world forecasting conditions for evaluating scenario responsiveness.

  • Energy (U.S. natural gas price): Supply and demand changes driven by weather, production, pipeline conditions, and LNG exports
  • Economy (U.S. dollar index): Changes in U.S. interest rates and economic conditions relative to those of other economies
  • Politics (party polling support): Changes in government performance and political competition across European countries

The 114 instances from these domains are forecasting anchors, each representing a distinct forecasting state within its domain. At each anchor, we evaluate multiple controlled scenarios describing plausible future conditions applicable across anchors within the same domain. Using the same model’s neutral forecast as the common reference, we examine whether revisions follow the expected direction, whether opposing scenarios produce contrasting revisions, whether stronger scenarios induce larger revisions, and whether irrelevant text leaves the forecast largely unchanged. This yields 55-68 relational tests per anchor and 6,954 in total across Direction, Contrast, Intensity, and Restraint (Tables 6-7). This design allows us to examine the main findings across different forecasting conditions both within and across domains, with multiple controlled scenario comparisons at each anchor.

Broader coverage of targets and domains would further strengthen the benchmark. To support such extensions, Section 3 and Appendices B-D cover anchor selection, reusable construction framework of controlled scenario sets with validation, and complementary evaluation metrics and sensitivity analyses. This common protocol allows additional targets and domains to be incorporated while preserving the same evaluation principles.


[W3, Q3] We agree that high Direction, Contrast, and Intensity scores do not guarantee correct onset or duration, since these metrics use horizon-averaged revisions.

CiK and CAF-7M include scenarios with explicit temporal information about future effects, allowing models to be evaluated under more precisely specified conditions. In many practical forecasting settings, however, a scenario may describe future events and their timing without specifying precisely when, how strongly, or for how long the target series will respond. WIT's current relational metrics focus on relations justified by the scenario and domain background, without assuming an exact response trajectory for an unobserved alternative future.

For supported anchor-scenario pairs, Grounded Comparison compares full revision trajectories with neutral-relative deviations in matched real cases at each forecast step (Section 4.4, Eq. 8). It can therefore reflect temporal mismatches that horizon averages miss, although matched cases provide empirical references rather than exact onset or duration labels.

Where scenario text and domain knowledge justify temporal constraints, relational checks can be extended to predefined time windows. We will clarify the scope of the current metrics and discuss this extension as future work in the revised manuscript.

AuthorsOpenReview ↗

Follow-up on our response and revised manuscript

We are following up on our earlier response to your review. We have now incorporated the following clarifications and additions into the revised manuscript:

  • [W1, Q1] We added TESS and CAMEF to Related Work and clarified how WIT's evaluation of cross-scenario forecast revisions differs from TESS's text-use diagnostics and CAMEF’s contrastive training (Section 2).

  • [W2, Q2] We clarified that the 114 forecasting anchors support 55–68 relational tests each, yielding 6,954 tests in total, and detailed the common construction protocol supporting extension to additional targets, domains, and observation periods (Tables 6–7; Section 3 and Appendix B).

  • [W3, Q3] We clarified Grounded Comparison's complementary assessment through timestep-wise trajectory comparisons. We also discussed extending relational checks to predefined time windows where scenario text and domain knowledge justify temporal constraints, as future work (Section 4.4; Appendices D.4.4 and G).

With approximately three hours remaining in the discussion period, we would be very grateful for even a brief follow-up on whether these revisions address your concerns.

Reviewer_vHiFOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper introduce What If TSF (WIT) 1, a benchmark designed to evaluate whether forecasts reflect the implications of contextual information under controlled future scenarios. WIT evaluates forecasts using controlled scenario sets, holding the numerical history and domain background context fixed while systematically varying the future scenario (See Figure 1). By expanding evaluation from individual text-conditioned forecasts to structured comparisons under the same forecasting state, WIT assesses both factual predictive utility and consistency in scenario-induced forecast responses. WIT evaluates these forecasts through three complementary comparisons. Factual Comparisonasks how accurately the model predicts the observed future, how much factual scenario information reduces forecast error on average, and how often that information improves the prediction. Relational Comparison tests whether scenario-induced revisions relative to the model’s neutral forecast satisfy explicit expectations about Direction, Contrast, Intensity, and Restraint. Finally, Grounded Comparison assesses how closely these neutral-relative revision trajectories align with deviations observed in comparable real cases. Together, these comparisons evaluate not only whether context improves observed-future prediction, but whether forecasts respond coherently and in empirically plausible ways as future scenarios change.

Critical strengths

A major strength of this paper is that it attempts to redefine time series forecasting as a new problem using “hypothetical” scenarios, which brings to mind counterfactual reasoning. This actually represents a higher level of forecasting. What does this mean? If you’ve read the book Why, you’ll find that correlation, intervention, and counterfactuals constitute what are known as the second and third levels—that is, a relatively advanced stage of causal inference. Although the paper’s title—which suggests redefining time series forecasting—is somewhat misleading, since we don’t need “what-if” scenarios in every situation, not all time series forecasts should be redefined as counterfactual reasoning; however, certain types of forecasting can indeed be defined this way. Essentially, the authors point to a highly compelling problem, and this insight offers considerable inspiration to the field. The authors have designed a relatively rich dataset and established a diverse range of mechanisms; the entire benchmark, including the evaluation scheme, is also relatively comprehensive,

Critical weaknesses

I feel that the author has chosen three fields in which it is relatively very difficult to construct a dataset, yet the analysis, justification and even experimentation regarding the reliability of the dataset are insufficient. I believe a fundamental flaw in this paper lies in the author’s choice of dataset;The assessment of impacts in various scenarios is too subjective. This is because unexpected or unreasonable events often occur in these three areas. I have serious doubts about the reliability of the dataset the author constructed, which is said to contain various causal relationships with measurable, concrete effects. Although the author attempts to use defensive writing to argue that these are not causal effects, this is, in fact, the underlying logic. In my view, such controlled—or, more accurately, partially controlled—time series are actually quite common; for example, in the fields of robotics or navigation, there are controlled real-world spatiotemporal trajectories. However, because the author fails to adequately analyze and justify the rationale for constructing scenarios in this manner, a problem arises: the dataset he selected comes from three fields, each of which contains numerous confounding factors, making it extremely difficult to quantify the causal effects of specific events or interventions. Economics, energy, and politics are all fields deeply influenced by multiple factors; it can be argued that identifying true causal effects in these fields is indeed very difficult.

Questions for authors

many thanks to the author; this is a very interesting piece of work that raises an intriguing question. I believe the following key issues have affected one of my marks.

1.How does the author demonstrate that the fixed order of labels, based on direction and intensity, is valid for all historical anchor points?

2.Why did the author choose these three domains rather than datasets from other fields?

8 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

Thank you for recognizing the research value of comparing forecasts under multiple hypothetical future conditions for the same forecasting problem, beyond individual forecast accuracy. We also appreciate your positive assessment of the dataset's richness, the diversity of scenario mechanisms, and the comprehensive evaluation framework.

WIT focuses on evaluating how models respond when future scenario information is provided. To better reflect this focus and avoid any ambiguity, we will revise the title to "What If TSF: Evaluating Scenario-Guided Multimodal Time Series Forecasting."

We understand your concern about the reliability of direction and intensity relations in domains where multiple factors interact. WIT draws on domain mechanisms to define expected revisions relative to the same model's neutral forecast when scenario information changes under a fixed numerical history and domain background. The reliability of this evaluation depends on whether the stated conditions justify these expected relations and whether they remain applicable at each selected anchor.

Below, we explain how the relations and their cross-anchor applicability were validated during benchmark construction (Q1), and then discuss our rationale for selecting the three domains (Q2).


[Q1] We agree that applying the same Direction and Intensity relations across different forecasting anchors requires clear justification. While Appendix B.5 states that the relations were reviewed and fixed before model evaluation, the manuscript does not sufficiently describe the independent validation procedure.

For clarity, the LLM-assisted evidence extraction and factual scenario construction described in Section 3.3, together with the source and semantic review in Appendix C.1.4, concern the grounding of factual scenarios. These procedures are distinct from validating controlled scenario relations.

Controlled alternative scenarios were constructed from the fixed domain background and underwent two levels of validation during the original benchmark construction, before benchmark finalization and model evaluation.

  • Relation-level validation: We examined whether the scenario text and domain background justified each expected Direction or Intensity ordering, ensuring that Intensity comparisons varied only one strength attribute while holding other relevant conditions fixed.
  • Cross-anchor applicability validation: We checked whether each retained relation remained justified at every selected anchor, without conflicting conditions or additional unstated assumptions.

Two domain experts per domain independently reviewed the candidate relations and all applications of the retained relations to the selected anchors. The proposed labels were withheld during their initial relation judgments. Disagreements were resolved through discussion, with insufficiently justified relations revised or excluded. The table below summarizes the construction-stage review coverage and decisions.

ValidationCoverageFindings and treatment
Relation-levelAll 141 candidate Direction and Intensity relations, each independently reviewed by two experts in the corresponding domain.6 candidates were excluded because their stated conditions did not sufficiently justify the proposed relation. Of the 135 retained relations, 23 were retained after clarification of their conditions or wording and subsequent validation. The final inventory contains 81 Direction and 54 Intensity relations.
Cross-anchor applicabilityIn each domain, both experts reviewed all retained Direction and Intensity relations across 38 anchors (5,130 applications across three domains).No anchor-specific ambiguities, conflicting conditions, or additional unstated assumptions undermining the retained relations were identified. All applications were retained, and no anchors were excluded or replaced on this basis.

Examples of validation decisions:

  • Retained (Direction): The initial wording does not clarify how equipment constraints affected gas deliveries to Henry Hub. After clarification, the scenario justified an upward neutral-relative revision.
    • Initial: "Output constraints on compression equipment along pipelines reduce ..."
    • Revised (E-PI): "Output ... pipelines feeding Henry Hub reduce the amount of natural gas that actually reaches the hub."
  • Excluded (Intensity): Greater equipment-damage severity alone does not determine a larger reduction in gas delivered to Henry Hub, so the proposed ordering was excluded.
    • Proposed pair: "... completely shut down due to equipment damage" → "... completely shut down due to more severe equipment damage."

In Section 3, we will clearly distinguish factual-scenario grounding from the construction and validation of controlled scenario relations. Appendix B.5 will provide further details on the validation procedures and supporting evidence.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

[Q2] We deliberately selected Energy, Economy, and Politics to evaluate forecasting settings in which future external conditions are described in text, but their precise numerical implications for the target are not specified. Models must interpret these conditions and translate their implications into forecast revisions. The three domains provide distinct mechanisms for examining this ability, involving energy supply and demand, relative macroeconomic conditions, and political responsibility and competition.

We agree that multiple interacting factors make it difficult to attribute observed changes to specific events. This also makes it difficult to determine numerical outcomes for alternative scenarios that did not occur. WIT instead evaluates forecast revisions under explicitly stated scenario conditions, with the expected relations justified and validated as described in Q1.

The controlled or partially controlled trajectory datasets you suggested in robotics and navigation also offer valuable settings where responses to different conditions can be observed more directly. We consider these settings complementary: controlled-trajectory evaluation can provide more direct empirical references, while WIT examines how models respond to textually specified alternative futures whose numerical outcomes are not observed for the same forecasting state. We will consider extending WIT to such controlled settings in future work.

We will revise the Introduction to more directly motivate our domain selection and the forecasting setting WIT aims to evaluate.


We hope these clarifications and the validation evidence help address your concerns about dataset reliability and domain selection.

Thanks for your reply, I raise the rating, but still please enhance this part discussion.

Reviewer_QnBtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces WIT, a benchmark for scenario-guided forecasting. It fixes the numerical history and the domain background and varies only the future scenario. Across 114 anchors in energy, economy and politics, it evaluates models in three ways: the accuracy gain from the factual scenario, consistency with predefined relations among alternative scenarios, and agreement with matched real cases.

Critical strengths

The reframing is sound. Measuring each scenario against the model's own neutral forecast under a fixed history isolates the scenario effect without synthesizing counterfactual futures.

Critical weaknesses
  • Unvalidated reference signals The relation labels that define correct behavior are written with LLM assistance, reviewed by prompts, and reused across anchors regardless of market state. No human audit or agreement rate is reported, so relational accuracy measures consistency with textbook priors rather than with domain truth. Grounded Comparison, the only check against observed outcomes, covers 20 of 3,078 alternative pairs and cannot validate the rest.

  • Scenario effects confounded with sampling noise. Each revision is measured against a single neutral forecast decoded at temperature 1.0, while the tolerance is only 0.1 times the median nonzero step. Two neutral samples may already differ by more than this, so Restraint and the sign tests partly reflect decoding variance. No noise floor is reported.

  • Narrow coverage and weak findings. Only five systems are evaluated, and the cutoff excludes all recent LLMs. In Table 26, numerical TSFMs alone outperform Mistral even when it receives the factual scenario.

  • Incremental positioning. CiK already evaluates irrelevant and future context, and TADiff already compares alternative futures under a shared history, leaving the neutral reference as the main addition. Anchors are screened with Chronos-2, which also serves as the forecaster inside MoiraiAgent, and Restraint rests on only three irrelevant scenarios per domain.

Questions for authors

Please refer to weakness

3 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

Thank you for recognizing WIT's neutral-relative evaluation under fixed numerical history and domain background. Below, we clarify the independent validation of scenario relations and the complementary role of Grounded Comparison.


[W1] We appreciate your concern about the reliability of the predefined scenario relations and agree that their validation should be more transparent.

The LLM-assisted evidence extraction and source review described in Section 3.3 and Appendix C.1.4 concern factual-scenario grounding. Controlled alternative scenarios were constructed separately from the fixed domain background by translating supported domain mechanisms into explicit, target-relevant hypothetical conditions.

Example (Economy — Capital Flows): The domain mechanism links international portfolio reallocations to currency demand.

  • Direction: M-CF-P1 explicitly specifies selling foreign currencies and buying U.S. dollars, justifying an upward neutral-relative revision of the U.S. dollar index. M-CF-M1 specifies the reverse transactions, justifying a downward revision.
  • Intensity: M-CF-P1 and M-CF-P3 illustrate small and large net portfolio reallocations under the same mechanism, holding the investor population, source and destination assets, and currency-basket composition fixed.

These candidate scenarios and relations then underwent independent expert validation before benchmark finalization and model evaluation. Two domain experts per domain independently reviewed whether the scenario text and fixed domain background justified each proposed Direction or Intensity relation, without access to the proposed labels. Disagreements were resolved through discussion, with insufficiently justified relations revised or excluded.

Examples of validation decisions (Energy — pipeline):

  • Retained (Direction): The initial wording does not clarify how equipment constraints affected gas deliveries to Henry Hub. After clarification, the scenario justified an upward neutral-relative revision.
    • Initial: "Output constraints on compression equipment along pipelines reduce ..."
    • Revised (E-PI): "Output ... pipelines feeding Henry Hub reduce the amount of natural gas that actually reaches the hub."
  • Excluded (Intensity): Greater equipment-damage severity alone does not determine a larger reduction in gas delivered to Henry Hub, so the proposed ordering was excluded.
    • Proposed pair: "... completely shut down due to equipment damage" → "... completely shut down due to more severe equipment damage."

Cross-anchor applicability was examined separately. For all 5,130 anchor–relation applications, two experts per domain used a checklist to assess whether (i) the scenario mechanism remained applicable to the forecasting target, (ii) the anchor's forecasting state introduced any ambiguities or conflicting conditions, and (iii) the expected relation required additional unstated or anchor-specific assumptions.

ValidationCoverage and findings
Relation-level141 candidate relations independently assessed; initial inter-reviewer agreement: 127/141 (90.1%) before discussion. 6 candidates were excluded, and 23 of the 135 retained relations were clarified and subsequently validated, yielding 81 Direction and 54 Intensity relations.
Cross-anchor applicabilityIn each domain, both experts reviewed all retained relations across 38 anchors (5,130 applications total). No anchor-specific ambiguities, conflicts, or additional unstated assumptions undermining the retained relations were identified.

Grounded comparison coverage. The availability of empirical matched cases is constrained by two factors. First, WIT uses observations from 2024 onward, after the LLM components' knowledge cutoffs, leaving a relatively short observation period. Second, finding cases where the same scenario condition actually occurred under comparable forecasting states is challenging. We therefore retained only pairs with similar numerical histories and forecasts, independently verified scenario occurrences, and at least three matched real cases, yielding 20 supported pairs.

We acknowledge that this limited coverage cannot validate every predefined relation against observed outcomes. Grounded Comparison provides complementary empirical evidence where matched real cases are available. To strengthen empirical coverage in future work, we will consider expanding the matched-case pool and extending WIT to controlled or partially controlled settings, such as robotics and navigation, where responses to alternative conditions can be observed more directly.


We will clarify the distinction between factual-scenario grounding and alternative scenario construction in Section 3.3 and 3.4, explain Grounded Comparison's complementary role and empirical coverage limitations in Section 3.5, and expand Appendix B.5 with details on scenario relation validation.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

[W2] Thank you for this constructive suggestion. We agree that a more direct assessment of stochastic decoding variability would strengthen our existing robustness analysis. As described in Appendix E.4, our original protocol used one neutral forecast per anchor in each of three runs and reported the mean and standard deviation of the resulting metrics. While the practical-tolerance sensitivity analysis examines the choice of evaluation thresholds, it does not directly quantify decoding variability or its impact on evaluation outcomes.

We therefore extended our analysis to examine neutral–neutral variability, sensitivity to neutral replacement, and decision reproducibility across the three runs, while keeping the decoding settings and tolerances unchanged.


Neutral–neutral variability. We compared all three pairs of neutral forecasts for each model and anchor. For Direction, we measured the absolute difference in horizon-averaged forecasts; for Restraint, the mean absolute pointwise difference. The table below reports the largest threshold-normalized 95th percentiles and tolerance-exceedance rates across model–domain groups (Mistral-7B, Energy). Although stochastic variability occasionally exceeded the practical tolerances, most neutral–neutral comparisons remained within these thresholds.

DiagnosticDirectionRestraint
Empirical 95th percentile of neutral–neutral differences1.12η1.34τ
Tolerance-exceedance rate6.1%8.8%

Sensitivity across the three comparisons. We held scenario forecasts fixed and replaced each run’s neutral forecasts with the corresponding outputs from each of the other two runs, yielding six cross-run replacements per model. For Grounded Comparison, we replaced both anchor and matched-case neutral forecasts while preserving the matched sets and shared-case reuse, and recomputed the corresponding no-revision baseline. The table below reports the maximum changes across the tested models and six cross-run replacements: percentage-point differences for FSG, CWR, and GSG; decision-flip rates for Direction and Restraint; and relative percentage changes for GCE.

ComparisonMetricMaximum changeModel / Domain
FactualOverall Factual Scenario Gain (FSG)5.8 ppMistral-7B (Overall)
FactualOverall Context Win Rate (CWR)2.6 ppMistral-7B (Overall)
RelationalDirection decisions3.4% flippedMistral-7B / Energy
RelationalRestraint decisions5.3% flippedMistral-7B / Energy
GroundedDomain-wise Grounded Curve Error (GCE)3.2% relativeMistral-7B / Politics
GroundedOverall Grounded Scenario Gain (GSG)1.8 ppMistral-7B (Overall)

Neutral replacement affected the three comparisons differently. Overall FSG remained positive for all models despite greater sensitivity in its magnitude. Most Direction and Restraint decisions remained unchanged, while GCE and GSG showed limited variation, with the signs of Overall GSG preserved across all models.

Decision reproducibility. We also compared the original run-specific decisions, allowing both neutral and scenario-conditioned forecasts to vary. Beyond the reported variation in category-level scores, this analysis examines whether individual relations receive consistent pass/fail decisions across the three runs. Unanimous agreement is the proportion of relations receiving identical decisions in all three runs. The persistent-violation rate is the proportion violated in all three runs among those violated at least once.

DiagnosticResultModel / Domain (Category)
Minimum three-run unanimous agreement for Direction93.2%Mistral-7B / Energy
Minimum three-run unanimous agreement for Restraint91.2%DoubleCast / Energy
Minimum persistent-violation rate75.6%DoubleCast / Energy (Restraint)

Although neutral forecasts exhibited stochastic variability, the assessed Factual and Grounded Comparison metrics remained stable under the tested neutral replacements, while most Direction and Restraint decisions were consistent across the three original runs. These diagnostics help alleviate concerns about dependence on a single neutral forecast, while identifying some decoding-sensitive decisions. We will provide the detailed analyses and limitations of the three-run assessment in Appendix D.5.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

[W3] We evaluate five scenario-conditioned systems, comprising three directly prompted LLMs and two context-aware forecasting systems. We required the LLM components to have documented knowledge cutoffs before the WIT evaluation period (Appendix E.2). Prior exposure to the evaluated events or outcomes could influence both neutral and scenario-conditioned forecasts, complicating the interpretation of responses to the supplied scenarios. We therefore prioritized reducing contamination risk and preserving the interpretability of scenario-induced revisions over broader coverage of newer LLMs.

We agree that Mistral’s factual forecasts have higher errors than the numerical-only TSFMs, as shown in Tables 2 and 26. However, this absolute comparison addresses a different question from the benefit of scenario information within the same model: Mistral achieves a 7.19% factual gain relative to its own neutral forecast (Table 2). WIT separately evaluates forecasting accuracy and the consistency of responses across alternative scenarios.

Importantly, our findings are not confined to Mistral. The evaluated DoubleCast and MoiraiAgent configurations report lower factual errors than the numerical-only references across all three domains (Tables 2 and 31). Nevertheless, the anchor-level analysis shows that relational violations remain even on anchors where factual conditioning improves their forecasts (Tables 37–39). This demonstrates the diagnostic value of evaluating cross-scenario responses alongside factual accuracy, rather than treating factual gains alone as evidence of reliable scenario use.


[W4] We appreciate your recognition that WIT evaluates scenario-induced forecast revisions relative to each model's neutral forecast under a fixed forecasting state, without relying on synthetic reference trajectories for unrealized alternative scenarios.

Positioning. We agree that CiK already examines future and irrelevant context, and that TADiff evaluates alternative future conditions under a shared history, including through semantic consistency scores. As detailed in Section 2 and Table 1, WIT goes beyond introducing a neutral reference by evaluating Direction and Restraint against neutral, and Contrast and Intensity through comparisons across scenarios, while separately assessing factual predictive gains and, where supported, empirical alignment with matched real cases. These complementary evaluations reveal that factual gains do not guarantee appropriate responses to alternative scenarios, and stronger relational consistency does not necessarily imply closer empirical alignment.

Anchor screening. Chronos-2 is shared with MoiraiAgent’s forecasting component, but screening uses persistent, same-direction deviations between observed futures and forecasts across three frozen history-only references (Chronos-2, TimesFM-3, and TiRex-2), rather than Chronos-2 alone. While reference-guided selection may influence the reported results on these anchors, screening did not use scenario-conditioned forecasts or relational evaluation outcomes. Notably, within this selected evaluation set, our finding that factual gains can coexist with relational violations also holds for GPT-4o and DoubleCast, which do not use Chronos-2 (Tables 37–39).

Restraint coverage. The three irrelevant scenarios per domain are each applied across 38 anchors. This tests whether fixed irrelevant inputs induce unnecessary forecast revisions across different forecasting states, but does not provide broad coverage of irrelevant content. We will make this scope explicit and consider expanding topic and wording diversity in future work.


Thank you again for your thoughtful feedback. We hope our responses have addressed your main concerns and provided clarity on WIT's contributions and evaluation design.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Remarks

We thank the reviewers for their constructive feedback and thoughtful discussion. The revised manuscript now incorporates the validation details, additional analyses, and scope clarifications discussed during the review.

WIT's central contribution is to distinguish factual predictive gains from appropriate responses to changing scenarios. Holding numerical history and domain background fixed, WIT evaluates scenario-induced revisions relative to each model’s own neutral forecast through complementary Factual, Relational, and Grounded comparisons. Our anchor-level analysis shows that relational violations persist even where factual conditioning improves forecast accuracy, including for the stronger context-aware forecasting systems (Section 5; Figure 4). This demonstrates the practical value of examining scenario responses alongside factual accuracy.

The revisions strengthen the basis for this evaluation in two key respects:

  • Relation validity and cross-anchor applicability (Section 3.4; Appendix B.5). We now document the original construction-stage validation by two experts per domain, independent of scenario construction. Initial judgments were made with proposed labels withheld, achieving 90.1% agreement. Following clarification or exclusion where needed, both experts checked all 5,130 applications of the retained relations to the selected anchors. The expanded discussion specifies the historical information and applicability criteria used, with a concrete example showing how the same neutral-relative relation remains justified under contrasting historical states while allowing forecast levels and revision magnitudes to differ. Relations and their applicability were finalized before model evaluation.
  • Decoding variability and evaluation stability (Appendix D.5). We added direct analyses of neutral–neutral variability, sensitivity to neutral replacement, and decision reproducibility using the three original runs. Across model–domain groups, at least 93.2% of Direction decisions and 91.2% of Restraint decisions were identical across all three runs, and most observed violations recurred. These analyses provide evidence for reproducible scenario-response failures while identifying decisions sensitive to decoding variability.

We also strengthened the positioning relative to prior work, clarified Grounded Comparison's complementary empirical role, and explicitly discussed the benchmark's coverage and temporal limitations. (Section 2; Appendices D.4.4 and G).

We appreciate the reviewers' engagement and believe the revised manuscript provides a clearer and more rigorous basis for WIT's contributions and findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.