acceptodds
Under review as a conference paper at ICLR 2027

WhatIf-Bench: Benchmarking Intervention-Aware Time Series Forecasting

Abstract

Recent time series foundation models (TSFMs) can consume covariates in-context, extending zero-shot forecasting to problems driven by external factors. This enables scenario analysis: forecasting how a target responds to events that set its covariates to chosen future values. Whether they can do this reliably is an open question. We introduce WhatIf-Bench, a benchmark measuring how well a forecaster uses in-context covariates to forecast a target's response to future events. WhatIf-Bench is semi-synthetic: realistic time series come from mechanistic simulators of data centers, power grids, road traffic, and water networks, on which we impose controlled covariate-target relationships grounded in each system. Each task presents a history of events that intervene on covariates, then asks to forecast a future event that either replays seen covariate values (a Replay future) or sets them to unseen ones (a Novel future). Varying the number of covariates, historical events, and event length yields 37,665 tasks. Evaluating covariate-aware TSFMs and frontier LLMs, we find the LLMs reach the highest skill scores (relative improvement over a naive baseline), ahead of the best TSFMs. Across the axes, however, every model forecasts well on Replay futures (65–85% skill) but drops to 30–50% on Novel ones. Skill also falls as events set more covariates jointly. More historical events raise skill across the board, especially on Novel futures, but large differences between models remain. WhatIf-Bench offers a testbed toward models that support arbitrary scenario analysis.

Primary area:
learning on time series and dynamical systems
Keywords:
time series foundation models, covariate-aware forecasting, benchmark, scenario analysis, in-context learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_jaJnOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WHATIF-BENCH, a semi-synthetic benchmark for evaluating whether forecasters can use historical and future covariates to predict a target’s response to future events. The benchmark begins with baseline trajectories from four mechanistic simulators covering data centers, power grids, road traffic, and water networks. It then overlays controlled, documented covariate-target response functions on these trajectories. Each task contains a history of events that set one or more covariates and asks the model to forecast the target under a future event

The key experimental comparison pairs two futures with the same observed history. A Replay future repeats previously observed covariate values, whereas a Novel future moves the covariates beyond their historical range. The benchmark additionally varies the number of active covariates, number and duration of historical events, and context/prediction length. The resulting benchmark contains 37,665 selected tasks across 100 target series

The paper evaluates six numerical foundation-model systems and two frontier LLMs. It reports that all evaluated models perform substantially better on Replay than Novel futures, that Novel performance generally deteriorates as more covariates are jointly varied, and that additional historical events tend to improve Novel performance. For example, Claude Opus decreases from 84.8% Replay skill to 50.3% Novel skill, while TS-ICL decreases from 83.6% to 32.5%. The model ordering also changes as the number of covariates increases

Overall, I find the benchmark construction useful and the main empirical phenomenon compelling. My reservations concern the interpretation of the failure modes and the comparability of the shared probabilistic leaderboard, rather than the value of the benchmark itself.

Critical strengths

1. A useful controlled target for covariate-aware forecasting

Pairing Replay and Novel futures over the same observed history is a strong experimental device. It keeps the historical evidence fixed while changing whether future covariates remain within or move beyond the observed range. This makes the resulting performance contrast more informative than aggregate accuracy on naturally occurring covariates alone.

The observed Replay-to-Novel degradation is substantial across every evaluated system, and changes in model ordering at larger covariate counts are particularly interesting. Strong Replay performance does not reliably predict strong Novel performance. This is a meaningful empirical finding and exposes a capability that existing forecasting benchmarks generally do not isolate.

2. Broad and carefully documented benchmark construction

The paper provides a relatively detailed account of the simulator trajectories, covariate selection, event structure, response functions, factor pools, task filtering, aggregation, model checkpoints, and LLM prompting protocol. It covers four distinct mechanistic domains and varies several dimensions of historical evidence and task complexity

The benchmark is also designed to support post hoc analysis. The authors state that they will release task construction code, evaluation code, per-task forecasts, and per-task metrics. This should allow researchers to recompute alternative metrics, construct matched subsets, and test interpretations beyond the aggregate leaderboard.

3. Reasonable handling and disclosure of model-family differences

The evaluation includes both numerical foundation models and frontier LLMs, making the empirical comparison timely and potentially useful. Failed LLM calls are assigned the naive forecast instead of being silently removed, and the paper reports the number of such failures. The paper also clearly discloses that the numerical models produce three quantiles while the LLMs produce only a median forecast

The common WAPE evaluation supports a legitimate point-forecast comparison across the two model families. In addition, the reported latency differences provide useful practical context, although API round-trip latency and on-device GPU execution are not directly controlled computational comparisons.

4. Appropriate positioning of the intervention terminology

The paper explicitly states that “intervention” refers to an externally driven change in the target through covariates and does not necessarily denote a formal causal do-operator intervention. It also distinguishes the simulator-generated baseline trajectory from the imposed covariate-response functions

This qualification is important and avoids a more serious causal interpretation problem.

Critical weaknesses

1. The failure-mode interpretation does not sufficiently separate extrapolation from insufficient identification

The paper’s central interpretation is that the Replay-to-Novel drop reflects a lack of robustness in learning covariate-target relationships. However, Appendix B.5 explicitly states that the construction does not ensure that these relationships are identifiable from each task’s history. Evidence can be particularly limited when multiple covariates are jointly set in only a small number of events, but non-identification can also arise in other configurations

This distinction matters. Multiple response functions may agree on the limited historical observations and produce different predictions at unseen covariate values. In such cases, the benchmark generator defines a unique target, but the forecaster may not have enough information to infer it. A large Novel error may therefore reflect insufficient evidence rather than an avoidable failure to extrapolate a learnable relationship.

The task axes also alter more than one statistical property at a time:

  • Active covariates are changed jointly and selected so their effects push the target in the same direction
  • Novel factors are constructed differently for direct and inverse relationships
  • Increasing the covariate count changes the included response functions, their combined response, and the joint damping
  • The feasible number of events depends on event and context length *The context/prediction axis changes context and forecast horizon together, with the largest pair differing somewhat across simulators

Therefore, marginal performance changes along these axes should not automatically be attributed to a single causal source of difficulty.

The current evaluation would be substantially more diagnostic with:

  • A covariate-agnostic forecaster
  • A simple response model fitted from the observed history
  • A model that knows the true response-function family but must estimate its parameters
  • An oracle-baseline forecast combined with an estimated covariate response
  • An oracle covariate response combined with an estimated baseline forecast
  • A clearly defined subset satisfying an explicit estimability or identification criterion

These additions would help determine whether the benchmark reveals failures of response learning, failures of ordinary time-series forecasting, or tasks for which the historical evidence is intrinsically insufficient.

2. The wQL leaderboard does not compare the same probabilistic forecasting objective across model families The numerical models output the 0.1, 0.5, and 0.9 quantiles, whereas the LLMs output only the median. Their reported wQL values are therefore computed from different quantile sets. The paper describes median-only scoring as disadvantaging the LLMs because calibrated outer quantiles can receive lower loss, but this does not establish a general ordering guarantee for the realized forecasts of the evaluated models

More fundamentally, the shared wQL column cannot establish which model family produces better probabilistic forecasts because one family does not produce a predictive distribution in this evaluation. This is especially important because calibrated uncertainty is presented as the appropriate response to evidence-poor or underidentified tasks, yet the LLMs have no mechanism to express that uncertainty under the current protocol.

The WAPE comparison is common across models and supports the paper’s point-accuracy result. However, the paper should distinguish more sharply between:

  • Comparative point-forecast accuracy, supported by WAPE
  • Within-family probabilistic evaluation of models producing common quantiles
  • Overall skill rankings that mix the two output protocols

A common output protocol would resolve this issue. Possible choices include eliciting the same quantiles from every model, sampling predictive distributions from the LLMs if technically feasible, or treating WAPE as the primary cross-family leaderboard and reporting probabilistic metrics only within a common-output subset.

3. Aggregate uncertainty and dependence across tasks are insufficiently characterize

The skill score is a geometric mean of clipped per-task error ratios, with each ratio constrained to the interval [10−2,102][10^{-2},10^2]. This may be a reasonable robust aggregate, but rankings could depend on the clipping choice, particularly when model differences are small

In addition, the 37,665 tasks reuse 100 baseline target series and a smaller set of simulator configurations. Thus, the nominal task count should not be interpreted as an equivalent number of independent experimental units. The paper currently reports point estimates without uncertainty intervals that account for this clustered structure.

I would like to see:

  • Bootstrap intervals clustered by baseline target and, if possible, simulator configuration
  • Simulator-balanced and target-balanced summaries
  • Sensitivity to the clipping bounds
  • Rank or pairwise-difference uncertainty for close-performing models
  • Matched Replay/Novel analyses using within-history differences

These analyses would make it clearer which leaderboard differences are stable and which should be treated as effectively tied.

Questions for authors
  1. What is learnable from the history? Can the authors define a subset of tasks for which the historical observations provide sufficient information to estimate the relevant covariate-response relationship? The criterion should specify the assumed response class, the parameters to be estimated, and the rank, coverage, or variation conditions required for estimation. More historical events alone may not guarantee identification when covariates move together.

  2. Can simple fitted-response baselines be added? Please compare the evaluated models with simple response models fitted only to the task history. A particularly informative baseline would know the true response-function family but still need to estimate its parameters from the observed events. This could distinguish failures of model inference from ambiguity inherent in the available history.

  3. Can the leaderboard use a common output protocol? Can all models be evaluated using the same quantile set or another common predictive representation? If this is not feasible, would the authors consider making WAPE the principal cross-family leaderboard and restricting probabilistic comparisons to models that produce the same quantiles?

  4. Can response learning be separated from baseline forecasting? Could the authors report oracle-component experiments in which the model is given either the intervention-free future trajectory or the true covariate-response contribution? These would separate ordinary time-series forecast error from error in estimating and extrapolating the covariate effect.

6 comments

We thank the reviewer for a very thorough review, and for finding the Replay/Novel pairing over the same history a strong experimental device, the degradation and the changes in model ranking a meaningful finding, and the construction broad and carefully documented. We address each point below.

W1 / Q1 / Q2 / Q4: Separating extrapolation from limited evidence

Thank you for these suggestions.

Covariate-agnostic forecasters. We re-ran each foundation model without covariates, so that it forecasts only the target's own dynamics:

ModelReplay: without → with covariatesNovel: without → with covariates
TimesFM-334.2 → 78.916.9 → 48.6
TS-ICL30.7 → 83.614.3 → 32.5
Chronos-229.9 → 72.314.9 → 35.7
t0-alpha28.5 → 64.615.4 → 33.5

The dynamics alone give 14 to 34% skill. The covariates add 36 to 53 points on Replay futures but only 18 to 32 on Novel futures: the models use the covariates far less well at unseen values.

Response models fitted to the history (Q2). We add Linear Regression, Ridge Regression and XGBoost, fit on each task's history to map the covariates to the target:

ModelReplayNovelNovel, tasks where each effect can be estimated
XGBoost65.06.714.8
Linear Regression49.0−6.7−2.7
Ridge Regression47.5−6.1−2.4
TimesFM-3, for reference78.948.655.4

They reproduce seen covariate values well but do not extrapolate, even where the history determines each effect. This shows that fitting a response without knowing its form is not enough.

Tasks where each effect can be estimated (Q1). We take as a response class a relative change in the target that is linear in each covariate's log factor, plus a constant. Its parameters can be estimated when the matrix of past events by log factors, with a constant column, has full rank: this needs at least one more event than covariates. It holds for 16,609 tasks:

ModelAll tasks: Replay → NovelTasks where each effect can be estimated: Replay → Novel
Claude-Opus-4.8 (High)84.8 → 50.385.7 → 64.8
TimesFM-378.9 → 48.679.4 → 55.4
TS-ICL83.6 → 32.583.8 → 36.5
Chronos-272.3 → 35.770.3 → 45.2
t0-alpha64.6 → 33.566.9 → 39.2

Novel skill rises as expected, but the drop from Replay remains 21 to 47 points and the ranking is unchanged: limited evidence explains part of the gap, not most of it.

A model that knows the true family, and oracle components (Q2, Q4). These would separate failures of inference from ambiguity in the history, and baseline error from response error. Two results address both for the drop in the paper studies. Where the history determines one effect per covariate (above), the drop stays large. And the two tasks of a Replay/Novel pair share the history and the target's course without the event, differing only in the future covariate values; their gap (30.3 to 51.1 points, bootstrap intervals above zero for every model; W3) therefore isolates the effect of unseen covariate values with the target's dynamics held fixed. The evaluated models are deliberately not given the family (the LLMs see placeholder variable names; Appendix D): each relationship must be inferred from the history, as in practice.

The drop thus reflects a failure to apply the relationships at unseen values, rather than a failure of ordinary forecasting. We are happy to add these results to the final version of the paper.

W1: Axes that change several properties at once

Thank you for listing these. Each case, and why it does not confound our conclusions:

  • Covariates set together in one direction. This is by design, so that their effects do not cancel; the same rule holds in every past event and in the future one, so the history shows how the effects add up (Appendix B.4).
  • Novel factors for direct and inverse relationships. The two rules differ (Appendix B.4): the largest factor seen plus an offset drawn from [5, 12] (direct), or the smallest seen times a value drawn from [0.1, 0.5] (inverse). Both place the future beyond every value seen, as the comparison with Replay futures requires. To compare them on one scale, we split Novel tasks into quarters by the ratio of the Novel factor to the most extreme factor seen (nearest: 1.7 to 2.4 times; farthest: 3.5 to 9 times):
ModelReplayNovel: nearest2nd3rdfarthest
Claude-Opus-4.8 (High)84.866.955.433.332.7
TimesFM-378.960.153.039.934.5
TS-ICL83.647.538.017.818.1
Chronos-272.351.843.818.816.9

Skill is lower in the farthest quarter than in the nearest for every model, and even the nearest quarter already costs a large share of skill. What drives the drop is therefore how far beyond the seen values a model must go, measured the same way under both rules, rather than which rule built the Novel value.

  • More covariates change the response and the damping. Covariates join the primary one in a fixed order, each added effect scaled by 1/√n when n > 1 are added (Appendix B.6); within a task, the covariates and the damping are fixed, so the history shows the joint rule to apply. To separate the number of covariates from the size of the response, we compare one with five covariates within groups of tasks with similar response sizes (Novel futures, one → five covariates):
ModelSmallest responses2nd3rdLargest responses
Claude-Opus-4.8 (High)59.5 → 7.770.2 → −32.384.3 → 45.585.2 → 27.6
TimesFM-358.8 → 40.664.5 → −82.074.8 → 67.654.5 → 30.0
TS-ICL29.1 → 6.647.3 → −193.870.6 → 47.840.1 → 28.9

Novel skill is lower with five covariates in every group, for every model, while Replay skill changes far less (Claude-Opus-4.8: at most 4.4 points). The drop with more covariates is therefore not explained by larger responses: extrapolating several covariates jointly is harder in itself. The strongly negative values in the second group come mostly from OpenDC tasks, where errors with five covariates become very large.

  • Number of events, event length and context. Events must fit in the context, so for each context and event length we build tasks with the fewest and the most events that fit. Each context has one horizon, the three smaller pairs being identical across simulators (Appendix B.3), and on Novel futures skill improves with longer contexts despite their longer horizons (Appendix F).

W2 / Q3: The wQL leaderboard across model families

Thank you for this point. Table 1 already reports WAPE next to wQL; we restate the results here along the split the reviewer suggests: WAPE to compare model families, and wQL only among models with the same quantiles.

  • Point accuracy across families (WAPE), on every model's median forecast. The LLMs lead (Claude-Opus-4.8 72.5%, Gemini-3.1-Pro 70.0%, foundation models at most 55.3%) in every bootstrap sample (W3), and every model drops sharply from Replay to Novel futures.
  • Probabilistic forecasts within the foundation models. All output the 0.1, 0.5 and 0.9 quantiles, and for their realized forecasts the outer quantiles do lower the loss: scored on the median alone (their WAPE), their skill is 11 to 17 points lower (Table 1). Their 0.1–0.9 intervals contain the true value less often on Novel than on Replay futures (TS-ICL 82% → 77%, TimesFM-3 80% → 72%, Chronos-2 76% → 69%, t0-alpha 66% → 64%; 80% if calibrated): with the least evidence, they become overconfident instead of widening their intervals, as wQL would reward (Appendix B.5).
  • Rankings mixing the two protocols. Ranking the models by WAPE or by wQL gives the same order, except for models within one point of each other.

The LLMs cannot express uncertainty under this protocol (Appendix B.5). We prompt them for the median because longer quantile outputs can fail: Gemini-3.1-Pro's answers were sometimes cut off before all quantiles on the longest horizons. We are happy to add this distinction to the final version of the paper.

W3: Uncertainty and dependence across tasks

Thank you for these suggestions. We take them in turn (wQL skill, %).

Intervals and balanced summaries. We bootstrap over the target series, resampling all tasks of a series together, and recompute the skill with each simulator, or each series, weighted equally:

ModelSkill [95% interval]Equal weight per simulatorEqual weight per series
Claude-Opus-4.8 (High)72.5 [66.2, 77.8]71.572.2
Gemini-3.1-Pro (High)70.0 [64.4, 75.0]68.369.9
TimesFM-367.0 [63.5, 70.4]65.966.5
TS-ICL66.7 [62.4, 70.6]66.266.0
Chronos-257.8 [54.0, 61.5]57.557.2
t0-alpha51.5 [47.3, 55.4]51.750.7

The ranking holds under both weightings, except that TimesFM-3 and TS-ICL swap under equal simulator weights; these two, and Gemini-3.1-Pro and TimesFM-3, are the only pairs not separated at 95%. The dependence between tasks of the same series or simulator therefore does not change the conclusions; only these two close pairs should be read as ties.

Clipping.

ModelNo clipPaper's clip [0.01, 100]Clip [0.1, 10]
Claude-Opus-4.8 (High)77.172.561.5
Gemini-3.1-Pro (High)74.970.058.4
TimesFM-365.967.065.4
TS-ICL64.866.761.6
Chronos-256.157.857.8
t0-alpha49.651.551.8

Removing the clip keeps the ranking. Only the tightest clip changes its top: capping any improvement over the naive baseline at a factor of 10 binds on about a quarter of the LLMs' tasks, against 13 to 18% for TimesFM-3 and TS-ICL, and so discounts the tasks where the LLMs improve most. Under WAPE, the LLMs stay first and second with every clip. The paper's clip affects under 10% of any model's tasks, so the reported skills are not driven by a few extreme tasks, and the ranking does not depend on the choice of clip within a wide range.

Matched Replay and Novel futures. Each Replay task has a Novel twin with the same history:

ModelReplay − Novel gap over pairs [95% interval]Pairs where Novel is worse
Claude-Opus-4.8 (High)34.5 [28.1, 41.2]72%
Gemini-3.1-Pro (High)41.7 [34.5, 48.6]72%
TimesFM-330.3 [23.9, 37.3]77%
TS-ICL51.1 [42.7, 59.6]88%
Chronos-236.6 [30.3, 43.7]80%
t0-alpha31.1 [25.0, 38.0]79%

The gap is above zero for every model, and on most pairs the Novel twin is the harder task. Since the twins share the same history and differ only in the future covariate values, the drop from Replay to Novel futures does not come from comparing different tasks: holding the history fixed, setting the covariates to unseen values alone makes the forecast harder. We are happy to add these results to the final version of the paper.

Reviewer_peVhOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the challenge that existing time-series forecasting benchmarks struggle to evaluate a model's predictive capability in scenarios involving unseen covariates. To this end, the authors propose constructing controllable relationships between covariates and target variables using semi-synthetic data, enabling the assessment of model performance across various future scenarios. Specifically, the paper introduces "WhatIf-Bench," which employs a simulator to generate foundational time series and artificially imposes the influence of covariates. It evaluates existing models through "Replay" and "Novel" tasks under diverse experimental settings. Experiments demonstrate that the performance of all models declines significantly on "Novel" tasks, a degradation that intensifies as the number of covariates increases, while access to a longer history of past events generally improves predictive performance.

Critical strengths

1.This paper addresses the limitation of existing time-series forecasting benchmarks in evaluating a model's predictive capability when faced with covariate values ​​not encountered during training, a well-motivated research problem with clear practical significance.

2.The authors construct a semi-synthetic benchmark comprising 37,665 tasks across four distinct domains, enabling a systematic evaluation of model performance under various controllable scenarios.

3.The study evaluates a range of time-series foundation models and LLMs, revealing performance disparities between replay tasks and novel tasks.

Critical weaknesses
  1. Rather than simulating the system response following an intervention via a simulator, the authors manually impose a predefined relationship between covariates and the target variable onto the base sequences generated by the simulation. Consequently, the benchmark functions more as a test of a model’s ability to learn and extrapolate from synthetic numerical functions; the extent to which these results reflect predictive capabilities in real-world, complex systems remains insufficiently validated.

  2. As noted in Appendix B.4, the "Novel" setting not only introduces previously unseen covariate values ​​but also constructs more extreme intervention conditions—by extending the historical maximum scaling factor (increasing it by 5–12 times) or further reducing the historical minimum. These adjustments can simultaneously alter both the magnitude of the target variable’s response and its numerical range. Therefore, any performance degradation observed in the "Novel" setting may stem from a combination of factors—extrapolation distance, intervention intensity, and shifts in the target value distribution—rather than being directly attributable to a failure of the model to adequately learn the relationship between the covariates and the target variable.

  3. In Appendix B.5, the authors acknowledge that for some tasks, the available historical data is insufficient to uniquely identify the relationship between the covariates and the target variable—a challenge particularly pronounced when there are many covariates but few historical events. In the absence of sufficient evidence, even a model that optimally utilizes the available information might fail to predict the specific extrapolation function predefined by the authors. Thus, prediction errors in these tasks cannot be taken as definitive proof that the model lacks the capacity to learn such relationships.

  4. The study primarily employs wQL, WAPE, and their aggregated scores to evaluate overall prediction error. However, these metrics are influenced by both the inherent dynamic prediction error of the target sequence itself and the prediction error related to the target’s response to covariates.

  5. In many cases, multiple covariates are altered simultaneously within a single event, with their joint impact on the target variable determined by manually defined combination rules and decay weights. Consequently, as the number of covariates increases, the tasks become more complex—not only by requiring the model to learn more relationships, but also by potentially altering the magnitude of the target response, the underlying functional structure, and the identifiability of the relationships. This makes it difficult to determine whether performance degradation arises from the inherent difficulty of modeling multiple covariates jointly, or from these concurrent, secondary changes.

Questions for authors

as weaknesses

3 comments

We thank the reviewer for their review, and for noting that the problem is well motivated and of clear practical significance, that the 37,665 tasks across four domains enable a systematic, controlled evaluation, and that WhatIf-Bench reveals clear gaps between Replay and Novel futures for foundation models and LLMs alike. We address each point below.

W1: Imposed relationships, and how far results reflect real systems

Thank you for raising this. To score a forecast at covariate values never seen in the history, we need a ground truth in which (i) the relationship seen in the history still holds at those values, so the future can be forecast in principle, and (ii) the target moves only because of the covariates, so any error is due to the model. Real data records only one course of events, and simulators re-run for each event often change operating regime at unseen values and add noise unrelated to the covariates (Appendix B.1). We therefore borrow the simulators' mechanisms: the targets and covariates are the simulators' own variables (e.g. a host's CPU utilization and its core count), the target series come from simulator runs on recorded inputs (e.g. a real workload trace, recorded weather), and each relationship is derived from the simulator's documented model (Appendix B.6). This follows the semi-synthetic benchmarks of causal inference (Hill, 2011; Dorie et al., 2019).

WhatIf-Bench thus tests a capability that forecasting in real systems requires: learning a covariate-target relationship from the context and applying it at new values. Models see only numbers or placeholder variable names, so none can rely on knowing the physics of a system. Even with a fixed relationship and no unrelated noise, every model loses much of its skill from Replay to Novel futures, a failure that real-world benchmarks such as fev-bench (Shchur et al., 2025) cannot reveal, since they test only the covariate values that occurred. Validating these findings on real data is an important next step, which we list as a limitation (Appendix E).

W2: Whether the Novel setting mixes several factors

Thank you for this careful point. To make the construction concrete: the Novel factor is the largest factor in the history plus an offset of 5 to 12 (Appendix B.4), so a largest factor of 8 becomes 13 to 20, i.e. 1.6 to 2.5 times larger. Replay and Novel futures use the same relationship with different covariate values, so extrapolation distance and intervention intensity are one quantity: how far beyond the history the model must carry the relationship, which Novel futures are meant to test. We examine it and the shift of the target values separately:

Extrapolation distance. In quarters of the Novel tasks by this distance, skill is lower in the farthest than in the nearest for every model, also at a fixed number of past events. Even the nearest (1.7 to 2.4 times the most extreme factor seen) costs much: Claude-Opus-4.8 drops from 84.8% on Replay to 66.9%, TS-ICL from 83.6% to 47.5%.

Shift of the target values. If the drop came from the target leaving the range seen in its history, Novel futures whose target stays within that range would score close to Replay futures that do. These two groups have responses of similar size (wQL skill, %):

ModelReplay, target within rangeNovel, target within rangeNovel, all tasks
Claude-Opus-4.8 (High)84.438.050.3
TimesFM-379.331.948.6
TS-ICL83.3−20.232.5
Chronos-271.3−6.135.7

Within the seen range, Novel skill is still 46 to 104 points below Replay, and even below the Novel average. The drop is therefore not explained by a shift of the target values, but by the unseen covariate values.

W3: Tasks with insufficient historical evidence

Thank you for this point. We do not draw conclusions from single tasks: limited-evidence tasks are included because they occur in practice, the number of events is an axis reported on its own, and on such tasks wQL scores a forecaster that widens its intervals better than one that commits to a wrong response (Appendix B.5). To check that our conclusions do not rest on them, we evaluate on the tasks whose past events set the covariates to enough different combinations that one effect per covariate, plus a constant, can be estimated from them unambiguously; this needs at least one more past event than the task has covariates. This gives 16,609 tasks (wQL skill, %):

ModelAll tasks: Replay → NovelTasks with enough evidence: Replay → Novel
Claude-Opus-4.8 (High)84.8 → 50.385.7 → 64.8
TimesFM-378.9 → 48.679.4 → 55.4
TS-ICL83.6 → 32.583.8 → 36.5
Chronos-272.3 → 35.770.3 → 45.2
t0-alpha64.6 → 33.566.9 → 39.2

With enough evidence, Novel skill improves as expected, but the drop from Replay remains large for every model (21 to 47 points) and the order of the models is unchanged: limited evidence explains part of the gap, not most of it. We are happy to add these results to the final version of the paper.

W4: Separating errors on the target's dynamics from errors on the covariate response

Thank you for this suggestion. We use wQL and WAPE, standard forecasting metrics, and aggregate them as in fev-bench. To separate the two sources of error, we re-run each foundation model without any covariates. Given only the target's history, a model can forecast only the target's own dynamics, so the difference between the two runs measures what the model gains from the covariates (wQL skill, %):

ModelReplay: without → with covariatesNovel: without → with covariates
TimesFM-334.2 → 78.916.9 → 48.6
TS-ICL30.7 → 83.614.3 → 32.5
Chronos-229.9 → 72.314.9 → 35.7
t0-alpha28.5 → 64.615.4 → 33.5

Forecasting the dynamics alone gives only 14 to 34% skill, so most of each model's skill comes from the covariates. This gain shrinks sharply on Novel futures, from 36–53 points to 18–32 points: models use the covariates far less well at unseen values. We are happy to add these results to the final version of the paper.

W5: Effects of the number of covariates

Thank you for this point. Setting more covariates at once does change several things together, as in real systems. Within a task, however, the rule combining the covariates' effects is the same in every past event and in the future one, so the history shows the joint relationship to apply (Appendix B.6). To separate the number of covariates from the size of the response, we compare one with five covariates within groups of tasks with similar response sizes (Novel futures, wQL skill, %, one covariate → five covariates):

ModelSmallest responses2nd3rdLargest responses
Claude-Opus-4.8 (High)59.5 → 7.770.2 → −32.384.3 → 45.585.2 → 27.6
TimesFM-358.8 → 40.664.5 → −82.074.8 → 67.654.5 → 30.0
TS-ICL29.1 → 6.647.3 → −193.870.6 → 47.840.1 → 28.9

(The strongly negative values come mostly from OpenDC tasks, where errors with five covariates become very large.)

Within every group, Novel skill is lower with five covariates than with one for every leaderboard model, while on Replay futures the changes for these models are small (at most 7 points) and go in both directions. On the tasks with enough evidence to learn the relationships (W3), skill also falls with more covariates: Claude-Opus-4.8 goes from 81.8% with one covariate to 69.4% with five (Replay and Novel futures together). The difficulty thus comes from extrapolating several covariates jointly, not only from a larger response or from too little evidence.


References

  • Dorie, V., Hill, J., Shalit, U., Scott, M., and Cervone, D. (2019). Automated versus Do-It-Yourself Methods for Causal Inference: Lessons Learned from a Data Analysis Competition. Statistical Science, 34(1), 43–68.
  • Hill, J. L. (2011). Bayesian Nonparametric Modeling for Causal Inference. Journal of Computational and Graphical Statistics, 20(1), 217–240.
  • Shchur, O., et al. (2025). fev-bench: A Realistic Benchmark for Time Series Forecasting. arXiv:2509.26468.

I appreciate the author’s detailed response. Through supplementary experiments, the authors have, to some extent, addressed issues such as the discrepancy in target value ranges between replay and novel tasks, the problem of insufficient historical evidence, and the variation in target response magnitudes within multi-covariate settings. Nevertheless, the paper still lacks validation against responses to interventions in real-world systems; discrepancies between the artificially constructed covariate relationships and those found in actual systems persist. Furthermore, as the current evaluation relies primarily on aggregate prediction error, without directly measuring the model’s ability to learn the underlying relationships between covariates and target variables, the construct validity of the benchmark remains to be further verified. Finally, given that this is a 4-point scale and the other two reviewers have also assigned a score of 4, and considering that the aforementioned issues suggest the paper does not yet reach the threshold for an oral presentation, I decide to maintain my score. That said, I believe the paper is worthy of acceptance; had the scores from the other two reviewers been different, I would have been open to increasing my rating to 3.

Reviewer_dDinOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WhatIf-Bench, a semi-synthetic benchmark for intervention-aware time-series forecasting. Built on four mechanism-based simulators, the benchmark evaluates models under Replay and Novel future covariate scenarios, with systematic variations in task difficulty. It contains 37,665 forecasting tasks across multiple application domains. Experiments on time-series foundation models and LLMs reveal substantial performance degradation under Novel scenarios, highlighting the limitations of existing models in extrapolating to unseen covariate values.

Critical strengths
  1. WHATIF-BENCH addresses an important gap in time-series forecasting evaluation by systematically testing models under unseen future covariate scenarios.

  2. The benchmark uses mechanism-based simulators to generate controlled interventions with known outcomes, providing a meaningful setting for evaluating intervention-aware forecasting.

  3. The experiments cover a large number of tasks, multiple model families, and several difficulty factors, providing useful insights into model limitations and generalization capabilities.

Critical weaknesses
  1. The covariate-target relationships generated by the simulators are not sufficiently explained in the main paper. It remains unclear whether the benchmark adequately captures complex interactions, temporal dependencies, and nonstationary behavior found in real-world systems.

  2. The evaluation may not be fully fair across model families. In particular, LLMs only produce median predictions, while other models provide multiple quantiles for wQL evaluation. Additionally, the paper lacks comparisons with established conditional forecasting methods such as VAR, BSTS, TFT, and DeepAR with exogenous variables.

  3. The experimental analysis is incomplete. The paper does not sufficiently investigate how performance changes with extrapolation distance, and it lacks probabilistic calibration analysis and detailed results across individual simulators. Filtering out tasks with negligible covariate effects may also limit the benchmark's ability to evaluate false responses to irrelevant interventions.

Questions for authors
  1. Can the authors clarify how covariate-target relationships are constructed in each simulator, particularly regarding variable interactions, temporal lags, and nonstationarity?

  2. How are Novel covariate values determined, and how does model performance change as the extrapolation distance from historical observations increases?

  3. Can the authors provide fairer probabilistic evaluations across model families and include established conditional forecasting baselines such as VAR, BSTS, TFT, and DeepAR?

  4. Why are tasks with negligible covariate effects excluded? Could including such tasks help evaluate whether models incorrectly respond to irrelevant interventions?

4 comments

We thank the reviewer for their careful review, and for noting that WhatIf-Bench fills an important gap, gives controlled interventions with known outcomes, and covers many tasks, models and difficulty factors. We address each point below.

W1 / Q1: How the covariate-target relationships are built

Thank you for this question. The relationships are detailed in Appendix B.6; we summarize them here, and are happy to add this summary to the main paper in the final version.

Why we impose them. Scoring a forecast at unseen covariate values requires a ground truth in which (i) the relationship seen in the history still holds at those values, so the future can be forecast in principle, and (ii) the target moves only because of the covariates, so any error is due to the model. Real data cannot provide this, and simulators run in their original environment are not built for it: at unseen values they often change operating regime, and they add noise (Appendix B.1). We therefore borrow their mechanisms: each simulator produces the target series, and we impose relationships derived from its documented model, e.g. a host's power draw is an idle power plus a part that scales with clock speed, as in OpenDC (Mastenbroek et al., 2021). This follows the semi-synthetic benchmarks of causal inference (Hill, 2011; Dorie et al., 2019).

SimulatorTarget series driven byForms of the relationshipsSeveral covariates at once
OpenDCa recorded workload traceclose to proportional, inverse, sublinear, asymmetricweighted; effects across hosts
GridLAB-Da year of recorded weatheroffsets, asymmetricmultiplicative or additive
EPANETa pump switching with the tank levelsublinearmultiplicative
SUMOrush-hour traffic demandexponential, decaying, piecewiseadditive or multiplicative

Interactions and dynamics. An event sets up to five covariates, whose effects combine by fixed rules (weights and a joint damping), including effects across variables and in one direction only (e.g. memory matters only when reduced). The series inherit each system's dynamics (workload bursts, the daily weather cycle, rush hours, tank cycles), which a model must separate from the covariate effects. Delayed effects and regime changes are future work (Appendix E). Even so, every model drops sharply from Replay to Novel futures (Claude-Opus-4.8 from 84.8% to 50.3%, TS-ICL from 83.6% to 32.5%), a failure that existing benchmarks cannot show (Table 1): fev-bench (Shchur et al., 2025) tests only covariate values that occurred, and Berthelier et al. (2026) use simple relationships without unseen values.

W2 / Q3: Fairness across model families, and classical baselines

Thank you for both points.

Classical baselines. These methods must be trained, and a task offers only its own history: one series of 1,024 to about 8,000 steps, in which the covariates leave their base values for only 64 to 4,096 steps. TFT and DeepAR would overfit such data, and VAR and BSTS respond to new covariate values through a regression on the covariates. We therefore add such regression baselines, fit on each task's history. WAPE skill (%), which scores every model on its median forecast:

ModelOverallReplayNovel
Claude-Opus-4.8 (High)72.584.850.3
Gemini-3.1-Pro (High)70.084.342.6
TS-ICL55.376.515.2
TimesFM-354.569.332.6
Chronos-242.061.213.5
t0-alpha34.749.515.6
XGBoost42.965.06.7
Linear Regression26.249.0−6.7

As expected for models fit on the history, they do well on Replay futures, which repeat seen covariate values; XGBoost is thus close to some foundation models overall. On Novel futures they fall the most, Linear Regression below the naive baseline.

LLMs. wQL applies the same loss to the quantiles each model outputs; for a median-only model it equals WAPE, so no model is penalized or credited for quantiles. WAPE (above) compares all families on the same median forecast, and the conclusions are unchanged. We ask the LLMs for the median because longer quantile outputs can fail: Gemini-3.1-Pro's answers were sometimes cut off before all quantiles on the longest horizons. We are happy to state explicitly in the final version of the paper that the comparison across model families rests on WAPE.

W3 / Q2: Extrapolation distance

Thank you for this question. A Novel factor exceeds the most extreme factor in the history by a random amount: the largest plus an offset drawn from [5, 12], or, for covariates that lower the target, the smallest times a value drawn from [0.1, 0.5] (Appendix B.4). The relationships keep the same direction beyond the history (Appendix B.2). We split the Novel tasks into quarters by this distance; the nearest have a future factor 1.7 to 2.4 times the most extreme one seen, the farthest 3.5 to 9 times. wQL skill (%):

ModelReplayNovel: nearest2nd3rdfarthest
Claude-Opus-4.8 (High)84.866.955.433.332.7
TimesFM-378.960.153.039.934.5
TS-ICL83.647.538.017.818.1
Chronos-272.351.843.818.816.9
Linear Regression49.016.89.9−22.3−51.5

Although the relationships keep their direction, skill falls steadily with the distance for every model. Even the nearest quarter costs a large share: Claude-Opus-4.8 drops from 84.8% to 66.9%, TS-ICL from 83.6% to 47.5%. Current models fail to extrapolate even slightly beyond the history.

W3: Calibration

Thank you for this suggestion. To check whether the foundation models' quantile forecasts can be trusted, we measure how often the true value falls inside the interval between their 0.1 and 0.9 quantiles, over all forecast steps. For a calibrated forecaster, this share is 80%.

TS-ICLTimesFM-3Chronos-2t0-alpha
Replay82%80%76%66%
Novel77%72%69%64%

On Replay futures, TS-ICL and TimesFM-3 are close to calibrated. On Novel futures, coverage falls for every model: the intervals become too narrow exactly when the models extrapolate to unseen covariate values. The models are therefore not only less accurate on Novel futures but also overconfident, which matters for scenario analysis, where users rely on the intervals to judge a forecast. The LLMs output only a median, so they have no interval to evaluate.

W3: Results per simulator

Thank you for this suggestion. Below is each model's skill on each simulator separately, from Replay to Novel futures (wQL skill, %). We are happy to add these results to the final version of the paper.

ModelEPANETGridLAB-DOpenDCSUMO
Claude-Opus-4.8 (High)54.0 → 36.564.4 → 52.085.4 → 57.696.7 → 57.0
TimesFM-366.2 → 63.068.4 → 51.480.3 → 28.988.7 → 40.1
TS-ICL60.4 → 40.470.8 → 45.787.6 → 13.393.9 → 30.5
Chronos-260.5 → 44.064.3 → 47.473.9 → 12.082.4 → 36.4
XGBoost5.9 → 3.430.6 → 10.882.1 → 2.085.1 → 12.3

The drop from Replay to Novel futures holds for every model on every simulator, so the main finding does not rest on any single domain. It is largest on OpenDC and SUMO, where the foundation models fall to 12–40% on Novel futures. The best model also varies: TimesFM-3 leads on EPANET Novel futures (63.0%), and Claude-Opus-4.8 on the other three, so no model handles Novel futures well everywhere.

W3 / Q4: Excluding tasks with negligible covariate effects

Thank you for this question. We drop a task if an event moves the target by less than 10% from its value without the event (Appendix B.2), removing 2,335 of 40,000 tasks. In such a task, ignoring the covariates costs almost nothing (about 3% for a 3% effect), so the task cannot separate models that use the covariates from those that do not. Testing false responses to irrelevant interventions is valuable, but it needs covariates with no effect rather than weak ones; we list it as future work (Appendix E), and WhatIf-Bench supports it by adding a covariate whose relationship leaves the target unchanged.

References

  • Berthelier, G., et al. (2026). Investigating simple target-covariate relationships for Chronos-2 and TabPFN-TS. arXiv:2605.12200.
  • Dorie, V., Hill, J., Shalit, U., Scott, M., and Cervone, D. (2019). Automated versus Do-It-Yourself Methods for Causal Inference: Lessons Learned from a Data Analysis Competition. Statistical Science, 34(1), 43–68.
  • Hill, J. L. (2011). Bayesian Nonparametric Modeling for Causal Inference. Journal of Computational and Graphical Statistics, 20(1), 217–240.
  • Mastenbroek, F., et al. (2021). OpenDC 2.0: Convenient Modeling and Simulation of Emerging Technologies in Cloud Datacenters. CCGrid 2021, 455–464.
  • Shchur, O., et al. (2025). fev-bench: A Realistic Benchmark for Time Series Forecasting. arXiv:2509.26468.

Thanks for the detailed clarification. Most of my concern have been addressed. I will raise my score to 4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.