- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces WHATIF-BENCH, a semi-synthetic benchmark for evaluating whether forecasters can use historical and future covariates to predict a target’s response to future events. The benchmark begins with baseline trajectories from four mechanistic simulators covering data centers, power grids, road traffic, and water networks. It then overlays controlled, documented covariate-target response functions on these trajectories. Each task contains a history of events that set one or more covariates and asks the model to forecast the target under a future event
The key experimental comparison pairs two futures with the same observed history. A Replay future repeats previously observed covariate values, whereas a Novel future moves the covariates beyond their historical range. The benchmark additionally varies the number of active covariates, number and duration of historical events, and context/prediction length. The resulting benchmark contains 37,665 selected tasks across 100 target series
The paper evaluates six numerical foundation-model systems and two frontier LLMs. It reports that all evaluated models perform substantially better on Replay than Novel futures, that Novel performance generally deteriorates as more covariates are jointly varied, and that additional historical events tend to improve Novel performance. For example, Claude Opus decreases from 84.8% Replay skill to 50.3% Novel skill, while TS-ICL decreases from 83.6% to 32.5%. The model ordering also changes as the number of covariates increases
Overall, I find the benchmark construction useful and the main empirical phenomenon compelling. My reservations concern the interpretation of the failure modes and the comparability of the shared probabilistic leaderboard, rather than the value of the benchmark itself.
Critical strengths
1. A useful controlled target for covariate-aware forecasting
Pairing Replay and Novel futures over the same observed history is a strong experimental device. It keeps the historical evidence fixed while changing whether future covariates remain within or move beyond the observed range. This makes the resulting performance contrast more informative than aggregate accuracy on naturally occurring covariates alone.
The observed Replay-to-Novel degradation is substantial across every evaluated system, and changes in model ordering at larger covariate counts are particularly interesting. Strong Replay performance does not reliably predict strong Novel performance. This is a meaningful empirical finding and exposes a capability that existing forecasting benchmarks generally do not isolate.
2. Broad and carefully documented benchmark construction
The paper provides a relatively detailed account of the simulator trajectories, covariate selection, event structure, response functions, factor pools, task filtering, aggregation, model checkpoints, and LLM prompting protocol. It covers four distinct mechanistic domains and varies several dimensions of historical evidence and task complexity
The benchmark is also designed to support post hoc analysis. The authors state that they will release task construction code, evaluation code, per-task forecasts, and per-task metrics. This should allow researchers to recompute alternative metrics, construct matched subsets, and test interpretations beyond the aggregate leaderboard.
3. Reasonable handling and disclosure of model-family differences
The evaluation includes both numerical foundation models and frontier LLMs, making the empirical comparison timely and potentially useful. Failed LLM calls are assigned the naive forecast instead of being silently removed, and the paper reports the number of such failures. The paper also clearly discloses that the numerical models produce three quantiles while the LLMs produce only a median forecast
The common WAPE evaluation supports a legitimate point-forecast comparison across the two model families. In addition, the reported latency differences provide useful practical context, although API round-trip latency and on-device GPU execution are not directly controlled computational comparisons.
4. Appropriate positioning of the intervention terminology
The paper explicitly states that “intervention” refers to an externally driven change in the target through covariates and does not necessarily denote a formal causal do-operator intervention. It also distinguishes the simulator-generated baseline trajectory from the imposed covariate-response functions
This qualification is important and avoids a more serious causal interpretation problem.
Critical weaknesses
1. The failure-mode interpretation does not sufficiently separate extrapolation from insufficient identification
The paper’s central interpretation is that the Replay-to-Novel drop reflects a lack of robustness in learning covariate-target relationships. However, Appendix B.5 explicitly states that the construction does not ensure that these relationships are identifiable from each task’s history. Evidence can be particularly limited when multiple covariates are jointly set in only a small number of events, but non-identification can also arise in other configurations
This distinction matters. Multiple response functions may agree on the limited historical observations and produce different predictions at unseen covariate values. In such cases, the benchmark generator defines a unique target, but the forecaster may not have enough information to infer it. A large Novel error may therefore reflect insufficient evidence rather than an avoidable failure to extrapolate a learnable relationship.
The task axes also alter more than one statistical property at a time:
- Active covariates are changed jointly and selected so their effects push the target in the same direction
- Novel factors are constructed differently for direct and inverse relationships
- Increasing the covariate count changes the included response functions, their combined response, and the joint damping
- The feasible number of events depends on event and context length *The context/prediction axis changes context and forecast horizon together, with the largest pair differing somewhat across simulators
Therefore, marginal performance changes along these axes should not automatically be attributed to a single causal source of difficulty.
The current evaluation would be substantially more diagnostic with:
- A covariate-agnostic forecaster
- A simple response model fitted from the observed history
- A model that knows the true response-function family but must estimate its parameters
- An oracle-baseline forecast combined with an estimated covariate response
- An oracle covariate response combined with an estimated baseline forecast
- A clearly defined subset satisfying an explicit estimability or identification criterion
These additions would help determine whether the benchmark reveals failures of response learning, failures of ordinary time-series forecasting, or tasks for which the historical evidence is intrinsically insufficient.
2. The wQL leaderboard does not compare the same probabilistic forecasting objective across model families The numerical models output the 0.1, 0.5, and 0.9 quantiles, whereas the LLMs output only the median. Their reported wQL values are therefore computed from different quantile sets. The paper describes median-only scoring as disadvantaging the LLMs because calibrated outer quantiles can receive lower loss, but this does not establish a general ordering guarantee for the realized forecasts of the evaluated models
More fundamentally, the shared wQL column cannot establish which model family produces better probabilistic forecasts because one family does not produce a predictive distribution in this evaluation. This is especially important because calibrated uncertainty is presented as the appropriate response to evidence-poor or underidentified tasks, yet the LLMs have no mechanism to express that uncertainty under the current protocol.
The WAPE comparison is common across models and supports the paper’s point-accuracy result. However, the paper should distinguish more sharply between:
- Comparative point-forecast accuracy, supported by WAPE
- Within-family probabilistic evaluation of models producing common quantiles
- Overall skill rankings that mix the two output protocols
A common output protocol would resolve this issue. Possible choices include eliciting the same quantiles from every model, sampling predictive distributions from the LLMs if technically feasible, or treating WAPE as the primary cross-family leaderboard and reporting probabilistic metrics only within a common-output subset.
3. Aggregate uncertainty and dependence across tasks are insufficiently characterize
The skill score is a geometric mean of clipped per-task error ratios, with each ratio constrained to the interval . This may be a reasonable robust aggregate, but rankings could depend on the clipping choice, particularly when model differences are small
In addition, the 37,665 tasks reuse 100 baseline target series and a smaller set of simulator configurations. Thus, the nominal task count should not be interpreted as an equivalent number of independent experimental units. The paper currently reports point estimates without uncertainty intervals that account for this clustered structure.
I would like to see:
- Bootstrap intervals clustered by baseline target and, if possible, simulator configuration
- Simulator-balanced and target-balanced summaries
- Sensitivity to the clipping bounds
- Rank or pairwise-difference uncertainty for close-performing models
- Matched Replay/Novel analyses using within-history differences
These analyses would make it clearer which leaderboard differences are stable and which should be treated as effectively tied.
Questions for authors
-
What is learnable from the history? Can the authors define a subset of tasks for which the historical observations provide sufficient information to estimate the relevant covariate-response relationship? The criterion should specify the assumed response class, the parameters to be estimated, and the rank, coverage, or variation conditions required for estimation. More historical events alone may not guarantee identification when covariates move together.
-
Can simple fitted-response baselines be added? Please compare the evaluated models with simple response models fitted only to the task history. A particularly informative baseline would know the true response-function family but still need to estimate its parameters from the observed events. This could distinguish failures of model inference from ambiguity inherent in the available history.
-
Can the leaderboard use a common output protocol? Can all models be evaluated using the same quantile set or another common predictive representation? If this is not feasible, would the authors consider making WAPE the principal cross-family leaderboard and restricting probabilistic comparisons to models that produce the same quantiles?
-
Can response learning be separated from baseline forecasting? Could the authors report oracle-component experiments in which the model is given either the intervention-free future trajectory or the true covariate-response contribution? These would separate ordinary time-series forecast error from error in estimating and extrapolating the covariate effect.
6 comments
W1: Axes that change several properties at once
Thank you for listing these. Each case, and why it does not confound our conclusions:
- Covariates set together in one direction. This is by design, so that their effects do not cancel; the same rule holds in every past event and in the future one, so the history shows how the effects add up (Appendix B.4).
- Novel factors for direct and inverse relationships. The two rules differ (Appendix B.4): the largest factor seen plus an offset drawn from [5, 12] (direct), or the smallest seen times a value drawn from [0.1, 0.5] (inverse). Both place the future beyond every value seen, as the comparison with Replay futures requires. To compare them on one scale, we split Novel tasks into quarters by the ratio of the Novel factor to the most extreme factor seen (nearest: 1.7 to 2.4 times; farthest: 3.5 to 9 times):
| Model | Replay | Novel: nearest | 2nd | 3rd | farthest |
|---|---|---|---|---|---|
| Claude-Opus-4.8 (High) | 84.8 | 66.9 | 55.4 | 33.3 | 32.7 |
| TimesFM-3 | 78.9 | 60.1 | 53.0 | 39.9 | 34.5 |
| TS-ICL | 83.6 | 47.5 | 38.0 | 17.8 | 18.1 |
| Chronos-2 | 72.3 | 51.8 | 43.8 | 18.8 | 16.9 |
Skill is lower in the farthest quarter than in the nearest for every model, and even the nearest quarter already costs a large share of skill. What drives the drop is therefore how far beyond the seen values a model must go, measured the same way under both rules, rather than which rule built the Novel value.
- More covariates change the response and the damping. Covariates join the primary one in a fixed order, each added effect scaled by 1/√n when n > 1 are added (Appendix B.6); within a task, the covariates and the damping are fixed, so the history shows the joint rule to apply. To separate the number of covariates from the size of the response, we compare one with five covariates within groups of tasks with similar response sizes (Novel futures, one → five covariates):
| Model | Smallest responses | 2nd | 3rd | Largest responses |
|---|---|---|---|---|
| Claude-Opus-4.8 (High) | 59.5 → 7.7 | 70.2 → −32.3 | 84.3 → 45.5 | 85.2 → 27.6 |
| TimesFM-3 | 58.8 → 40.6 | 64.5 → −82.0 | 74.8 → 67.6 | 54.5 → 30.0 |
| TS-ICL | 29.1 → 6.6 | 47.3 → −193.8 | 70.6 → 47.8 | 40.1 → 28.9 |
Novel skill is lower with five covariates in every group, for every model, while Replay skill changes far less (Claude-Opus-4.8: at most 4.4 points). The drop with more covariates is therefore not explained by larger responses: extrapolating several covariates jointly is harder in itself. The strongly negative values in the second group come mostly from OpenDC tasks, where errors with five covariates become very large.
- Number of events, event length and context. Events must fit in the context, so for each context and event length we build tasks with the fewest and the most events that fit. Each context has one horizon, the three smaller pairs being identical across simulators (Appendix B.3), and on Novel futures skill improves with longer contexts despite their longer horizons (Appendix F).
W2 / Q3: The wQL leaderboard across model families
Thank you for this point. Table 1 already reports WAPE next to wQL; we restate the results here along the split the reviewer suggests: WAPE to compare model families, and wQL only among models with the same quantiles.
- Point accuracy across families (WAPE), on every model's median forecast. The LLMs lead (Claude-Opus-4.8 72.5%, Gemini-3.1-Pro 70.0%, foundation models at most 55.3%) in every bootstrap sample (W3), and every model drops sharply from Replay to Novel futures.
- Probabilistic forecasts within the foundation models. All output the 0.1, 0.5 and 0.9 quantiles, and for their realized forecasts the outer quantiles do lower the loss: scored on the median alone (their WAPE), their skill is 11 to 17 points lower (Table 1). Their 0.1–0.9 intervals contain the true value less often on Novel than on Replay futures (TS-ICL 82% → 77%, TimesFM-3 80% → 72%, Chronos-2 76% → 69%, t0-alpha 66% → 64%; 80% if calibrated): with the least evidence, they become overconfident instead of widening their intervals, as wQL would reward (Appendix B.5).
- Rankings mixing the two protocols. Ranking the models by WAPE or by wQL gives the same order, except for models within one point of each other.
The LLMs cannot express uncertainty under this protocol (Appendix B.5). We prompt them for the median because longer quantile outputs can fail: Gemini-3.1-Pro's answers were sometimes cut off before all quantiles on the longest horizons. We are happy to add this distinction to the final version of the paper.
W3: Uncertainty and dependence across tasks
Thank you for these suggestions. We take them in turn (wQL skill, %).
Intervals and balanced summaries. We bootstrap over the target series, resampling all tasks of a series together, and recompute the skill with each simulator, or each series, weighted equally:
| Model | Skill [95% interval] | Equal weight per simulator | Equal weight per series |
|---|---|---|---|
| Claude-Opus-4.8 (High) | 72.5 [66.2, 77.8] | 71.5 | 72.2 |
| Gemini-3.1-Pro (High) | 70.0 [64.4, 75.0] | 68.3 | 69.9 |
| TimesFM-3 | 67.0 [63.5, 70.4] | 65.9 | 66.5 |
| TS-ICL | 66.7 [62.4, 70.6] | 66.2 | 66.0 |
| Chronos-2 | 57.8 [54.0, 61.5] | 57.5 | 57.2 |
| t0-alpha | 51.5 [47.3, 55.4] | 51.7 | 50.7 |
The ranking holds under both weightings, except that TimesFM-3 and TS-ICL swap under equal simulator weights; these two, and Gemini-3.1-Pro and TimesFM-3, are the only pairs not separated at 95%. The dependence between tasks of the same series or simulator therefore does not change the conclusions; only these two close pairs should be read as ties.
Clipping.
| Model | No clip | Paper's clip [0.01, 100] | Clip [0.1, 10] |
|---|---|---|---|
| Claude-Opus-4.8 (High) | 77.1 | 72.5 | 61.5 |
| Gemini-3.1-Pro (High) | 74.9 | 70.0 | 58.4 |
| TimesFM-3 | 65.9 | 67.0 | 65.4 |
| TS-ICL | 64.8 | 66.7 | 61.6 |
| Chronos-2 | 56.1 | 57.8 | 57.8 |
| t0-alpha | 49.6 | 51.5 | 51.8 |
Removing the clip keeps the ranking. Only the tightest clip changes its top: capping any improvement over the naive baseline at a factor of 10 binds on about a quarter of the LLMs' tasks, against 13 to 18% for TimesFM-3 and TS-ICL, and so discounts the tasks where the LLMs improve most. Under WAPE, the LLMs stay first and second with every clip. The paper's clip affects under 10% of any model's tasks, so the reported skills are not driven by a few extreme tasks, and the ranking does not depend on the choice of clip within a wide range.
Matched Replay and Novel futures. Each Replay task has a Novel twin with the same history:
| Model | Replay − Novel gap over pairs [95% interval] | Pairs where Novel is worse |
|---|---|---|
| Claude-Opus-4.8 (High) | 34.5 [28.1, 41.2] | 72% |
| Gemini-3.1-Pro (High) | 41.7 [34.5, 48.6] | 72% |
| TimesFM-3 | 30.3 [23.9, 37.3] | 77% |
| TS-ICL | 51.1 [42.7, 59.6] | 88% |
| Chronos-2 | 36.6 [30.3, 43.7] | 80% |
| t0-alpha | 31.1 [25.0, 38.0] | 79% |
The gap is above zero for every model, and on most pairs the Novel twin is the harder task. Since the twins share the same history and differ only in the future covariate values, the drop from Replay to Novel futures does not come from comparing different tasks: holding the history fixed, setting the covariates to unseen values alone makes the forecast harder. We are happy to add these results to the final version of the paper.
We thank the reviewer for a very thorough review, and for finding the Replay/Novel pairing over the same history a strong experimental device, the degradation and the changes in model ranking a meaningful finding, and the construction broad and carefully documented. We address each point below.
W1 / Q1 / Q2 / Q4: Separating extrapolation from limited evidence
Thank you for these suggestions.
Covariate-agnostic forecasters. We re-ran each foundation model without covariates, so that it forecasts only the target's own dynamics:
The dynamics alone give 14 to 34% skill. The covariates add 36 to 53 points on Replay futures but only 18 to 32 on Novel futures: the models use the covariates far less well at unseen values.
Response models fitted to the history (Q2). We add Linear Regression, Ridge Regression and XGBoost, fit on each task's history to map the covariates to the target:
They reproduce seen covariate values well but do not extrapolate, even where the history determines each effect. This shows that fitting a response without knowing its form is not enough.
Tasks where each effect can be estimated (Q1). We take as a response class a relative change in the target that is linear in each covariate's log factor, plus a constant. Its parameters can be estimated when the matrix of past events by log factors, with a constant column, has full rank: this needs at least one more event than covariates. It holds for 16,609 tasks:
Novel skill rises as expected, but the drop from Replay remains 21 to 47 points and the ranking is unchanged: limited evidence explains part of the gap, not most of it.
A model that knows the true family, and oracle components (Q2, Q4). These would separate failures of inference from ambiguity in the history, and baseline error from response error. Two results address both for the drop in the paper studies. Where the history determines one effect per covariate (above), the drop stays large. And the two tasks of a Replay/Novel pair share the history and the target's course without the event, differing only in the future covariate values; their gap (30.3 to 51.1 points, bootstrap intervals above zero for every model; W3) therefore isolates the effect of unseen covariate values with the target's dynamics held fixed. The evaluated models are deliberately not given the family (the LLMs see placeholder variable names; Appendix D): each relationship must be inferred from the history, as in practice.
The drop thus reflects a failure to apply the relationships at unseen values, rather than a failure of ordinary forecasting. We are happy to add these results to the final version of the paper.