- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces GIFT-Eval-CTX, a benchmark for generating numerical forecasts from structured information about future behavior. It contains 10,000 instances drawn from 20 source datasets, with macro-level and localized contextual effects derived from the realized future. The evaluation compares time-series foundation models, direct language-model forecasting, and a tool-using agent, and analyzes performance across effect types and forecast horizons.
Critical strengths
-
The paper defines a useful and clearly scoped evaluation problem: translating supplied contextual information into numerical forecasts. Its structured representation separates broad future behavior from localized changes and provides a common interface across diverse datasets. This makes the target capability easier to evaluate consistently.
-
The empirical analysis goes beyond aggregate performance to examine where contextual forecasting succeeds and fails. The paper distinguishes the contributions of macro and micro effects, compares different event types, and identifies output-length failures. These analyses help explain why context and tool use benefit some forecasting settings more than others.
Critical weaknesses
-
The curation process remains unclear or uncertified to my understanding. The reliability of the contextual annotations is not established with sufficiently concrete evidence. The construction process includes automated validation and mentions human inspection, but the extent and outcomes of that inspection are unclear. Because these annotations define the task, uncertainty about their accuracy directly affects confidence in the benchmark.
-
The evaluation step will need substantial improvement. The comparisons do not yet establish how demanding the benchmark is or fully explain the agent’s advantage. The supplied context includes detailed information about the realized future, while the evaluated methods differ in their ability to generate and manipulate numerical outputs. Comparisons with simple constraint-based methods and methods given equivalent tools would help distinguish contextual reasoning from direct constraint application and output-format advantages.
Questions for authors
-
How reliable are the contextual annotations that define the benchmark? How many examples received human review, how were they sampled, and what error rates were observed? Please distinguish errors in qualitative effect descriptions from errors in numerical attributes, temporal boundaries, and consistency between macro and micro annotations.
-
What level of forecasting capability is required beyond directly applying the supplied constraints? Specifically, how would a simple method that adjusts a baseline forecast using the provided ranges, event intervals, and intensities perform? A context-only comparison would also help establish how much the historical observations contribute beyond the information already supplied about the future.
-
How does the difficulty-based selection procedure affect the benchmark’s conclusions? Which forecasting models were used to retain examples above the error threshold, and do these overlap with the evaluated baselines? How sensitive are method rankings and the measured benefit of context to the threshold or to evaluation on an unfiltered sample?
-
Which parts of MoiraiAgent account for its advantage over direct language-model forecasting? Are there any comparisons in which the same language model receives code execution but no Moirai forecasting tool? This would help separate the contribution of the forecasting backbone from the benefits of numerical computation and reliable generation of the required number of values.
-
How much do output-length failures and their handling influence the reported performance? What values are used to pad short forecasts, and how much of each method’s error is associated with padded outputs? Given the high failure rates at longer horizons, do the main conclusions remain consistent under alternative reasonable handling rules or a breakdown separating compliant and noncompliant outputs?
4 comments
Responses (2/n)
[Q2] What level of forecasting capability is required beyond directly applying the supplied constraints? Specifically, how would a simple method that adjusts a baseline forecast using the provided ranges, event intervals, and intensities perform? A context-only comparison would also help establish how much the historical observations contribute beyond the information already supplied about the future.
[A2] The numerical attributes in GIFT-Eval-CTX are part of the supplied future context, analogous to an expert specifying a future level, peak demand, or seasonal amplitude. Except for up/down impulse effects, the other effects only constrain the forecast without uniquely determining it. For example, Macro level_range specifies the future minimum and maximum, but the model must predict where they occur from the series’ temporal patterns. For a sustained upward or downward excursion, intensity gives the interval maximum or minimum, while the extremum’s location and the trajectory toward and away from it must be inferred from local context and historical patterns. For seasonal amplitude changes spanning several cycles, intensity provides a representative cycle amplitude (the maximum or median), leaving the model to determine individual cycle amplitudes and waveform shapes from the local periodic structure and history. Therefore, accurate forecasting requires combining the supplied effects with the series’ temporal dynamics, beyond merely matching numerical targets. This is a deliberate design choice in our schema, as we do not want the benchmark to become an exercise in numerical computation.
To address your concern more directly, we provide an additional result: Moirai-2 with deterministic rules achieves an overall MASE of 0.6084. This Moirai-2 + Rules baseline applies the following three deterministic rules: 1. Range clipping: Forecast values are clipped to the range specified by Macro level_range. 2. Mean-shift adjustment: For a mean shift, the forecast is ramped over the labeled interval toward the settled level derived from intensity, then maintained at that level for the rest of the horizon. 3. Excursion correction: For an impulse or sustained excursion, an upward or downward bump is added. The bump is zero at both interval boundaries and scaled so that the interval maximum or minimum matches the level derived from intensity. Existing values within the interval beyond the target peak or trough are clipped to that target. Other effects were not used because they could not uniquely determine the numerical magnitude or event duration that needed to be applied.
Comparing the Moirai-2 + Rules result with those of MoiraiAgent reveals a clear performance gap (0.6084 vs. 0.497 overall MASE). This gap shows that simply modifying a TSFM forecast according to numerical constraints is not sufficient to match MoiraiAgent’s performance.
Regarding your suggested context-only comparison, our early experiments compared History + Effects with an Effects-only setting in which the last observed value was provided as a scale anchor. Effects-only forecasting typically incurred roughly three times the MASE of History + Effects on these early subsets. These results indicate that historical observations contribute substantially.
Responses (3/n)
[Q3] How does the difficulty-based selection procedure affect the benchmark’s conclusions? Which forecasting models were used to retain examples above the error threshold, and do these overlap with the evaluated baselines? How sensitive are method rankings and the measured benefit of context to the threshold or to evaluation on an unfiltered sample?
[A3] Difficulty-based selection focuses the benchmark on windows where history-only TSFMs perform poorly and contextual forecasting has room to improve. Including many windows that these models already forecast accurately would leave little headroom for measuring the contribution of future context.
The filter uses Chronos-2 and Toto-2 and retains a window only when both models have a range-normalized MAE (MAE divided by the future value range) of at least 0.10. These two models are also evaluated as baselines. However, the ranking of the TSFMs is not the central focus of this work.
Regarding threshold sensitivity, we reaggregate the existing forecasts on subsets of the benchmark satisfying stricter thresholds of 0.15 and 0.20. The original benchmark contains 10,000 instances across 78 groups at a threshold of 0.10. Raising the threshold to 0.15 leaves 5,749 instances across 70 groups, while raising it to 0.20 leaves 3,499 instances across 65 groups. When evaluated on the same 65 group configurations, MoiraiAgent and the two LLMs with effects retain the same relative ranking across all three thresholds.
[Q4] Which parts of MoiraiAgent account for its advantage over direct language-model forecasting? Are there any comparisons in which the same language model receives code execution but no Moirai forecasting tool? This would help separate the contribution of the forecasting backbone from the benefits of numerical computation and reliable generation of the required number of values.
[A4] We evaluate two additional configurations using GPT-5.6 Sol in the history + structured effects setting. In the first, we replace Moirai-2 with TimesFM-3. In the second, we enable code execution without a forecasting tool. The results are shown below:
| Configuration | Short | Medium | Long | Overall |
|---|---|---|---|---|
| GPT-5.6-Sol | 0.4172 | 0.7270 | 0.7819 | 0.5651 |
| MoiraiAgent (Code + Moirai-2) | 0.3982 | 0.6233 | 0.6027 | 0.4968 |
| MoiraiAgent (Code + TimesFM-3) | 0.4020 | 0.6149 | 0.5786 | 0.4921 |
| MoiraiAgent (Code Only) | 0.4032 | 0.5863 | 0.5788 | 0.4869 |
The results show that replacing Moirai-2 with TimesFM-3 yields only a minor overall improvement. The code-only MoiraiAgent slightly outperforms both TSFM-assisted variants and achieves the best overall result. These results indicate that the TSFM-assisted forecasts do not improve on those generated in the code-only setting. We speculate that this is because the TSFMs cannot take contextual information as input, whereas the LLM incorporates the context when generating forecasts through code. These results further motivate developing TSFMs that directly accept contextual information alongside history, allowing future guidance to shape numerical generation within the forecasting model.
Responses (4/n)
[Q5] How much do output-length failures and their handling influence the reported performance? What values are used to pad short forecasts, and how much of each method’s error is associated with padded outputs? Given the high failure rates at longer horizons, do the main conclusions remain consistent under alternative reasonable handling rules or a breakdown separating compliant and noncompliant outputs?
[A5] Short LLM forecasts are padded by repeating the last value generated by the model.
We split each forecast's MASE into contributions from the values generated by the LLM and the values added by padding, then average each contribution across all instances. Under History + Effects mode, padded values account for 0.07% of GPT-5.6 Sol's average MASE and 2.91% of Claude Opus 4.8's. The corresponding shares under History-only mode are 0.02% and 0.31%. Therefore, most forecasting error comes from the model-generated values, with padding contributing only a small fraction.
Under History + Effects mode, we compare each LLM with MoiraiAgent on exactly the same instances where the LLM returns the correct number of values. For GPT-5.6 Sol, this subset contains 7,911 instances: aggregate relative MASE is 0.559 for GPT-5.6 Sol and 0.501 for MoiraiAgent. Claude Opus 4.8 has a different subset of 6,610 instances, with scores of 0.646 for Claude and 0.445 for MoiraiAgent. MoiraiAgent also outperforms each LLM on the instances where that LLM's output is too long or too short, so its advantage holds in both length-compliant and noncompliant groups.
Finally, we replace the padded suffixes using two alternative rules: repeating the last observed historical value, or using the seasonal-naive forecast at the missing positions. Across these alternatives, each LLM's aggregate relative MASE changes by at most 0.0044. Both LLMs continue to benefit from effects, and MoiraiAgent remains the best method overall.
- We thank the reviewer for the valuable feedback, which has helped us substantially improve the paper. We will incorporate the suggestions and the additional results discussed in our responses into the revised manuscript.
Responses (1/n)
We thank the reviewer for recognizing the clear scope of our evaluation task, the common structured-effects interface across datasets, and the detailed analysis of effect contributions and forecasting failures. We address your questions below.
[Q1] How reliable are the contextual annotations that define the benchmark? How many examples received human review, how were they sampled, and what error rates were observed? Please distinguish errors in qualitative effect descriptions from errors in numerical attributes, temporal boundaries, and consistency between macro and micro annotations.
[A1] Before scaling the annotation pipeline to the full benchmark, we constructed a development subset of approximately 1,000 instances sampled across dataset–frequency–term configurations to cover diverse forecasting settings. We manually reviewed the annotations on this subset during pipeline development and assessed their quality before proceeding to benchmark-scale annotation.
For the full benchmark, the Joint Critic Agent also serves as an LLM-as-a-judge, assigning each instance a score from 0 to 1 for its Macro and Micro annotations. The rubric considers semantic correctness, completeness, boundary accuracy, and evidence supporting direction, subtype, and intensity. Scores of 0.90–1.00 indicate exceptionally well-supported labels, 0.75–0.89 indicate reliable labels with minor ambiguity, and 0.50–0.74 indicate usable labels with uncertainty or omissions. Across all retained benchmark instances, the mean score is 0.8647 and the median is 0.86. A total of 98.3% score at least 0.75. The remaining 1.7% score between 0.61 and 0.74. No instance scores below 0.50.
We manually reviewed instances with lower scores and found that these cases admit multiple reasonable interpretations, even among human annotators. For example, a temporary rise in an irregular, noisy series may be interpreted as an upward excursion by one annotator and as ordinary background variation by another. Moreover, we find that a relatively low instance-level score does not necessarily imply that the retained annotations are incorrect. It may also reflect incomplete coverage when the agent leaves ambiguous regions unannotated. We consider this conservative strategy reasonable, since omitting an uncertain effect is preferable to supplying incorrect future guidance. We therefore did not discard effects solely because they received lower judge scores. However, we discarded instances with empty Micro annotations, as they did not provide sufficient actionable information about the future.