acceptodds
Under review as a conference paper at ICLR 2027

GIFT-Eval-CTX: A Benchmark For Contextual Forecasting Model Evaluation

Abstract

Contextual forecasting seeks to improve time series forecasting by incorporating forward-looking information, such as planned events, external conditions, and expert expectations. However, reliable evaluation remains difficult: existing benchmarks often have limited domain coverage, lack standardized history–future splits, or provide text that is not guaranteed to inform the future. We introduce GIFT-Eval-CTX, a benchmark that isolates whether forecasting methods can translate verified future information into accurate numerical trajectories. Built upon GIFT-Eval, GIFT-Eval-CTX contains 10,000 instances across 20 datasets. Each instance provides structured effects derived from the realized future: Macro effects describe horizon-wide behavior, while Micro effects specify localized changes. We construct these annotations with an agentic pipeline combining LLM-based discovery and deterministic code validation. We benchmark TSFMs, frontier LLMs, and tool-using agentic systems. The results show that structured effects reduce aggregate relative MASE by 40.8% for GPT-5.6 Sol and 30.1% for Claude Opus 4.8 relative to history-only prompting. The evaluated MoiraiAgent system achieves the best overall result. We further analyze the contributions of structured effects and agentic systems to performance, and identify opportunities for future research.

Primary area:
datasets and benchmarks
Keywords:
Time Series, Contextual Forecasting, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_17WBOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces GIFT-Eval-CTX, a benchmark for generating numerical forecasts from structured information about future behavior. It contains 10,000 instances drawn from 20 source datasets, with macro-level and localized contextual effects derived from the realized future. The evaluation compares time-series foundation models, direct language-model forecasting, and a tool-using agent, and analyzes performance across effect types and forecast horizons.

Critical strengths
  • The paper defines a useful and clearly scoped evaluation problem: translating supplied contextual information into numerical forecasts. Its structured representation separates broad future behavior from localized changes and provides a common interface across diverse datasets. This makes the target capability easier to evaluate consistently.

  • The empirical analysis goes beyond aggregate performance to examine where contextual forecasting succeeds and fails. The paper distinguishes the contributions of macro and micro effects, compares different event types, and identifies output-length failures. These analyses help explain why context and tool use benefit some forecasting settings more than others.

Critical weaknesses
  • The curation process remains unclear or uncertified to my understanding. The reliability of the contextual annotations is not established with sufficiently concrete evidence. The construction process includes automated validation and mentions human inspection, but the extent and outcomes of that inspection are unclear. Because these annotations define the task, uncertainty about their accuracy directly affects confidence in the benchmark.

  • The evaluation step will need substantial improvement. The comparisons do not yet establish how demanding the benchmark is or fully explain the agent’s advantage. The supplied context includes detailed information about the realized future, while the evaluated methods differ in their ability to generate and manipulate numerical outputs. Comparisons with simple constraint-based methods and methods given equivalent tools would help distinguish contextual reasoning from direct constraint application and output-format advantages.

Questions for authors
  1. How reliable are the contextual annotations that define the benchmark? How many examples received human review, how were they sampled, and what error rates were observed? Please distinguish errors in qualitative effect descriptions from errors in numerical attributes, temporal boundaries, and consistency between macro and micro annotations.

  2. What level of forecasting capability is required beyond directly applying the supplied constraints? Specifically, how would a simple method that adjusts a baseline forecast using the provided ranges, event intervals, and intensities perform? A context-only comparison would also help establish how much the historical observations contribute beyond the information already supplied about the future.

  3. How does the difficulty-based selection procedure affect the benchmark’s conclusions? Which forecasting models were used to retain examples above the error threshold, and do these overlap with the evaluated baselines? How sensitive are method rankings and the measured benefit of context to the threshold or to evaluation on an unfiltered sample?

  4. Which parts of MoiraiAgent account for its advantage over direct language-model forecasting? Are there any comparisons in which the same language model receives code execution but no Moirai forecasting tool? This would help separate the contribution of the forecasting backbone from the benefits of numerical computation and reliable generation of the required number of values.

  5. How much do output-length failures and their handling influence the reported performance? What values are used to pad short forecasts, and how much of each method’s error is associated with padded outputs? Given the high failure rates at longer horizons, do the main conclusions remain consistent under alternative reasonable handling rules or a breakdown separating compliant and noncompliant outputs?

4 comments
AuthorsOpenReview ↗

Responses (1/n)

We thank the reviewer for recognizing the clear scope of our evaluation task, the common structured-effects interface across datasets, and the detailed analysis of effect contributions and forecasting failures. We address your questions below.

[Q1] How reliable are the contextual annotations that define the benchmark? How many examples received human review, how were they sampled, and what error rates were observed? Please distinguish errors in qualitative effect descriptions from errors in numerical attributes, temporal boundaries, and consistency between macro and micro annotations.

[A1] Before scaling the annotation pipeline to the full benchmark, we constructed a development subset of approximately 1,000 instances sampled across dataset–frequency–term configurations to cover diverse forecasting settings. We manually reviewed the annotations on this subset during pipeline development and assessed their quality before proceeding to benchmark-scale annotation.

For the full benchmark, the Joint Critic Agent also serves as an LLM-as-a-judge, assigning each instance a score from 0 to 1 for its Macro and Micro annotations. The rubric considers semantic correctness, completeness, boundary accuracy, and evidence supporting direction, subtype, and intensity. Scores of 0.90–1.00 indicate exceptionally well-supported labels, 0.75–0.89 indicate reliable labels with minor ambiguity, and 0.50–0.74 indicate usable labels with uncertainty or omissions. Across all retained benchmark instances, the mean score is 0.8647 and the median is 0.86. A total of 98.3% score at least 0.75. The remaining 1.7% score between 0.61 and 0.74. No instance scores below 0.50.

We manually reviewed instances with lower scores and found that these cases admit multiple reasonable interpretations, even among human annotators. For example, a temporary rise in an irregular, noisy series may be interpreted as an upward excursion by one annotator and as ordinary background variation by another. Moreover, we find that a relatively low instance-level score does not necessarily imply that the retained annotations are incorrect. It may also reflect incomplete coverage when the agent leaves ambiguous regions unannotated. We consider this conservative strategy reasonable, since omitting an uncertain effect is preferable to supplying incorrect future guidance. We therefore did not discard effects solely because they received lower judge scores. However, we discarded instances with empty Micro annotations, as they did not provide sufficient actionable information about the future.

AuthorsOpenReview ↗

Responses (2/n)

[Q2] What level of forecasting capability is required beyond directly applying the supplied constraints? Specifically, how would a simple method that adjusts a baseline forecast using the provided ranges, event intervals, and intensities perform? A context-only comparison would also help establish how much the historical observations contribute beyond the information already supplied about the future.

[A2] The numerical attributes in GIFT-Eval-CTX are part of the supplied future context, analogous to an expert specifying a future level, peak demand, or seasonal amplitude. Except for up/down impulse effects, the other effects only constrain the forecast without uniquely determining it. For example, Macro level_range specifies the future minimum and maximum, but the model must predict where they occur from the series’ temporal patterns. For a sustained upward or downward excursion, intensity gives the interval maximum or minimum, while the extremum’s location and the trajectory toward and away from it must be inferred from local context and historical patterns. For seasonal amplitude changes spanning several cycles, intensity provides a representative cycle amplitude (the maximum or median), leaving the model to determine individual cycle amplitudes and waveform shapes from the local periodic structure and history. Therefore, accurate forecasting requires combining the supplied effects with the series’ temporal dynamics, beyond merely matching numerical targets. This is a deliberate design choice in our schema, as we do not want the benchmark to become an exercise in numerical computation.

To address your concern more directly, we provide an additional result: Moirai-2 with deterministic rules achieves an overall MASE of 0.6084. This Moirai-2 + Rules baseline applies the following three deterministic rules: 1. Range clipping: Forecast values are clipped to the range specified by Macro level_range. 2. Mean-shift adjustment: For a mean shift, the forecast is ramped over the labeled interval toward the settled level derived from intensity, then maintained at that level for the rest of the horizon. 3. Excursion correction: For an impulse or sustained excursion, an upward or downward bump is added. The bump is zero at both interval boundaries and scaled so that the interval maximum or minimum matches the level derived from intensity. Existing values within the interval beyond the target peak or trough are clipped to that target. Other effects were not used because they could not uniquely determine the numerical magnitude or event duration that needed to be applied.

Comparing the Moirai-2 + Rules result with those of MoiraiAgent reveals a clear performance gap (0.6084 vs. 0.497 overall MASE). This gap shows that simply modifying a TSFM forecast according to numerical constraints is not sufficient to match MoiraiAgent’s performance.

Regarding your suggested context-only comparison, our early experiments compared History + Effects with an Effects-only setting in which the last observed value was provided as a scale anchor. Effects-only forecasting typically incurred roughly three times the MASE of History + Effects on these early subsets. These results indicate that historical observations contribute substantially.

AuthorsOpenReview ↗

Responses (3/n)

[Q3] How does the difficulty-based selection procedure affect the benchmark’s conclusions? Which forecasting models were used to retain examples above the error threshold, and do these overlap with the evaluated baselines? How sensitive are method rankings and the measured benefit of context to the threshold or to evaluation on an unfiltered sample?

[A3] Difficulty-based selection focuses the benchmark on windows where history-only TSFMs perform poorly and contextual forecasting has room to improve. Including many windows that these models already forecast accurately would leave little headroom for measuring the contribution of future context.

The filter uses Chronos-2 and Toto-2 and retains a window only when both models have a range-normalized MAE (MAE divided by the future value range) of at least 0.10. These two models are also evaluated as baselines. However, the ranking of the TSFMs is not the central focus of this work.

Regarding threshold sensitivity, we reaggregate the existing forecasts on subsets of the benchmark satisfying stricter thresholds of 0.15 and 0.20. The original benchmark contains 10,000 instances across 78 groups at a threshold of 0.10. Raising the threshold to 0.15 leaves 5,749 instances across 70 groups, while raising it to 0.20 leaves 3,499 instances across 65 groups. When evaluated on the same 65 group configurations, MoiraiAgent and the two LLMs with effects retain the same relative ranking across all three thresholds.

[Q4] Which parts of MoiraiAgent account for its advantage over direct language-model forecasting? Are there any comparisons in which the same language model receives code execution but no Moirai forecasting tool? This would help separate the contribution of the forecasting backbone from the benefits of numerical computation and reliable generation of the required number of values.

[A4] We evaluate two additional configurations using GPT-5.6 Sol in the history + structured effects setting. In the first, we replace Moirai-2 with TimesFM-3. In the second, we enable code execution without a forecasting tool. The results are shown below:

ConfigurationShortMediumLongOverall
GPT-5.6-Sol0.41720.72700.78190.5651
MoiraiAgent (Code + Moirai-2)0.39820.62330.60270.4968
MoiraiAgent (Code + TimesFM-3)0.40200.61490.57860.4921
MoiraiAgent (Code Only)0.40320.58630.57880.4869

The results show that replacing Moirai-2 with TimesFM-3 yields only a minor overall improvement. The code-only MoiraiAgent slightly outperforms both TSFM-assisted variants and achieves the best overall result. These results indicate that the TSFM-assisted forecasts do not improve on those generated in the code-only setting. We speculate that this is because the TSFMs cannot take contextual information as input, whereas the LLM incorporates the context when generating forecasts through code. These results further motivate developing TSFMs that directly accept contextual information alongside history, allowing future guidance to shape numerical generation within the forecasting model.

AuthorsOpenReview ↗

Responses (4/n)

[Q5] How much do output-length failures and their handling influence the reported performance? What values are used to pad short forecasts, and how much of each method’s error is associated with padded outputs? Given the high failure rates at longer horizons, do the main conclusions remain consistent under alternative reasonable handling rules or a breakdown separating compliant and noncompliant outputs?

[A5] Short LLM forecasts are padded by repeating the last value generated by the model.

We split each forecast's MASE into contributions from the values generated by the LLM and the values added by padding, then average each contribution across all instances. Under History + Effects mode, padded values account for 0.07% of GPT-5.6 Sol's average MASE and 2.91% of Claude Opus 4.8's. The corresponding shares under History-only mode are 0.02% and 0.31%. Therefore, most forecasting error comes from the model-generated values, with padding contributing only a small fraction.

Under History + Effects mode, we compare each LLM with MoiraiAgent on exactly the same instances where the LLM returns the correct number of values. For GPT-5.6 Sol, this subset contains 7,911 instances: aggregate relative MASE is 0.559 for GPT-5.6 Sol and 0.501 for MoiraiAgent. Claude Opus 4.8 has a different subset of 6,610 instances, with scores of 0.646 for Claude and 0.445 for MoiraiAgent. MoiraiAgent also outperforms each LLM on the instances where that LLM's output is too long or too short, so its advantage holds in both length-compliant and noncompliant groups.

Finally, we replace the padded suffixes using two alternative rules: repeating the last observed historical value, or using the seasonal-naive forecast at the missing positions. Across these alternatives, each LLM's aggregate relative MASE changes by at most 0.0044. Both LLMs continue to benefit from effects, and MoiraiAgent remains the best method overall.

- We thank the reviewer for the valuable feedback, which has helped us substantially improve the paper. We will incorporate the suggestions and the additional results discussed in our responses into the revised manuscript.

Reviewer_yjR1OpenReview ↗
Rating
2
Confidence
2(−2)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GIFT-Eval-CTX, a benchmark designed to evaluate how forecasting models use structured future information to generate numerical time series predictions. Built upon GIFT-Eval, the benchmark contains 10,000 instances across 20 datasets, with annotations derived from the ground-truth future. These annotations are organized into Macro effects describing global trends, seasonality, and value ranges, and Micro effects capturing localized changes such as excursions, mean shifts, and seasonal variations. The authors develop a multi-agent annotation pipeline with deterministic validation to generate these structured effects and evaluate several time series foundation models, frontier LLMs, and an agentic forecasting system. The experiments compare history-only and context-aware forecasting performance, with additional analyses examining the contributions of different effect types and the role of tool use in generating numerical forecasts.

Critical strengths
  1. In general, I like the effort put into constructing a large-scale benchmark with 10,000 instances across 20 time series datasets. The standardized evaluation setting and the coverage of different forecasting horizons provide a useful foundation for studying how models incorporate future contextual information.

  2. The paper includes a fairly detailed experimental analysis of different forecasting approaches, including multiple time series baselines and agentic settings. I particularly like the ablation studies on Macro and Micro effects, which help illustrate how different types of structured context contribute to forecasting performance.

Critical weaknesses
  1. For this paper, I found the novelty of this benchmark rather limited, as contextual forecasting and the use of external information for time series prediction have already been extensively explored in prior work. The main difference here seems to be converting future observations into structured effects, but I am not convinced that this introduces a sufficiently new or meaningful evaluation setting. The paper also does not clearly establish what unique capabilities this benchmark can assess beyond existing alternatives.

  2. Besides, I have serious concerns about the benchmark design and the experimental evaluation. Since the structured effects are derived directly from the ground-truth future, including exact value ranges, event boundaries, and intensities, the task seems closer to constrained numerical reconstruction than realistic contextual forecasting. The reported improvements are substantial, but without stronger baselines and more careful analysis, I found it difficult to tell whether the benchmark genuinely evaluates contextual forecasting capabilities or simply rewards models for following explicit numerical constraints.

Questions for authors
  1. Could you clarify how GIFT-Eval-CTX fundamentally differs from existing contextual forecasting benchmarks such as Context is Key and TemporalBench?

  2. How much of the reported improvement comes from providing exact numerical information extracted from the ground-truth future, rather than from the models' contextual understanding?

  3. Why are there no simpler context-aware baselines that directly modify TSFM predictions using the provided level ranges and intensities?

  4. The annotation pipeline involves several LLM agents and deterministic validators, but I could not find a sufficiently detailed quantitative evaluation of annotation quality.

3 comments
AuthorsOpenReview ↗

Responses (1/n)

We thank the reviewer for recognizing the scale of our benchmark, its standardized evaluation across forecast horizons, and the value of the Macro/Micro ablation studies. We address your concerns below.

[Q1] Could you clarify how GIFT-Eval-CTX fundamentally differs from existing contextual forecasting benchmarks such as Context is Key and TemporalBench?

[A1] As we enter the agentic era, we believe contextual forecasting should also evolve toward agentic systems. In the system we envision, the task is decomposed into two subtasks: contextual understanding and numerical realization. We expect frontier LLMs to become increasingly capable of interpreting user-provided time series, text, and images and translating relevant information into actionable guidance for forecasting. Then a crucial question is raised: are current forecasting systems ready to turn that guidance into accurate numerical trajectories? GIFT-Eval-CTX is developed to answer this question, and our evaluation results show that, although the current agentic systems achieve some good results, there is still much room for improvement.

Returning to the existing benchmarks you mentioned (e.g., Context is Key and TemporalBench), their contextual forecasting settings require models to both interpret natural-language context and translate its implications into numerical forecasts. An inaccurate forecast may therefore reflect either a misunderstanding of the context or a failure to realize the understood information numerically, making these capabilities difficult to assess separately. GIFT-Eval-CTX provides verified future information through a structured-effects interface to isolate the downstream capability of numerical realization. This is the fundamental difference.

Moreover, GIFT-Eval-CTX contains 10,000 forecasting instances, compared with 355 instances in the published Context is Key evaluation and 191 distinct time series instances in TemporalBench. This larger evaluation set enables systematic comparisons across datasets, forecast horizons, and effect families. It also supports separate analyses inside and outside effect intervals, showing how well methods realize local events and connect them into coherent forecasts.

AuthorsOpenReview ↗

Responses (2/n)

[Q2] How much of the reported improvement comes from providing exact numerical information extracted from the ground-truth future, rather than from the models' contextual understanding?

[Q3] Why are there no simpler context-aware baselines that directly modify TSFM predictions using the provided level ranges and intensities?

[A2 & A3] The numerical attributes in GIFT-Eval-CTX are part of the supplied future context, analogous to an expert specifying a future level, peak demand, or seasonal amplitude. Except for up/down impulse effects, the other effects only constrain the forecast without uniquely determining it. For example, Macro level_range specifies the future minimum and maximum, but the model must predict where they occur from the series’ temporal patterns. For a sustained upward or downward excursion, intensity gives the interval maximum or minimum, while the extremum’s location and the trajectory toward and away from it must be inferred from local context and historical patterns. For seasonal amplitude changes spanning several cycles, intensity provides a representative cycle amplitude (the maximum or median), leaving the model to determine individual cycle amplitudes and waveform shapes from the local periodic structure and history. Therefore, accurate forecasting requires combining the supplied effects with the series’ temporal dynamics, beyond merely matching numerical targets. This is a deliberate design choice in our schema, as we do not want the benchmark to become an exercise in numerical computation.

To address your concern more directly, we provide an additional result: Moirai-2 with deterministic rules achieves an overall MASE of 0.6084. This Moirai-2 + Rules baseline applies the following three deterministic rules: 1. Range clipping: Forecast values are clipped to the range specified by Macro level_range. 2. Mean-shift adjustment: For a mean shift, the forecast is ramped over the labeled interval toward the settled level derived from intensity, then maintained at that level for the rest of the horizon. 3. Excursion correction: For an impulse or sustained excursion, an upward or downward bump is added. The bump is zero at both interval boundaries and scaled so that the interval maximum or minimum matches the level derived from intensity. Existing values within the interval beyond the target peak or trough are clipped to that target. Other effects were not used because they could not uniquely determine the numerical magnitude or event duration that needed to be applied.

Comparing the Moirai-2 + Rules result with those of MoiraiAgent reveals a clear performance gap (0.6084 vs. 0.497 overall MASE). This gap shows that simply modifying a TSFM forecast according to numerical constraints is not sufficient to match MoiraiAgent’s performance.

AuthorsOpenReview ↗

Responses (3/n)

[Q4] The annotation pipeline involves several LLM agents and deterministic validators, but I could not find a sufficiently detailed quantitative evaluation of annotation quality.

[A4] Before scaling the annotation pipeline to the full benchmark, we constructed a development subset of approximately 1,000 instances sampled across dataset–frequency–term configurations to cover diverse forecasting settings. We manually reviewed the annotations on this subset during pipeline development and assessed their quality before proceeding to benchmark-scale annotation.

For the full benchmark, the Joint Critic Agent also serves as an LLM-as-a-judge, assigning each instance a score from 0 to 1 for its Macro and Micro annotations. The rubric considers semantic correctness, completeness, boundary accuracy, and evidence supporting direction, subtype, and intensity. Scores of 0.90–1.00 indicate exceptionally well-supported labels, 0.75–0.89 indicate reliable labels with minor ambiguity, and 0.50–0.74 indicate usable labels with uncertainty or omissions. Across all retained benchmark instances, the mean score is 0.8647 and the median is 0.86. A total of 98.3% score at least 0.75. The remaining 1.7% score between 0.61 and 0.74. No instance scores below 0.50.

We manually reviewed instances with lower scores and found that these cases admit multiple reasonable interpretations, even among human annotators. For example, a temporary rise in an irregular, noisy series may be interpreted as an upward excursion by one annotator and as ordinary background variation by another. Moreover, we find that a relatively low instance-level score does not necessarily imply that the retained annotations are incorrect. It may also reflect incomplete coverage when the agent leaves ambiguous regions unannotated. We consider this conservative strategy reasonable, since omitting an uncertain effect is preferable to supplying incorrect future guidance. We therefore did not discard effects solely because they received lower judge scores. However, we discarded instances with empty Micro annotations, as they did not provide sufficient actionable information about the future.

- We thank the reviewer for the valuable feedback, which has helped us substantially improve the paper. We will incorporate the suggestions and the additional results discussed in our responses into the revised manuscript.

Reviewer_nRR7OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

GIFT-Eval-CTX proposes a benchmark for contextual time series forecasting that deliberately isolates numerical realization from raw text comprehension. The framework uses a domain-agnostic Structured Effects Schema - split into Macro and Micro effects - to encode verified future information. The authors built a 10,000-instance dataset across 20 domains via a multi-agent annotation pipeline backed by deterministic code validation. Crucially, the benchmark filters out instances where history-only Time Series Foundation Models (TSFMs) succeed, forcing models to rely on the provided context. Evaluations across TSFMs, frontier LLMs, and a proposed MoiraiAgent show that structured context significantly lifts LLM performance, with MoiraiAgent reaching state-of-the-art accuracy.

Critical strengths

Isolating numerical realization: Decoupling raw text comprehension from the numerical realization of future events is a highly valuable conceptual step. The Structured Effects Schema provides a clean, interpretable mechanism for injecting future context without language-style confounders.

Rigorous annotation pipeline: Backing the multi-agent annotation setup with deterministic code validation establishes a strong methodology for generating high-quality benchmark data. The qualitative analysis is also refreshingly transparent, specifically regarding MoiraiAgent's tendency to lose natural temporal dynamics.

Critical weaknesses

Missing LLM-specific baselines: Evaluating frontier LLMs purely via direct numerical generation is a known weak baseline prone to tokenization and exact-length failures. The authors skip established LLM-based forecasters specifically built to align text and context with time series data (e.g., Time-LLM, LLMTime, PromptCast, UniTime). Omitting these artificially inflates MoiraiAgent's relative novelty and performance.

Built-in structural advantage: The pipeline extracts Micro effects based on the residuals of a TSFM reference forecast that includes Moirai-2. Since MoiraiAgent uses Moirai-2 for its base trajectory, feeding it these specific Micro effects basically provides a localized error map for its own blind spots. This reduces the benchmark to a deterministic residual-correction exercise for this specific agent rather than a fair test of contextual forecasting.

Inadequate evaluation metrics: MoiraiAgent achieves a lower MASE by overwriting natural fluctuations with unnatural extended plateaus or perfectly straight lines via code edits. As a point-wise metric, MASE fails to penalize this severe loss of realistic temporal variance. The evaluation protocol needs distributional metrics like CRPS or spectral loss to capture this degradation.

Questions for authors

Why were established LLM-based forecasting architectures like Time-LLM, LLMTime, and PromptCast excluded from the baselines? How would they perform on GIFT-Eval-CTX when provided the structured effects schema?

Defining ground-truth Micro effects based on TSFM residuals hands MoiraiAgent a structural advantage. How do you address the concern that this essentially turns the benchmark into a deterministic residual-correction task tailored to this specific agent?

Figure 3 illustrates that MoiraiAgent flattens natural variance into plateaus. Why were distributional metrics (e.g., CRPS or spectral loss) excluded from the evaluation protocol, given that point-wise MASE fails to penalize this behavior?

1 comment
AuthorsOpenReview ↗

Responses (1/1)

We thank the reviewer for recognizing the value of separating contextual understanding from numerical realization and the rigor of the annotation pipeline. We address the three concerns below.

[Q1] Why were established LLM-based forecasting architectures like Time-LLM, LLMTime, and PromptCast excluded from the baselines? How would they perform on GIFT-Eval-CTX when provided the structured effects schema?

[A1] The suggested methods belong to an earlier generation of LLM-based forecasting. Time-LLM uses Llama-7B, UniTime uses GPT-2, and PromptCast evaluates earlier language-generation models such as T5 and BART. Given these older backbones and forecasting-focused training setups, we do not expect them to reliably interpret and follow the compositional structured effects used in our benchmark. Our choice of frontier LLMs is motivated by precisely this requirement: a model must understand the supplied effects before it can translate them into numerical forecasts.

To address your concern with a contemporary model, we additionally provide the performance of TimeBraid-6.7B (https://arxiv.org/abs/2609.29792), a recent unified time-series and language model built on a Qwen3 language backbone and trained for both understanding and forecasting. On GIFT-Eval-CTX, its aggregate relative MASE is 0.6847 on short horizons, 0.8720 on medium horizons, 0.8641 on long horizons, and 0.7733 overall. The overall performance slightly outperforms TimesFM-3 and remains substantially behind GPT-5.6 Sol and MoiraiAgent.

[Q2] Defining ground-truth Micro effects based on TSFM residuals hands MoiraiAgent a structural advantage. How do you address the concern that this essentially turns the benchmark into a deterministic residual-correction task tailored to this specific agent?

[A2] We would like to clarify that Moirai-2 participates in neither difficulty filtering nor the construction of the reference forecast. The annotation reference is the pointwise average of Chronos-2 and Toto-2. This is also kind of reflected in the TSFMs’ overall performance ranking on the benchmark: Chronos-2 and Toto-2 perform slightly worse than Moirai-2 and TimesFM-3. However, the ranking of the TSFMs is not the central focus of this work.

[Q3] Figure 3 illustrates that MoiraiAgent flattens natural variance into plateaus. Why were distributional metrics (e.g., CRPS or spectral loss) excluded from the evaluation protocol, given that point-wise MASE fails to penalize this behavior?

[A3] MASE is appropriate for the point-forecasting task evaluated in this paper. All methods return a single numerical trajectory for comparison, including the median predictions of the TSFMs. CRPS evaluates predictive distributions. We evaluate LLM and agentic methods based on API calls, estimating CRPS would require repeatedly generating K forecast trajectories for every instance. With 10,000 benchmark instances, evaluation would require 10,000K generations per method. For agents, each sample additionally incurs the associated LLM reasoning and tool-execution costs. The resulting API costs would be prohibitively high.

Moreover, LLM-based methods do not naturally support a distributional forecast unlike TSFMs’ quantile outputs. Therefore, running the LLM K time to get a forecast trajectory does not necessarily create a quantile distribution but just different predictions at the point forecast.

- We thank the reviewer for the valuable feedback, which has helped us substantially improve the paper. We will incorporate the suggestions and the additional results discussed in our responses into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.