acceptodds
Under review as a conference paper at ICLR 2027

LEAF: A Living Benchmark for Event-Augmented Forecasting

Abstract

Large Language Models (LLMs) are increasingly applied to real-world forecasting tasks, yet evaluating their true predictive capability remains compromised by pre-training data contamination and look-ahead leakage in automated search. Existing benchmarks either rely on static contexts, restrict evaluations to narrow environments, or fail to audit auxiliary textual events for future information leakage. To establish a rigorous evaluation paradigm, we propose LEAF, the first living benchmark for event-augmented forecasting tasks, including trend, event, and time series forecasting. LEAF couples a recursive retrieval agent system with dual-agent cross-validation to gather comprehensive, relevant, and temporally aligned auxiliary context. A comprehensive audit across 500 tasks by 47 domain specialists demonstrates that our pipeline suppresses future information leakage from 8.6% to 1.6%. Across extensive evaluations of 16 frontier proprietary and open-weight models on our benchmark, we show that LLMs effectively extract signals from verified events to boost trend and event forecasting. The code and data are provided here: https://anonymous.4open.science/r/LEAF-4565.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Context-aid Forecasting, Time Series Forecasting, Event Forecasting

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_HJFROpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces LEAF, a living benchmark for event-augmented forecasting. The three tasks are event forecasting on Polymarket questions, trend forecasting as 3-class or 5-class classification, and time series forecasting. The data cover large-cap and mid-cap stocks, commodities, electricity load, weather and Polymarket events, and the evaluation horizons in Table 2 run from 10.2025 to 09.2026. A retrieval agent powered by Gemini 3.1 Pro and augmented with Gemini Grounded Search collects the events. An event is kept only when two fact-checking agents, Gemini 3.1 Pro and Claude 4.6 Sonnet, agree on both its authenticity and its public release time.

The evidence comes in three parts. First, 47 domain experts audited 500 sampled tasks. The fact-checking module reduces look-ahead leakage from 43 to 8 of these tasks, or from 8.6% to 1.6%, and raises date accuracy from 305 to 405 tasks. Eight of the 500 contain factual fabrications. Second, the temporal ablation in Table 11 shows that TS-only accuracy on stock trend forecasting is much higher on a historical test period from January 2023 to June 2024 than on out-of-sample living data. Third, the authors evaluate 16 LLMs with and without events and compare their time series forecasts with TimesFM 3.0, Chronos-2, the Last Value Baseline and the Historical Mean Baseline.

The paper finds that events improve trend and event forecasting for most models, and that most LLMs perform on par with the TSFMs on hourly point forecasting, with Claude Opus 5 doing better. The Last Value Baseline has the lowest MAPE in Table 9, and the predictability levels assigned by LLMs correlate with stock trend forecasting accuracy. The authors also use sentiment-consistent text in Sec. 3.2 and Fig. 8 to validate that the retrieved events are relevant.

Critical strengths

The paper addresses a real problem. LLM forecasting evaluations suffer both from pre-training data contamination and from look-ahead leakage in automated search. LEAF targets both problems and pairs textual events with time series data. The evaluation is broad, covering 16 LLMs, two TSFMs and two naive baselines. Tables 12 and 15 to 18 report 95% confidence intervals, and the anonymous repository with code and data is accessible.

The human-expert audit is a contribution on its own. It shows that unverified automated search leaks future information in 43 of the 500 audited tasks, and that 8 leaks remain after dual-agent cross-validation. Tables 5 and 6 list these eight residual leaks and the eight hallucinations, with verbatim excerpts. Appendix C.3 finds that the generated causality labels reach only 17.3% expert acceptance, or 52 of 300 tasks, which is a clear warning against trusting rationales written by LLMs. Table 11 shows a large gap between the historical test period and the living data. For 3-class mid-cap stocks at the 6-month horizon, for example, Gemini 3.1 Pro reaches 92.36 on the historical period but only 46.02 on the living data. This gap is consistent with pre-training data contamination. These findings are useful for anyone who builds living benchmarks for event-augmented forecasting.

Critical weaknesses

Event forecasting is scored only on the winning outcome. Sec. 4.1 says the model is asked for "the predicted probability of the actual ground-truth outcome", and the caption of Fig. 13 says target_outcome "represents the ultimate winning outcome". So o_t = 1 for every scored forecast, and the Brier score reduces to (1 - f_t)^2. A constant forecast of 1 would get a perfect score. A constant forecast of 0.5 would score 0.25 and beat every model at the 100% time-to-resolution cut-off, with or without events, since Table 15 reports Brier scores from 0.398 to 0.598. Because the other potential outcomes are never queried, a higher probability on the target outcome after adding events could simply mean that the model now gives higher probabilities across the board. The prompt in Fig. 13 asks the model to weigh market pricing, yet no baseline that uses the Polymarket odds is reported.

The weather events already contain official forecasts. Items [54] and [57] in Fig. 19 and Listing 2 include NWS forecasts of daytime highs and overnight lows for the coming days, as well as CPC 6-10 and 8-14 day outlooks. The TSFMs receive none of this, and no baseline uses these forecasts directly. The gains in Tables 3, 4 and 16 to 18 therefore cannot be credited to the LLM's own reasoning.

The trend tasks have no baseline that always predicts the most frequent class, even though the label distribution is imbalanced. In Table 10, 43.69% of the ground truth for long-term mid-cap stocks is Strong Up and 29.40% is Strong Down. For electricity, App. D.1 reports 36.70% Strong Up and 17.32% Up. The trend prompts in Figs. 14 and 15 suggest that the 3-class Up label covers both Strong Up and Up. In that case, always predicting Up would reach about 53% to 54% on these label distributions, which is higher than many TS-only accuracies in Tables 12 and 16. The weather label uses the minimum temperature over the future N days, so the label distribution may also change with the horizon.

In stock price forecasting, the prompt differs between the TS and w/ Evts settings. Fig. 12 shows two analytical frameworks. The one used when event intelligence is present contains a Random Walk step that tells the model to default to a stable outlook. The one used when no events are available guides the model to rely on technical price action and momentum. The shared system prompt also says "Do NOT lazily repeat the last closing price". The comparison between TS and w/ Evts for stocks in Table 9 therefore mixes the effect of the events with the effect of the prompt. Sec. 4.3 and App. F.1 also conclude that market price forecasting is inherently infeasible. This conclusion does not follow from a setup that prohibits the model from repeating the last closing price, since that is exactly how the Last Value Baseline wins in Table 9.

The sentiment-consistent text in Sec. 3.2 is selected using the ground-truth change. It shows that models respond to events whose sentiment matches the actual future trajectory, but not that the retrieved events are informative without this selection.

Several claims go beyond the tables. The phrase "consistently outperform ... across all settings" in Sec. 4.2 conflicts with Table 13 and with two cases in Table 3, where Gemini 3.7 Flash drops from 71.34 to 70.73 on electricity at the 7-day horizon and Claude Opus 5 drops from 39.72 to 38.94 on large-cap stocks at the 1-day horizon. App. F.2.1 says the correlation between predictability levels and accuracy "vanishes when events are excluded", yet Table 14 reports TS-only correlations of 0.919, 0.945 and 0.829. The Introduction says that evaluating on data before training cutoffs inflates forecasting accuracy "by up to 147%". In Table 11, however, 147% is the mean of the 16 relative performance gains. The median is 98.6% and the maximum is 769.5%, so 147% is an average and not an upper bound.

Questions for authors

What is the documented training cutoff of each of the 16 models, and which date range is scored for each model and task?

Which Polymarket questions are scored? Table 2 lists 66 entities from 04.2026 to 09.2026, App. D.2.5 mentions 30 tasks, and the scripts in the repository list identifiers dated from 2025 to early 2026. Please define question, market, outcome and entity.

In event forecasting, is the model ever asked about potential outcomes other than the winning outcome?

How were the 500 audited samples drawn? Sec. 3.2 calls them events, Sec. 3.4 calls them tasks, and App. C calls them event-driven forecasting instances. Which unit was sampled, and from which entities? Also, what does RAM / Comm. in Table 7 refer to, given that App. C names this domain Hardware/Commodities and App. D.2.2 lists ten commodities in energy, metals and agriculture?

For the 1-day large-cap stock task, does the forecast start before or after the closing price of the cut-off date is known? The 3-day example in Fig. 18 includes items [76] to [80], which carry the same date as the cut-off.

Please explain the following mismatches.

The Gemma rows differ between Table 3 and Table 16. With TS only at the 7-day horizon, Gemma3-27B-it scores 45.73 on electricity in Table 3 and 54.9 in Table 16, and Gemma3-12B-it scores 42.75 on weather in Table 3 and 51.3 in Table 16.

For Claude 4.6 Opus on long-term mid-cap stocks at the 6-month horizon with events, Table 12 reports 28.6, while Table 14 reports between 40.88 and 57.58 at every predictability level.

Sec. 3.4 reports a 0% false negative rate, which does not fit the definitions in App. C.5.

Impact Type is reported as 179/201, but the two domains that evaluate this attribute in Table 7 have only 200 Evaluated Tasks.

App. C.1 gives an average of 13.6 years of experience, but the specialist counts and average years listed for each domain give 12.73.

What does DLV in Table 9 stand for?

App. E.2 computes the Overall score as the mean over the metrics available for each model, and some models have missing cells, such as Grok 4.20 non-reason in Table 17. Are the Overall scores in Fig. 4 comparable across models evaluated on different subsets of metrics?

2 comments

Thank you for your detailed comments, many of which point out writing and clarification issues that we will correct in the revised paper; below, we focus on addressing your primary concerns:

W1: Event forecasting is scored only on the winning outcome.

We expanded our evaluation to predict all candidate outcomes across every question, not just the eventual winners; under this setting, adding text events (w/ Event) still consistently improves LLM performance. Note that while our initial scoring focused on winning outcomes, our event retrieval relied solely on the questions, ensuring zero outcome bias. We will incorporate the detailed results (including each cut-off) and prompts into the revised paper:

ModelMean Brier Score (w/o Evts)Mean Brier Score (w/ Evts)
Claude 4.6 Opus0.1810.129
Claude 4.6 Sonnet0.1740.123
Claude 4.8 Opus0.1710.128
Claude 5 Opus0.1690.112
Claude 5 Sonnet0.1770.119
Gemini 3.1 Pro0.1820.123
Gemini 3.1 Flash0.1880.143
Gemini 3.5 Flash0.1890.135
Gemini 3.7 Flash0.1750.115
Gemini 4 Argon0.1690.092
Grok 4.200.1750.125
Grok 4.20 (non-reasoning)0.1960.153
Llama 4 Maverick0.1800.128

Following the same experimental setting described above, we also collected every resolved question in our target domains since October 2025, expanding the event forecasting benchmark to 1,199 questions. On this dataset, our preliminary findings show that models can outperform the market baseline (Brier Score of market odds is 0.159) when integrating both market odds and text events.

Model / Baselinew/o Evts, w/o Oddsw/ Evts, w/o Oddsw/o Evts, w/ Oddsw/ Evts, w/ Odds
Claude 5 Opus0.2540.1710.1610.147
Gemini 3.8 Flash0.3220.1760.1630.150

W2: The weather events already contain official forecasts

Thank you for the comment. As a benchmark, our goal is to evaluate the foundational forecasting capabilities of LLMs; comprehending textual weather forecasts and integrating them to improve predictions directly reflects this capability. Furthermore, we do not claim to propose a SOTA forecasting method, our core claim is simply that textual event context is beneficial for forecasting.

W3: The prompt differs between the TS and w/ Evts settings in Stock price forecasting

Yes, the Stock Price Forecasting task (Appendix F.2.1) is the sole exception where the prompts differ, as its primary purpose is to demonstrate the infeasibility of predicting exact stock prices. Even after testing multiple prompt variations and selecting the highest-performing one, the model still fell significantly short of the simplest baseline. In contrast, across all main forecasting experiments in the paper, the w/ Events and w/o Events settings use completely identical prompts. We will make this distinction clear in the revised manuscript.

W4: The sentiment-consistent text does not reflect that events are informative

Thank you for the comment. Event validation is intended to verify relevance: our ablation shows that selecting sentiment-consistent texts from other targets yields no strong correlation between accuracy and the proportion of relevant texts. Meanwhile, our main forecasting experiments directly prove that retrieved events are informative for prediction. We will clarify this objective in the revision.

W5: No baseline that always predicts the most frequent class

Thank you for the suggestion. However, this does not serve as a valid baseline because all models only have access to information prior to the cut-off date, and forcing a model to always predict the most frequent test label leaks future information, making it an unfair comparison.

Q1: Supplementary details on Polymarket questions, model cutoffs, and audited samples

We will add these details in the revision. Most models have knowledge cutoffs prior to the test periods. While a few overlap with the weather and electricity data, these datasets are fetched via specialized APIs and are unlikely to be in pre-training corpora—as evidenced by the poor w/o Event results, which trail simple baselines. For the manual audit, we randomly sampled an equal number of events per domain.

Q2: Clarification on stock cut-off date, Gemma3-27B-it results (Tables 3 & 16), and Claude results (Tables 12 & 14)

Yes, stock tasks include all data up to and on the cut-off date. For Gemma3-27B-it, we updated the re-run results in Table 3 but missed Table 16, which we will sync in the revision. Tables 12 and 14 are not directly comparable as they use different data batches: since LEAF is a living benchmark, we only update main results with new data rather than re-running prior validation experiments (e.g., Table 14 and Event Validation, which still strictly post-date model cut-offs). We will clarify this in the revision.

Dear Reviewer HJFR,

We truly appreciate the time and constructive insights you dedicated to reviewing our submission. With the author–reviewer discussion period wrapping up soon, we wanted to check in and see if our rebuttal addresses your questions, or if there are any open points you would like us to elaborate on before the window closes.

Best Regards,

Authors

Reviewer_yruuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

At present, the ability of LLMs in real-world forecasting tasks usually lacks a reliable benchmark for evaluating forecasting models’ ability to predict the future. The reason usually lies in two types of information contamination in existing Benchmarks: first, during pretraining, models may be exposed in advance to the events to be predicted and their structure; second, even with dynamic benchmarks, models may learn the forecasting information in advance while searching historical news. Therefore, the authors propose the LEAF Benchmark method, which continuously updates real-world forecasting tasks to mitigate pretraining contamination, and is based on multi-agent collection and verification to ensure that the news and events delivered to the forecasting models were in principle public before the prediction. The LEAF method transforms event-augmented forecasting into systematic Benchmark tasks and can analyze the effect of external events on LLM analysis. Based on 500 cross-domain event samples validated by domain experts, the authors validated the quality of event construction; at the same time, they manually reviewed the authenticity of events and the reliability of retrieval, etc.

Critical strengths
  1. Proposes a time-aware dual-agent fact-checking mechanism. Through collaboration between two different LLM agents, it ensures that retrieved news sources cannot leak information about the actual occurrence of events to the model, and effectively reduces the future information leakage rate.
  2. Uses the Living Benchmark approach to mitigate pretraining data contamination, i.e., LEAF continuously collects new forecasting targets, historical observations, and related events, thereby to some extent overcoming the shortcomings of static Benchmarks.
  3. The Benchmark covers 3 representative forecasting tasks: event forecasting, trend forecasting, and time-series forecasting, testing model forecasting performance from multiple aspects.
Critical weaknesses
  1. Manual review is a major advantage of this paper, but the selected 500 samples cover 47 different domains, which seems to have problems of insufficient data volume and insufficient representativeness; moreover, the 1.6% leakage rate is only an estimate for the current samples, not for all events; the number of expert-reviewed samples should be further expanded.
  2. The prompts used for w/ Events and w/o Events are not completely consistent; that is, the required model behavior differs considerably with and without events, which may introduce irrelevant elements into the experiment. The authors should explain this.
  3. Regarding the forecasting results, the paper seems to select the prediction targets for Event forecasting from true outcomes; that is, the authors may have known in advance whether events occurred and then specifically selected data to obtain experimental results. The authors are suggested to further explain how the prediction targets were specifically selected and whether the selection was fair and impartial, so as to ensure that the experiment can effectively measure the model’s true forecasting ability.
Questions for authors
  1. How do the authors ensure that the test data for different models are all after their respective cutoff times? The paper seems not to specify this in detail.
  2. Why are the prompts used for w/ Events and w/o Events not completely consistent?
  3. In addition to the forecasting baselines used in the paper, did the authors also compare with other work using baselines with the same external information sources?
2 comments
AuthorsOpenReview ↗

Thanks for the constructive comments!

We sincerely thank the reviewer for their constructive feedback and for recognizing the key strengths of our work, including our time-aware dual-agent fact-checking mechanism that effectively minimizes future information leakage, and our comprehensive evaluation across diverse forecasting tasks. Below, we address your specific questions and concerns in detail:

W1: Insufficient sample size for manual review

Thank you for recognizing our manual review efforts. There is a misunderstanding here: "47" refers to the number of domain experts involved in the manual audit, rather than the number of domains. We selected at least 100 samples per domain for verification and we will clarify this in the revised paper. Manual expert review is very expensive, making it infeasible for us to scale up the expert annotation further (we mentioned expert qualifications and cost in Appendix C.1). Moreover, since the primary concern of this verification is future information leakage, evaluating 500 samples and observing only a 1.6% leakage rate provides a relatively high level of confidence. Similarly, a comparable sample size (502 samples) was used for manual leakage inspection in AIA Forecaster [1]

[1] Rohan Alur, Bradly C. Stadie, Daniel Kang, Ryan Chen, Matt McManus, Michael Rickert, Tyler Lee, Michael Federici, Richard Zhu, Dennis Fogerty, Hayley Williamson, Nina Lozinski, Aaron Linsky, and Jasjeet S. Sekhon. 2025. AIA Forecaster: Technical Report. arXiv preprint arXiv:2511.07678.

W2: The prompts used for w/ Events and w/o Events are not completely consistent

For all main forecasting experiments in this paper, the prompts for w/ Events and w/o Events are completely identical. The only exception is the Stock Price Forecasting experiment (in Appendix F.2.1.), which is mainly intended to show that predicting exact stock prices is infeasible. After testing various prompts, we used the best-performing one for exact stock price prediction, yet it still failed (significantly underperforming the simplest baseline). We will clarify this in the paper.

W3: Select the prediction targets for Event forecasting from true outcomes

Thank you for the suggestion. While our initial evaluation only predicted the eventual winners (The event collection relied solely on the questions without any bias toward the final outcomes), we have conducted additional experiments predicting all candidate outcomes across all questions, including non-winning ones. The results show that providing text events (w/ Event) still consistently improves LLM performance. We will update the paper with detailed results, including each forecasting cut-off,

ModelMean Brier Score (w/o Evts)Mean Brier Score (w/ Evts)
Claude 4.6 Opus0.1810.129
Claude 4.6 Sonnet0.1740.123
Claude 4.8 Opus0.1710.128
Claude 5 Opus0.1690.112
Claude 5 Sonnet0.1770.119
Gemini 3.1 Pro0.1820.123
Gemini 3.1 Flash0.1880.143
Gemini 3.5 Flash0.1890.135
Gemini 3.7 Flash0.1750.115
Gemini 4 Argon0.1690.092
Grok 4.200.1750.125
Grok 4.20 (non-reasoning)0.1960.153
Llama 4 Maverick0.1800.128

In addition, we have scaled our event forecasting data to 1,199 questions, encompassing every question in our target domains resolved since October 2025. Initial experiments on this expanded set reveal that combining text events with market odds allows models to beat the market baseline (The Brier Score of market odds is 0.159). We will include the complete dataset and full empirical analysis in the revised version:

Model / Baselinew/o Evts, w/o Oddsw/ Evts, w/o Oddsw/o Evts, w/ Oddsw/ Evts, w/ Odds
Claude 5 Opus0.2540.1710.1610.147
Gemini 3.8 Flash0.3220.1760.1630.150

Q1: Ensuring model knowledge cutoff precedes test data

The knowledge cutoffs of most evaluated models precede the test data timeframe. Only a few models have cutoffs later than the weather and electricity datasets; however, since these datasets are fetched via specific APIs, they are unlikely to appear in pre-training corpora. This low contamination is further evidenced by the poor performance in the w/o Event setting (which significantly underperforms simple baselines).

Q2: Compare with other work using the same external events

Thank you for the suggestion. As a benchmark paper, our primary focus is to evaluate the foundational forecasting ability of base models augmented with events. Other methods mostly involve complex agentic frameworks or specialized training, which fall outside the scope of this work.

Dear Reviewer yruu,

Thanks for your time and comments, which helps a lot to further improve this work. As the discussion due is approaching, we would greatly appreciate it if you could join the discussion and let us know if there are any remaining areas in the paper that we can further clarify.

Best Regards,

Authors

Reviewer_KEwHOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

LEAF combines event, trend, and numerical time-series forecasting across questions from multiple sources. It constructs textual context through recursive retrieval and dual-agent verification. An expert audit of 500 examples reports a reduction in future-information leakage from 8.6% to 1.6%. Evaluations of 16 LLMs compare forecasts with and without textual context, alongside numerical forecasting baselines.

Critical strengths
  1. The audit examines temporal leakage, factuality, and generated attributes rather than assuming retrieved context is reliable.

  2. Comparing event probabilities, categorical trends, and numerical forecasts helps distinguish settings where textual context is useful from those where numerical baselines remain competitive.

Critical weaknesses
  1. Section 4.1 states that models predict the probability of the actual ground-truth outcome. Figure 13 explicitly identifies the “ultimate winning outcome” from Polymarket. This protocol cannot establish calibrated forecasting ability. It differs fundamentally from eliciting a normalized distribution over all outcomes.

  2. Live event evaluation is already what previous benchmarks have incorporated. Context is Key [1] evaluates text-conditioned time-series forecasting. LEAF’s combination of tasks and verification pipeline is potentially useful, but the paper does not clearly establish what substantial new understanding it enables. Some supporting analysis also has the risk of overclaiming: selecting context using the future outcome demonstrates sensitivity to input that depends on the outcome, rather than validating predictive relevance; at the same time, comparing historical forecasts does not isolate memorization as the cause of the reported 147% performance difference.

[1] Williams, Andrew Robert, et al. "Context is key: A benchmark for forecasting with essential textual information." arXiv preprint arXiv:2410.18959 (2024).

Questions for authors
  1. Does event evaluation query only the eventual winning outcome, as Figure 13 states? Please provide results using outcome-independent targets or normalized forecasts over all candidate outcomes.

  2. Do the forecasting gains persist after excluding every context containing expert-identified leakage? What evidence supports treating one-day leakage as negligible for three- and seven-day forecasts?

  3. Beyond combining existing benchmarks, what scientific finding is demonstrated specifically by LEAF?

3 comments
AuthorsOpenReview ↗

Thanks for the comments.

We thank the reviewer for their feedback and for recognizing the key strengths of our work. We also believe that some of the concerns raised may not be fully warranted (W2 below), and we address your specific questions and concerns in detail below:

W1 & Q1: Event prediction only targets the "ultimate winning outcome"

Thank you for the suggestion. We have conducted additional experiments predicting all candidate outcomes across all questions (including non-winning outcomes). The results show that providing text events (w/ Event) consistently improves LLM performance. We will update the paper with detailed results with each forecasting cut-off,

ModelMean Brier Score (w/o Evts)Mean Brier Score (w/ Evts)
Claude 4.6 Opus0.1810.129
Claude 4.6 Sonnet0.1740.123
Claude 4.8 Opus0.1710.128
Claude 5 Opus0.1690.112
Claude 5 Sonnet0.1770.119
Gemini 3.1 Pro0.1820.123
Gemini 3.1 Flash0.1880.143
Gemini 3.5 Flash0.1890.135
Gemini 3.7 Flash0.1750.115
Gemini 4 Argon0.1690.092
Grok 4.200.1750.125
Grok 4.20 (non-reasoning)0.1960.153
Llama 4 Maverick0.1800.128

Furthermore, since submission, we expanded the event forecasting dataset to 1,199 questions (covering all resolved questions in our target domains after October 2025). Results indicate that incorporating both market odds and events enables forecasts to outperform the market (the Brier Score of market odds is 0.159). The full experimental results and dataset will be updated in the paper.

Model / Baselinew/o Evts, w/o Oddsw/ Evts, w/o Oddsw/o Evts, w/ Oddsw/ Evts, w/ Odds
Claude 5 Opus0.2540.1710.1610.147
Gemini 3.8 Flash0.3220.1760.1630.150

W2: Live event evaluation has been incorporated

Prior time series forecasting benchmarks are largely limited by their data sources, which rely on synthetic data, restricting collection to well-defined domains, or using a single API that struggles to capture the most relevant events. In contrast, we focus on complex real-world scenarios and demonstrate that event context improves forecasting under strict leak-free conditions. Furthermore, Context is Key[1] relies on manually crafted contexts that contain substantial future information. For example, in the electricity consumption task (Appendix B.2 of [1]), the prompt explicitly tells the model about future events and their exact impact on the time series ("A heatwave struck the city, which began on 2012-10-09 18:00:00 and lasted for approximately 3 hours... the peak of the heatwave reached approximately 5 times..."). Similarly, the task in Appendix C.11 of [1] (Figure 19) directly reveals future events ("ATM is closed from 1997-09-05 00:00:00, for 8 days"). However, our events consist only of real-world events available prior to the forecasting cut-off timestamp.

While our Events Validation experiment is questioned for whether selecting events via outcomes measures true "predictive relevance," Context is Key—which explicitly includes future information in its forecasting inputs—is treated by the reviewer as a valid test of forecasting ability. This makes us question whether the reviewer thoroughly examined both the cited paper and our work when providing feedback.

[1] Williams, Andrew Robert, et al. "Context is key: A benchmark for forecasting with essential textual information." arXiv preprint arXiv:2410.18959 (2024).

Q2: Impact of leaked events on overall forecasting accuracy

We conducted an ablation study on the leaked events to evaluate their impact on accuracy before and after removal, testing the flagship model of each model family with 5 repeated runs per prediction. As shown in the table below, leaked events provide a minor boost—increasing weather and electricity trend forecasting accuracy by an average of 1.2% and reducing the event forecasting Brier score by 0.0012. However, since such events account for only 1.6% of the dataset, their impact on overall forecasting accuracy is negligible (only 0.0192% and 0.0000192, respectively). We will include the complete results and analysis in the revised paper.

ModelΔ Trend (accuracy, %)Δ EF (1 − Brier)
Gemini 3.1 Pro−3.6+0.0050
Claude Opus 5+0.0−0.0014
Grok 4.20+7.1+0.0000
Mean+1.2+0.0012

Q3: The scientific finding is demonstrated by LEAF

Compared to prior time series forecasting benchmarks, LEAF evaluates the forecasting capabilities of LLMs in realistic, complex environments where the causal relationships and impacts of events are neither obvious nor direct. Furthermore, LEAF is continuously updated to prevent pre-training data contamination, and its backtesting environment guarantees a leak-free setup for developing forecasting methods.

Dear Reviewer KEwH,

Thank you very much for your feedback. We have posted our response along with additional experiments addressing your comments. As the discussion deadline is approaching, we would greatly appreciate it if you could join the discussion and let us know if there are any remaining areas in the paper that we can further clarify or improve

Best Regards,

Authors

W1 & Q1 Table: To make the additional experiments completed, we also summarize the detailed results from W1 & Q1 in the two tables below.

ModelSettingp1 (100%)p2 (80%)p3 (60%)p4 (40%)p5 (20%)p6 (10%)all
Claude 4.6 Opusw/ Evts, no odds0.1610.1430.1340.1220.1250.0910.129
Claude 4.6 Opusno Evts, no odds0.1760.1760.1740.1760.1900.1960.181
Claude 4.6 Sonnetw/ Evts, no odds0.1510.1410.1220.1200.1120.0920.123
Claude 4.6 Sonnetno Evts, no odds0.1760.1660.1690.1730.1790.1800.174
Claude 4.8 Opusw/ Evts, no odds0.1560.1370.1290.1340.1220.0910.128
Claude 4.8 Opusno Evts, no odds0.1770.1740.1700.1660.1730.1680.171
Claude 5 Opusw/ Evts, no odds0.1380.1390.1210.1100.0980.0630.112
Claude 5 Opusno Evts, no odds0.1750.1630.1730.1630.1670.1770.169
Claude 5 Sonnetw/ Evts, no odds0.1430.1360.1090.1200.1190.0860.119
Claude 5 Sonnetno Evts, no odds0.1750.1870.1620.1790.1800.1790.177
Gemini 3.1 Prow/ Evts, no odds0.1540.1430.1150.1250.1100.0890.123
Gemini 3.1 Prono Evts, no odds0.1770.1690.1820.1920.1970.1790.182
Gemini 3.1 Flashw/ Evts, no odds0.1540.1620.1380.1410.1640.0990.143
Gemini 3.1 Flashno Evts, no odds0.1810.1870.1870.1930.1900.1910.188
Gemini 3.5 Flashw/ Evts, no odds0.1630.1530.1470.1370.1280.0790.135
Gemini 3.5 Flashno Evts, no odds0.1840.1810.1800.1860.1910.2100.189
Gemini 3.7 Flashw/ Evts, no odds0.1540.1490.1190.1060.0970.0630.115
Gemini 3.7 Flashno Evts, no odds0.1660.1680.1720.1650.1900.1890.175
Gemini 4 Argonw/ Evts, no odds0.1150.1060.0930.0910.0850.0620.092
Gemini 4 Argonno Evts, no odds0.1650.1700.1700.1660.1650.1770.169
Grok 4.20w/ Evts, no odds0.1550.1390.1280.1220.1160.0900.125
Grok 4.20no Evts, no odds0.1750.1700.1720.1740.1830.1750.175
Grok 4.20 (non-reasoning)w/ Evts, no odds0.1790.1650.1520.1420.1570.1210.153
Grok 4.20 (non-reasoning)no Evts, no odds0.1930.2000.2040.1900.2020.1900.196
Llama 4 Maverickw/ Evts, no odds0.1440.1300.1260.1280.1250.1120.128
Llama 4 Maverickno Evts, no odds0.1820.1740.1740.1620.1880.2020.180
ModelSettingp1 (100%)p2 (80%)p3 (60%)p4 (40%)p5 (20%)p6 (10%)All
Claude 5 Opusw/ Evts, w/ Odds0.1940.1800.1730.1560.1320.0450.147
Claude 5 Opusw/o Evts, w/ Odds0.2010.1870.1810.1660.1380.0900.161
Claude 5 Opusw/ Evts, w/o Odds0.2170.2030.1890.1710.1610.0840.171
Claude 5 Opusw/o Evts, w/o Odds0.2520.2500.2480.2500.2540.2700.254
Gemini 3.8 Flashw/ Evts, w/ Odds0.2050.1880.1770.1560.1310.0450.150
Gemini 3.8 Flashw/o Evts, w/ Odds0.2110.1920.1830.1660.1370.0920.163
Gemini 3.8 Flashw/ Evts, w/o Odds0.2370.2120.1960.1730.1560.0810.176
Gemini 3.8 Flashw/o Evts, w/o Odds0.3130.3140.3170.3190.3240.3430.322
Marketodds at cut-off0.2100.1900.1800.1600.1310.0820.159

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.