- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces LEAF, a living benchmark for event-augmented forecasting. The three tasks are event forecasting on Polymarket questions, trend forecasting as 3-class or 5-class classification, and time series forecasting. The data cover large-cap and mid-cap stocks, commodities, electricity load, weather and Polymarket events, and the evaluation horizons in Table 2 run from 10.2025 to 09.2026. A retrieval agent powered by Gemini 3.1 Pro and augmented with Gemini Grounded Search collects the events. An event is kept only when two fact-checking agents, Gemini 3.1 Pro and Claude 4.6 Sonnet, agree on both its authenticity and its public release time.
The evidence comes in three parts. First, 47 domain experts audited 500 sampled tasks. The fact-checking module reduces look-ahead leakage from 43 to 8 of these tasks, or from 8.6% to 1.6%, and raises date accuracy from 305 to 405 tasks. Eight of the 500 contain factual fabrications. Second, the temporal ablation in Table 11 shows that TS-only accuracy on stock trend forecasting is much higher on a historical test period from January 2023 to June 2024 than on out-of-sample living data. Third, the authors evaluate 16 LLMs with and without events and compare their time series forecasts with TimesFM 3.0, Chronos-2, the Last Value Baseline and the Historical Mean Baseline.
The paper finds that events improve trend and event forecasting for most models, and that most LLMs perform on par with the TSFMs on hourly point forecasting, with Claude Opus 5 doing better. The Last Value Baseline has the lowest MAPE in Table 9, and the predictability levels assigned by LLMs correlate with stock trend forecasting accuracy. The authors also use sentiment-consistent text in Sec. 3.2 and Fig. 8 to validate that the retrieved events are relevant.
Critical strengths
The paper addresses a real problem. LLM forecasting evaluations suffer both from pre-training data contamination and from look-ahead leakage in automated search. LEAF targets both problems and pairs textual events with time series data. The evaluation is broad, covering 16 LLMs, two TSFMs and two naive baselines. Tables 12 and 15 to 18 report 95% confidence intervals, and the anonymous repository with code and data is accessible.
The human-expert audit is a contribution on its own. It shows that unverified automated search leaks future information in 43 of the 500 audited tasks, and that 8 leaks remain after dual-agent cross-validation. Tables 5 and 6 list these eight residual leaks and the eight hallucinations, with verbatim excerpts. Appendix C.3 finds that the generated causality labels reach only 17.3% expert acceptance, or 52 of 300 tasks, which is a clear warning against trusting rationales written by LLMs. Table 11 shows a large gap between the historical test period and the living data. For 3-class mid-cap stocks at the 6-month horizon, for example, Gemini 3.1 Pro reaches 92.36 on the historical period but only 46.02 on the living data. This gap is consistent with pre-training data contamination. These findings are useful for anyone who builds living benchmarks for event-augmented forecasting.
Critical weaknesses
Event forecasting is scored only on the winning outcome. Sec. 4.1 says the model is asked for "the predicted probability of the actual ground-truth outcome", and the caption of Fig. 13 says target_outcome "represents the ultimate winning outcome". So o_t = 1 for every scored forecast, and the Brier score reduces to (1 - f_t)^2. A constant forecast of 1 would get a perfect score. A constant forecast of 0.5 would score 0.25 and beat every model at the 100% time-to-resolution cut-off, with or without events, since Table 15 reports Brier scores from 0.398 to 0.598. Because the other potential outcomes are never queried, a higher probability on the target outcome after adding events could simply mean that the model now gives higher probabilities across the board. The prompt in Fig. 13 asks the model to weigh market pricing, yet no baseline that uses the Polymarket odds is reported.
The weather events already contain official forecasts. Items [54] and [57] in Fig. 19 and Listing 2 include NWS forecasts of daytime highs and overnight lows for the coming days, as well as CPC 6-10 and 8-14 day outlooks. The TSFMs receive none of this, and no baseline uses these forecasts directly. The gains in Tables 3, 4 and 16 to 18 therefore cannot be credited to the LLM's own reasoning.
The trend tasks have no baseline that always predicts the most frequent class, even though the label distribution is imbalanced. In Table 10, 43.69% of the ground truth for long-term mid-cap stocks is Strong Up and 29.40% is Strong Down. For electricity, App. D.1 reports 36.70% Strong Up and 17.32% Up. The trend prompts in Figs. 14 and 15 suggest that the 3-class Up label covers both Strong Up and Up. In that case, always predicting Up would reach about 53% to 54% on these label distributions, which is higher than many TS-only accuracies in Tables 12 and 16. The weather label uses the minimum temperature over the future N days, so the label distribution may also change with the horizon.
In stock price forecasting, the prompt differs between the TS and w/ Evts settings. Fig. 12 shows two analytical frameworks. The one used when event intelligence is present contains a Random Walk step that tells the model to default to a stable outlook. The one used when no events are available guides the model to rely on technical price action and momentum. The shared system prompt also says "Do NOT lazily repeat the last closing price". The comparison between TS and w/ Evts for stocks in Table 9 therefore mixes the effect of the events with the effect of the prompt. Sec. 4.3 and App. F.1 also conclude that market price forecasting is inherently infeasible. This conclusion does not follow from a setup that prohibits the model from repeating the last closing price, since that is exactly how the Last Value Baseline wins in Table 9.
The sentiment-consistent text in Sec. 3.2 is selected using the ground-truth change. It shows that models respond to events whose sentiment matches the actual future trajectory, but not that the retrieved events are informative without this selection.
Several claims go beyond the tables. The phrase "consistently outperform ... across all settings" in Sec. 4.2 conflicts with Table 13 and with two cases in Table 3, where Gemini 3.7 Flash drops from 71.34 to 70.73 on electricity at the 7-day horizon and Claude Opus 5 drops from 39.72 to 38.94 on large-cap stocks at the 1-day horizon. App. F.2.1 says the correlation between predictability levels and accuracy "vanishes when events are excluded", yet Table 14 reports TS-only correlations of 0.919, 0.945 and 0.829. The Introduction says that evaluating on data before training cutoffs inflates forecasting accuracy "by up to 147%". In Table 11, however, 147% is the mean of the 16 relative performance gains. The median is 98.6% and the maximum is 769.5%, so 147% is an average and not an upper bound.
Questions for authors
What is the documented training cutoff of each of the 16 models, and which date range is scored for each model and task?
Which Polymarket questions are scored? Table 2 lists 66 entities from 04.2026 to 09.2026, App. D.2.5 mentions 30 tasks, and the scripts in the repository list identifiers dated from 2025 to early 2026. Please define question, market, outcome and entity.
In event forecasting, is the model ever asked about potential outcomes other than the winning outcome?
How were the 500 audited samples drawn? Sec. 3.2 calls them events, Sec. 3.4 calls them tasks, and App. C calls them event-driven forecasting instances. Which unit was sampled, and from which entities? Also, what does RAM / Comm. in Table 7 refer to, given that App. C names this domain Hardware/Commodities and App. D.2.2 lists ten commodities in energy, metals and agriculture?
For the 1-day large-cap stock task, does the forecast start before or after the closing price of the cut-off date is known? The 3-day example in Fig. 18 includes items [76] to [80], which carry the same date as the cut-off.
Please explain the following mismatches.
The Gemma rows differ between Table 3 and Table 16. With TS only at the 7-day horizon, Gemma3-27B-it scores 45.73 on electricity in Table 3 and 54.9 in Table 16, and Gemma3-12B-it scores 42.75 on weather in Table 3 and 51.3 in Table 16.
For Claude 4.6 Opus on long-term mid-cap stocks at the 6-month horizon with events, Table 12 reports 28.6, while Table 14 reports between 40.88 and 57.58 at every predictability level.
Sec. 3.4 reports a 0% false negative rate, which does not fit the definitions in App. C.5.
Impact Type is reported as 179/201, but the two domains that evaluate this attribute in Table 7 have only 200 Evaluated Tasks.
App. C.1 gives an average of 13.6 years of experience, but the specialist counts and average years listed for each domain give 12.73.
What does DLV in Table 9 stand for?
App. E.2 computes the Overall score as the mean over the metrics available for each model, and some models have missing cells, such as Grok 4.20 non-reason in Table 17. Are the Overall scores in Fig. 4 comparable across models evaluated on different subsets of metrics?
2 comments
Dear Reviewer HJFR,
We truly appreciate the time and constructive insights you dedicated to reviewing our submission. With the author–reviewer discussion period wrapping up soon, we wanted to check in and see if our rebuttal addresses your questions, or if there are any open points you would like us to elaborate on before the window closes.
Best Regards,
Authors
Thank you for your detailed comments, many of which point out writing and clarification issues that we will correct in the revised paper; below, we focus on addressing your primary concerns:
W1: Event forecasting is scored only on the winning outcome.
We expanded our evaluation to predict all candidate outcomes across every question, not just the eventual winners; under this setting, adding text events (w/ Event) still consistently improves LLM performance. Note that while our initial scoring focused on winning outcomes, our event retrieval relied solely on the questions, ensuring zero outcome bias. We will incorporate the detailed results (including each cut-off) and prompts into the revised paper:
Following the same experimental setting described above, we also collected every resolved question in our target domains since October 2025, expanding the event forecasting benchmark to 1,199 questions. On this dataset, our preliminary findings show that models can outperform the market baseline (Brier Score of market odds is 0.159) when integrating both market odds and text events.
W2: The weather events already contain official forecasts
Thank you for the comment. As a benchmark, our goal is to evaluate the foundational forecasting capabilities of LLMs; comprehending textual weather forecasts and integrating them to improve predictions directly reflects this capability. Furthermore, we do not claim to propose a SOTA forecasting method, our core claim is simply that textual event context is beneficial for forecasting.
W3: The prompt differs between the TS and w/ Evts settings in Stock price forecasting
Yes, the Stock Price Forecasting task (Appendix F.2.1) is the sole exception where the prompts differ, as its primary purpose is to demonstrate the infeasibility of predicting exact stock prices. Even after testing multiple prompt variations and selecting the highest-performing one, the model still fell significantly short of the simplest baseline. In contrast, across all main forecasting experiments in the paper, the w/ Events and w/o Events settings use completely identical prompts. We will make this distinction clear in the revised manuscript.
W4: The sentiment-consistent text does not reflect that events are informative
Thank you for the comment. Event validation is intended to verify relevance: our ablation shows that selecting sentiment-consistent texts from other targets yields no strong correlation between accuracy and the proportion of relevant texts. Meanwhile, our main forecasting experiments directly prove that retrieved events are informative for prediction. We will clarify this objective in the revision.
W5: No baseline that always predicts the most frequent class
Thank you for the suggestion. However, this does not serve as a valid baseline because all models only have access to information prior to the cut-off date, and forcing a model to always predict the most frequent test label leaks future information, making it an unfair comparison.
Q1: Supplementary details on Polymarket questions, model cutoffs, and audited samples
We will add these details in the revision. Most models have knowledge cutoffs prior to the test periods. While a few overlap with the weather and electricity data, these datasets are fetched via specialized APIs and are unlikely to be in pre-training corpora—as evidenced by the poor w/o Event results, which trail simple baselines. For the manual audit, we randomly sampled an equal number of events per domain.
Q2: Clarification on stock cut-off date, Gemma3-27B-it results (Tables 3 & 16), and Claude results (Tables 12 & 14)
Yes, stock tasks include all data up to and on the cut-off date. For Gemma3-27B-it, we updated the re-run results in Table 3 but missed Table 16, which we will sync in the revision. Tables 12 and 14 are not directly comparable as they use different data batches: since LEAF is a living benchmark, we only update main results with new data rather than re-running prior validation experiments (e.g., Table 14 and Event Validation, which still strictly post-date model cut-offs). We will clarify this in the revision.