acceptodds
Under review as a conference paper at ICLR 2027

Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents

Abstract

We introduce **Forecast-Dojo**, a replayable environment for benchmarking and training LLM forecasting agents. It combines resolved prediction-market questions with dated news, allowing agents to research an event and revisit their predictions at successive historical dates. The same tasks and tools support repeated evaluation, collection of training interactions, and feedback from recorded outcomes without waiting for new events to resolve. **Forecast-Dojo** contains 1,568 Polymarket events, split by time into training and evaluation periods, and 18.8M dated news articles. In an evaluation of 12 models, research tools lower Brier score for all 12. Forecasts also improve as events unfold, with the largest gains at steps where more newly dated evidence is recorded. Every model still trails historical market forecasts in both Brier score and accuracy. A belief notebook carried between dates lowers research cost but does not consistently improve forecast quality. A supervised fine-tuning demonstration uses the collected trajectories to improve a student's accuracy from 34.8% to 42.8% on later, held-out events.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, Event forecasting, RL environment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yLXvOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper presents Forecast-Dojo, a replayable environment for evaluating and training LLM forecasting agents. It pairs 1,568 resolved Polymarket events with 18.8M English CC-News articles assigned day-level dates. Each event is replayed at 3–10 fixed forecast dates, one per time bin, chosen to favor days with large market-price moves and high overall news volume. Every agent therefore faces the same dates and the same corpus cutoff. At each date, the agent can search and read only articles dated on or before that day and run sandboxed Python. In a memory-on mode, a structured belief notebook is carried to the next date. Events are split by time into 1,338 training and 230 evaluation events.
  • The contributions are the environment, the dataset, and a study of 12 proprietary and open-weight models (4 rollouts each) under three settings: no tools, tools, and tools plus notebook.
  • They fine-tune Qwen3-30B-A3B-Thinking-2507 on Qwen3-235B-A22B-Thinking-2507 trajectories. One trajectory per task is selected by a score combining Brier, accuracy and tool-call count. Fine-tuning raises held-out accuracy from 34.8% to 42.8%. Validation uses paired event-level bootstrap CIs, failure accounting under three treatments of unusable outputs, and a no-tools check for outcome recall.
Critical strengths
  1. Useful infrastructure that fills a real gap. Fixed forecast schedules, date-restricted retrieval over a large corpus, sandboxed tools and a temporal train/eval split let different agents be compared on the same dates and corpus cutoff. They also allow trajectories with outcome feedback to be collected without waiting for events to resolve. Live benchmarks cannot be replayed, and per the paper’s comparison (Table 1), few prior benchmarks combine agentic research with a shared multi-date schedule. This weighs heavily in my assessment because the main contribution is infrastructure, and the paper links code and data. Its value therefore does not depend on any single empirical finding.

  2. Careful, transparent evaluation. The study covers 12 models × 3 settings × 4 rollouts and uses paired event-level bootstrap CIs. It reports failure rates per model and shows Brier under three treatments of unusable outputs (Tables 13, 24), along with cost and tool use per model. Null and fragile results are reported candidly: the memory null, the sensitivity of the κ correlation, and the selection analysis in C.4. This makes the two most robust claims credible: tools help every model, and every configuration trails the market in Brier and Information-α (all 36 CIs exclude zero). The same transparency is what made the weaker claims below auditable.

Critical weaknesses
  1. I don't think the two longitudinal claims in the abstract are established yet, and those are what set this benchmark apart from earlier work.

The first issue is that event selection uses the realized outcome in several places (Appendix A.2, Table 5). A day only counts as "active" if the market leg for the eventual winner traded enough, which can exclude events won by thinly traded options such as long shots (190 of the 230 test events are multi-option). The a_Q filter drops events where the market already gave the realized outcome near-certain probability. The C_Q filter requires the market's net movement toward the realized outcome to be at least 20% of its total movement, and events that fail it survive only if their total movement is large (V_Q of at least 0.5). To be fair, Table 6b shows that the nontriviality filters (a_Q and C_Q) removed only 45 of 423 evaluation candidates. But the activity filter removed most candidates (7,590 down to 423), and the paper doesn't separate how much of that comes from measuring activity on the winning leg. On top of this, forecast dates within each time bin are picked mostly by market movement (weight 0.7), and same-day articles are visible to the agent.

Table 27 shows why this matters. The entire early-to-late improvement (0.670 to 0.606) comes from the 175 events where the market's Brier fell. On the other 49 events, memory-free agents don't improve at all (+0.005, CI [-0.027, 0.038]). The tools vs. no-tools contrast is real, but as the authors say themselves, the size of the effect depends on how many such events the filters keep. They acknowledge this in Section 5.3 and Appendix C.4, but the abstract states the finding without the caveat, and there are no results under outcome-agnostic selection.

The second issue is that the claim about gains being largest where more new evidence appears relies on a proxy that depends on agent behavior. The index counts other models' self-reported notebook entries dated within the interval, and the authors note that it can miss evidence no model found. The market improves even in the lowest quartile (-0.051) while the agents barely move, which fits that concern. I also couldn't find any control for the number of days between forecast dates or the distance to close. Longer gaps mechanically allow more dated entries, and you'd also expect bigger Brier changes over longer gaps. The related model-level correlation in Fig. 2(c) (Spearman 0.74 across 12 models) drops to 0.14 under worst-case scoring and is close to zero within models (r = -0.017).

Last, the episodes are short. 163 of the 230 events have only 3 forecast dates, and the median time from first forecast to close is 9 days. For most events the notebook is only carried forward twice, so I'm not sure the memory null says much about longer horizons, where memory should matter more.

If these claims don't hold up, what's left is that tools help, markets are better, and memory saves money. Those are useful results, but they're largely in line with earlier retrieval-augmented forecasting work, and the benchmark's distinctive longitudinal angle wouldn't be validated.

  1. The training half of the paper isn't really demonstrated yet. The SFT experiment shows that a weak student that rarely calls tools gets better when trained on a stronger model's selected trajectories. It doesn't show what the environment's outcome feedback adds.

The teacher's score on the 230 evaluation events isn't reported, so I can't tell how close the student gets to just running the teacher. There's also no ablation of the selection rule, such as taking a random eligible rollout or selecting on tool count alone. That matters because the selection score weights tool-call count at 0.5, next to 0.7 for Brier and 0.3 for accuracy, and tool calls go from 1.4 to 3.7 per forecast. A lot of the gain may simply be the student learning to use the tools.

In terms of probabilistic quality, the trained student is still around uniform level. Its Brier is 0.749 against 0.771 for uniform, and it is actually worse than uniform on log loss (1.89 vs. 1.74) and on binary events (0.539 vs. 0.500). The authors point out that oracle shrinkage of the base model nearly matches the raw Brier gain (0.752 vs. 0.749). The accuracy gain is real (42.8% vs. 34.8%, with uniform at 22.9%), and shrinkage doesn't change it. Only one training run is reported, at the final checkpoint, and neither RL (marked as possible in Fig. 1) nor on-policy distillation is tried.

Training is half the title and one of the main reasons to build a replayable environment instead of a static dataset. A date-restricted corpus does make it possible to research resolved questions without leakage, which is a real advantage. As it stands, though, the experiment doesn't show that the outcome feedback adds anything beyond imitating a stronger model's tool use.

Questions for authors
  1. Outcome-agnostic selection. Table 6b gives counts per stage, but the activity stage is measured on the realized outcome's leg. How many evaluation events would pass outcome-agnostic versions of the activity and nontriviality filters (for example, activity on total event volume, uncertainty from the market's maximum probability, and total movement without direction)? Could you rerun the main analyses on that set: tools vs. no tools, the late-minus-early change, Fig. 2(b), and the market gap? A version with uniformly spaced forecast dates would also help. If the longitudinal results hold up there, most of my first concern goes away.

  2. Evidence index. Does the Fig. 2(b) gradient survive controls for days between consecutive forecast dates, days to close, step index, domain and number of outcomes? Does it hold with a corpus-side measure that doesn't depend on what agents found, for example relevant articles dated in the interval that fixed event-specific queries retrieve?

  3. Training. Could you report (a) the teacher's scores on the evaluation set under the same protocol, (b) SFT on randomly selected or tool-count-only selected rollouts, (c) a non-oracle recalibration baseline for the base student, with shrinkage fit on training events, and (d) the number of evaluation rollouts or seeds behind Table 3? If outcome-based selection doesn't beat random selection, it's unclear what the outcome feedback contributes to the gain.

  4. Leakage in what agents actually read. As I understand Appendix A.2, the content-level leakage check covers the 15 documents the judge retrieves per event at close minus 2 days. Agents read many more than that (about 27 full articles per forecast for GPT-5.6 Sol, Table 16). Did you audit a sample of the articles agents actually scraped for information from after the forecast date? Did you look at the event-dates where agents beat the market by the widest margin? And since Appendix A.1 says the copy kept for a URL crawled more than once isn't necessarily the earliest, how often does the kept version postdate its assigned date?

  5. The judge's confident_lean field. The judge prompt says this field is there so you can later check whether SUFFICIENT verdicts track correct calibration, but I couldn't find that analysis. How often does it match the realized outcome? If the match rate is very high, late-step questions may be largely answerable from the corpus, which would change how I read the late-step scores.

  6. Accuracy vs. market. The abstract says every model trails the market in accuracy, but paired tests are only reported for Brier and Information-α (Table 21). Could you add paired accuracy CIs (best model 57.69% vs. market 64.55%)?

  7. Future models. Models trained on data from after June 2026 may know the evaluation outcomes. Is there a plan to refresh the evaluation split, and will the market-wrong recall check from Appendix B.1 be part of the standard protocol? This seems important given the advantage over live benchmarks argued in Section 1.

7 comments

We thank the reviewer for the thoughtful and detailed feedback, and for recognizing the value of our replayable infrastructure, rigorous evaluation, and experimental transparency. Below, we address all the concerns and questions.

W1 & Q1. Outcome-Agnostic Event Selection and Forecast-Date Robustness

We thank the reviewer for raising this important concern. We agree that outcome-dependent event filtering and movement-prioritized forecast-date selection may influence the observed longitudinal findings and warrant direct validation. We therefore conducted two targeted robustness experiments addressing these choices separately. We re-evaluated six representative models (GPT-5.6 Sol, GPT-5.5, Claude Opus 4.8, GLM-5, Kimi K2.5, and MiniMax M2.5), covering proprietary and open-weight models, with four rollouts per forecast date across all three evaluation settings.

1. Outcome-agnostic event selection.

Rationale for the original filter. Forecast-Dojo uses historical Polymarket prices as a proxy for contemporaneous market beliefs, making the reliability of these price-based references important for evaluation. In multi-option markets, high overall trading volume does not necessarily imply sufficient activity on every outcome leg. We therefore originally prioritized activity on the realized-outcome leg, since Information-α directly compares the agent's probability of the realized outcome with the market's corresponding probability, and the market's Brier score also depends on this probability as part of its full distribution. The winner-leg filter was intended to prioritize reliable historical market references, although we recognize that it could affect which events are selected.

Additional experiment. Following the reviewer's suggestion, we reconstructed the evaluation set using outcome-agnostic filters based on total event trading activity, uncertainty measured from the market's maximum outcome probability, and undirected market-belief movement. This yields 268 events (907 forecast dates), retaining 192/230 (83.5%) original evaluation events and introducing 76 newly eligible events. We reused the original forecasts for overlapping events and evaluated the 76 new events under the same protocol.

Results. The four main findings remain consistent:

(i) Research tools improve forecasting. All six models retain significant Brier improvements, with a pooled memory-free versus no-tools difference of −0.103 [−0.139, −0.071], compared with −0.119 on the original six-model evaluation.

(ii) Longitudinal improvements persist. The late-minus-early Brier change is −0.069 [−0.091, −0.048] for memory-free and −0.069 [−0.093, −0.048] for memory-on, while no-tools remains nearly flat (−0.003).

(iii) The evidence gradient persists. On the 76 newly included events, more newly recorded evidence remains associated with larger Brier improvements for memory-free (β = −0.043 [−0.085, −0.009]) and memory-on (β = −0.037 [−0.081, −0.001]), but not no-tools.

(iv) The market gap remains. All 18 model–setting configurations continue to trail the historical market in paired Brier comparisons, with positive gap CIs, and retain negative Information-α.

Importantly, the 76 newly included events alone also show significant pooled improvements from research tools (−0.033 [−0.064, −0.004]) and across forecast stages (memory-free: −0.056 [−0.080, −0.032]). Thus, the main patterns are not supported solely by events shared with the original evaluation set.

2. Uniformly spaced forecast dates.

Additional experiment. We also agree that prioritizing dates with large market-belief movements could influence longitudinal measurements. To isolate this effect, we replaced the original movement-prioritized schedule with uniformly spaced forecast dates, selecting the midpoint of each temporal bin while preserving the original forecasting horizon constraints and number-of-steps rule. This yields 227 events (789 forecast dates), on which we evaluated the same six models across all three settings.

Results. The same four findings persist:

(i) Research tools improve forecasting. All six models achieve significant Brier improvements, with a pooled memory-free versus no-tools difference of −0.111 [−0.150, −0.073].

(ii) Longitudinal improvements persist. Memory-free and memory-on forecasts improve significantly from early to late stages (−0.051 [−0.074, −0.028] and −0.055 [−0.079, −0.033], respectively), while no-tools remains nearly unchanged (−0.003 [−0.012, +0.005]).

(iii) The evidence gradient persists. The highest-versus-lowest new-evidence quartile contrast in step-to-step Brier change remains negative for memory-free (−0.033) and memory-on (−0.046), compared with −0.006 for no-tools, reproducing the pattern in Figure 2(b).

(iv) The market gap remains. All 18 configurations continue to trail the market in paired Brier comparisons, with positive gap CIs and negative Information-α. Moreover, the market's Brier is essentially unchanged between the original and uniform-date schedules (paired difference −0.001 [−0.010, +0.009]).

Overall, these two ablations reinforce our central findings: research tools improve forecasting, research-enabled agents improve as events unfold, larger improvements are associated with newly available evidence, and historical market forecasts remain stronger. These patterns persist under outcome-agnostic event selection and uniformly spaced forecast dates, supporting their robustness to both design choices.

The revised manuscript includes the full model-wise comparisons and longitudinal plots in §5.6 and Appendix C.6 (Tables 30–31 and Figure 5).

Clarification on episode length and memory. We would like to clarify the scope of our finding. We do not claim that memory is ineffective for longer-horizon forecasting. As stated in our paper (lines 22–24 and 80–82), our conclusion is specific to the evaluated setting (median horizon of nine days): persistent memory consistently reduces research costs but does not yield consistent improvements in forecast quality. We do not extrapolate our findings to longer horizons, where memory may play a different role.

Q2. Evidence Index: Temporal Controls and Independent Corpus-Side Validation

We conducted two additional analyses to assess whether the evidence gradient in Fig. 2(b) is driven by temporal confounders or agent-dependent evidence measurement.

1. Temporal-controlled analysis. We regress consecutive-step Brier changes on the original leave-one-model-out (LOMO) evidence index across 230 evaluation events (567 forecast transitions). Specifically, we include the calendar-day gap between forecasts, days remaining until market close, and forecast step index as temporal covariates, alongside domain indicators, log₂(number of outcomes), and model fixed effects to account for event and model heterogeneity. We use event-balanced estimation, giving each event equal weight, and compute 95% confidence intervals from 4,000 event-level bootstrap resamples.

SettingUnadjusted βAdjusted β [95% CI]Adjusted Q4−Q1 [95% CI]
Memory-free−0.044−0.046 [−0.071, −0.022]−0.069 [−0.107, −0.032]
Memory-on−0.048−0.047 [−0.073, −0.024]−0.068 [−0.103, −0.034]
No-tools+0.001+0.003 [−0.005, +0.012]+0.004 [−0.006, +0.016]

Here, β measures the association between log(1+evidence index) and consecutive-step Brier change (negative indicates improvement). Q4−Q1 compares adjusted Brier changes between the highest and lowest evidence quartiles.

The gradient remains nearly unchanged after adjustment. For memory-free agents, β changes from −0.044 to −0.046, with a substantial adjusted Q4−Q1 difference of −0.069. Memory-on agents show the same pattern, whereas no-tools agents exhibit no comparable association. The results also persist with event fixed effects (adjusted Q4−Q1 = −0.082 for memory-free and −0.075 for memory-on), nonlinear gap-length controls, and worst-case scoring of unusable forecasts.

2. Independent corpus-side evidence index. To eliminate dependence on agent-recorded evidence, we construct a separate index directly from the dated CC-News corpus. For each event, we reuse fixed queries generated during evidence-sufficiency screening, together with the event title, without access to outcomes, market prices, or agent forecasts. Within each forecast interval, we retrieve candidates using dense (Qwen3-Embedding-8B) and BM25 search (top-200 each per query), then count unique articles with a maximum dense cosine similarity ≥ 0.55 across the fixed queries, independently of agent-read evidence.

The corpus-side index agrees strongly with the original LOMO index (Spearman ρ = 0.70), with 89% of transitions assigned to the same or adjacent evidence quartiles. More importantly, it reproduces the evidence gradient under the same temporal controls:

Evidence indexMemory-free adjusted Q4−Q1 [95% CI]Memory-on adjusted Q4−Q1 [95% CI]No-tools adjusted Q4−Q1
LOMO notebook−0.069 [−0.107, −0.032]−0.068 [−0.103, −0.034]+0.004
Corpus-side−0.061 [−0.117, −0.014]−0.076 [−0.130, −0.030]+0.005

The gradient also persists across alternative cosine thresholds (0.55, 0.60, 0.65).

Together, these results support the robustness of our longitudinal finding. The evidence–forecast improvement association persists after controlling for the proposed confounders and when evidence availability is measured independently of agent notebooks. Moreover, the gradient appears consistently in research-enabled settings but not in no-tools forecasting. While these analyses remain observational, they show that the reported pattern is not solely an artifact of temporal spacing or agents' evidence-recording behavior.

We have incorporated these analyses into the revised manuscript in Section 5.6 and Appendix C.7 (Figure 6 and Table 32).

W2 & Q3. Training Ablations and Forecasting Improvements

We thank the reviewer for these constructive suggestions. To better isolate the contribution of outcome-based feedback to demonstration selection, we conducted additional SFT ablations under identical training and evaluation settings. Random-SFT uniformly selects eligible teacher trajectories, Tool-SFT selects those with the most tool calls, and Dojo SFT uses outcome-guided selection. All three use 3,965 demonstrations, with the selected teacher trajectories averaging 3.26, 4.19, and 3.79 tool calls, respectively. All students use identical training configurations and are evaluated on 230 held-out events (797 event–date pairs; four rollouts each; evaluation seed 0). We additionally report teacher performance and a non-oracle recalibration baseline fitted exclusively on training events.

ModelAcc. (%) ↑Brier ↓ΔAcc. vs. Base (pp, 95% CI)ΔBrier vs. Base (95% CI)
Base34.80.924——
Random-SFT40.60.788+5.8 [3.0, 8.6]−0.136 [−0.174, −0.101]
Tool-SFT41.10.782+6.3 [3.4, 9.3]−0.143 [−0.184, −0.105]
Dojo SFT42.80.749+8.1 [5.1, 11.2]−0.176 [−0.214, −0.139]
Teacher44.30.721+9.5 [6.7, 12.3]−0.203 [−0.242, −0.164]
Base + recalibration34.80.758—−0.166 [−0.205, −0.129]

All three SFT strategies substantially improve forecasting over the base model, with outcome-guided selection achieving the best average performance. Dojo SFT improves accuracy by 2.2 and 1.7 percentage points over Random-SFT and Tool-SFT, respectively, while reducing Brier by 0.039 and 0.033. Notably, Dojo SFT also uses fewer tools at evaluation than Tool-SFT (3.7 vs. 4.6 calls per forecast), suggesting that outcome feedback provides a useful selection signal beyond simply favoring more tool calls.

Interpreting the SFT gains. We emphasize that uniform forecasting is a conservative no-information reference, not a worst-case Brier baseline. Brier penalizes overconfident errors, so even models with meaningful predictive ability may score near or worse than uniform. This motivates evaluating both probabilistic quality (Brier score) and outcome selection (accuracy). Dojo SFT improves accuracy by 8.1 percentage points and reduces Brier by approximately 19% relative to the base student, with both paired CIs excluding zero. Its accuracy also substantially exceeds uniform (42.8% vs. 22.9%). Moreover, non-oracle recalibration reduces the base Brier to 0.758 but leaves accuracy unchanged at 34.8%. Thus, confidence adjustment alone cannot explain the accuracy improvements achieved through SFT.

We have incorporated these training ablations, teacher comparisons, and non-oracle recalibration results into the revised manuscript in Section 5.5 (Table 3). Additional details on demonstration selection, training configurations, and calibration diagnostics are provided in Appendix C.5 (Tables 28–29 and Figure 4).

Reviewer_jfcGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Forecast-Dojo provides replayable forecasting environments built from resolved Polymarket events and dated news articles. Agents research questions at fixed historical checkpoints, optionally carrying a belief notebook between dates. Experiments evaluate 12 models on 230 temporally held-out events with four rollouts. Research tools consistently improve forecasting, while memory reduces cost without consistently improving accuracy. A supervised fine-tuning demonstration improves student accuracy from 34.8% to 42.8%.

Critical strengths
  1. Useful experimental infrastructure. Fixed date checkpoints, shared tool interface, and temporal training/evaluation splits support repeatable comparisons and agent training. This addresses a practical limitation of live forecasting evaluations.

  2. Careful empirical analysis. The study goes beyond rankings through paired comparisons, event-level uncertainty estimates, failure analysis, and cost measurements. It also acknowledges important qualifications, including the limited evidence for memory in improving forecast quality and the role of recalibration in the training gains.

Critical weaknesses
  1. Forecast-Dojo substitutes a filtered CC-News archive and dense retrieval for the information environment that live forecasting agents encounter. The paper does not establish how closely this environment matches live search in publisher coverage, topic coverage, freshness, source quality, or access to primary sources. Consequently, observed model rankings, memory benefits, research costs, and training gains may depend on the archive and retrieval system. Acknowledging limited corpus coverage does not quantify this gap or establish which findings transfer to live forecasting.

  2. Historical forecasting with dated news, sequential forecast revision, and agent interaction has already been explored by Autocast [1], EvolveCast [2], and FutureSim [3]. Forecast-Dojo’s clearest additions appear to be fixed, matched forecasting checkpoints and an integrated evaluation/training interface. These are useful engineering choices, but the paper does not convincingly establish what important scientific question they allow us to answer beyond existing environments. The empirical findings provide useful measurements, yet offer limited explanation of agents’ forecasting behavior or evidence that the conclusions transfer beyond this particular corpus and harness.

[1] Zou, Andy, et al. "Forecasting future world events with neural networks." Advances in Neural Information Processing Systems 35 (2022): 27293-27305. [2] Yuan, Zhangdie, Zifeng Ding, and Andreas Vlachos. "Do Language Models Update their Forecasts with New Information?." arXiv preprint arXiv:2509.23936 (2025). [3] Goel, Shashwat, et al. "FutureSim: Replaying world events to evaluate adaptive agents." arXiv preprint arXiv:2605.15188 (2026).

Questions for authors

How representative is the benchmark’s information environment of the sources and evidence encountered by live forecasting agents? Can you quantify source overlap, missing evidence, and retrieval differences against representative live-search systems?

What evidence supports transfer of the benchmark’s findings to live forecasting? In particular, do model rankings, memory’s cost-quality tradeoff, and training improvements persist in a matched prospective comparison using live search? If such validation is unavailable, which conclusions should be restricted to forecasting within this particular archive?

What is the strongest scientific finding enabled specifically by Forecast-Dojo’s design that existing historical forecasting environments could not adequately establish? Which experiment demonstrates this advantage?

4 comments

We thank the reviewer for their thoughtful assessment and for recognizing the value of our experimental infrastructure and the rigor of our empirical analysis, including paired comparisons, uncertainty estimates, failure analysis, and cost measurements. We also appreciate the reviewer's careful consideration of the limitations surrounding memory and calibration. Below, we address the concerns and questions.

W1 & Q1–Q2: Information Environment and Generalization to Live Forecasting

We thank the reviewer for raising these concerns. We believe it is important to distinguish two questions: (1) whether forecasting capabilities learned in Forecast-Dojo generalize beyond its historical information environment, and (2) whether findings from its controlled evaluation should be expected to transfer to live forecasting. We address the former with new empirical evidence and clarify the latter in light of Forecast-Dojo's design objectives.

(1) Generalizability of the forecasting environment (W1 & Q1).

We agree that quantifying source overlap and retrieval differences against historical live search would be informative. However, such a comparison cannot be reliably reconstructed for our resolved events, as the search results, indices, and webpage contents available at their original forecast dates were not preserved. Querying today's search APIs with historical date filters would not provide a faithful reference. FutureSim [3, Appendix B.5] documents precisely this issue, where Brave Search exposed future outcomes despite an explicit historical cutoff.

We therefore investigate the underlying generalization concern more directly: can forecasting agents trained in Forecast-Dojo improve when deployed in a genuinely different, live information environment?

To answer this, we conducted an additional prospective evaluation on 68 newly resolved Polymarket questions, selected before their outcomes were known. We compared Qwen3-30B-A3B-Thinking-2507 before and after Forecast-Dojo SFT, using live Google Search and webpage retrieval through the Serper API instead of the historical CC-News corpus. We preserved the original prompts, tool interface, inference configurations, and tool budgets, with four independent rollouts per question. All predictions were timestamped and frozen before resolution. The results are shown in the table below.

ModelBrier ↓Acc. (%) ↑Δ Brier (95% CI)Δ Acc. (pp, 95% CI)
Base0.85232.2——
+ Forecast-Dojo SFT0.74639.6−0.106 [−0.196, −0.020]+7.5 [+0.1, +14.8]
Market reference0.53758.1——

Forecast-Dojo SFT achieves a 12.4% relative reduction in Brier score and a 7.5 percentage-point accuracy improvement under live web research. The contemporaneous market reference, never exposed to the agents, remains ahead of both models. We also audited the recorded research trajectories for incidental exposure to third-party prediction-market prices. Excluding affected rollouts changes Brier scores by at most 0.01. These results provide direct evidence that the training benefits of Forecast-Dojo extend beyond the historical information environment in which the training trajectories were collected.

We have incorporated this prospective live-forecasting evaluation into the revised manuscript in Section 5.5 (Table 3), with experimental details in Appendix C.9.

We further quantify the coverage and composition of our historical corpus. The processed CC-News archive contains 18.83M articles from 8,708 source domains, with the largest 10 and 100 domains contributing 17.0% and 48.3% of articles, respectively. Source coverage is sustained over time, with 3,900–5,200 active domains per month across March 2025–May 2026.

(2) Transferability of controlled evaluation findings (W1 & Q2).

We would also like to clarify that live forecasting is a complementary evaluation setting, rather than a reference standard for controlled comparisons. Forecast-Dojo was specifically designed to address several challenges that live forecasting presents for systematic agent evaluation and development.

First, online information availability changes over time. Even when agents begin simultaneously, differences in research trajectories and execution times may expose them to different information states. FutureX (Sec. 3.2.3) explicitly recognizes this issue, noting that delayed evaluations can introduce unfair comparisons as online information evolves. Second, once the information environment changes, past forecasting conditions cannot be reliably recreated, limiting reproducibility and systematic ablations. Third, outcome feedback arrives only after event resolution, slowing iterative evaluation and training.

We address these challenges by fixing forecast checkpoints, accessible information, and tool protocols across agents. Agents independently research and form predictions, but operate under shared historical information boundaries. This enables matched, reproducible comparisons of forecasting performance, longitudinal belief revision, and memory's effects on forecast quality and research cost. The same forecasting interactions can also be replayed for training with immediate outcome-based feedback.

Accordingly, our evaluation findings characterize agent behavior under explicitly controlled and shared information conditions, rather than claiming invariant model rankings or cost effects across arbitrary live-search systems. Their scientific value lies in enabling systematic comparisons without uncontrolled changes in the underlying information environment, not in reproducing identical effect sizes under a different evaluation protocol. Meanwhile, our new prospective experiment provides complementary evidence that the training benefits obtained through Forecast-Dojo can transfer to real-world live forecasting.

W2 & Q3: Scientific Contribution Beyond Existing Forecasting Environments

We would like to clarify that our primary contribution is a unified environment for evaluating and training forecasting agents, rather than a benchmark designed to measure a previously unexplored capability. We develop Forecast-Dojo to make interactive, multi-date forecasting a reproducible experimental and learning process. By fixing forecast checkpoints and information access across agents, we enable controlled comparisons while preserving autonomous research and probabilistic forecasting. The same forecasting states can be replayed for repeated evaluation and training with immediate outcome-based feedback.

Our objective differs from that of existing forecasting environments. Autocast primarily studies historical forecasting, EvolveCast examines belief revision given new evidence, and FutureSim focuses on open-ended adaptation in an evolving world. In contrast, we design Forecast-Dojo around matched, event-centered forecasting episodes, allowing us to compare agents under the same historical conditions and reuse their recorded interactions for learning. We demonstrate both capabilities empirically. Our 12-model benchmark supports systematic evaluation and longitudinal analysis, while SFT on collected research trajectories improves accuracy from 34.8% to 42.8% and Brier score from 0.924 to 0.749 on 230 temporally held-out events. Thus, our training capability is not merely an interface-level design choice, but is demonstrated through measurable improvements on unseen events.

We also demonstrate the scientific value of this design through matched longitudinal experiments (Table 2, Figures 2-3), which allow us to examine the effects of research access and persistent memory on the same event-date pairs. We find that research tools improve forecasting across all 12 models, with larger improvements associated with more newly recorded evidence. In contrast, retaining previous research through a belief notebook reduces research costs by a median of 24% without consistently improving forecast quality. These findings reveal a distinction between acquiring new evidence and preserving prior research: the former is associated with improved predictive quality, whereas the latter more reliably improves research efficiency.


Thank you again for the thoughtful and constructive review. We have carefully addressed your concerns through new prospective live-forecasting evaluations, expanded corpus coverage analyses, and detailed clarifications of Forecast-Dojo's scientific contributions and training utility. We believe these additional results and clarifications strengthen the empirical support and positioning of our work, and we hope they satisfactorily address your concerns and support consideration for acceptance.

AuthorsOpenReview ↗

Additional Prospective Live Forecasting Results

We would like to share an update on our prospective live forecasting evaluation, which addresses the reviewer's concern regarding whether forecasting capabilities learned in Forecast-Dojo generalize beyond its historical information environment.

Our November 9 evaluation now has additional resolved outcomes. Of the questions forecast during the same session on November 9, 2026, 100 questions had resolved by November 16 (included 68 in our earlier response). All predictions were made and frozen before their outcomes were known, with question-selection criteria fixed in advance.

The updated results are:

ModelBrier ↓Acc. (%) ↑Δ Brier [95% CI]Δ Acc. (pp) [95% CI]
Student (base)0.86634.0——
+ Forecast-Dojo SFT0.71343.0−0.153 [−0.230, −0.084]+9.0 [+2.7, +15.4]
Market reference0.54052.5——

The improvement persists on the expanded evaluation set. Forecast-Dojo SFT reduces Brier by 17.7% and improves accuracy by 9.0 percentage points, with both paired 95% confidence intervals excluding zero. While the historical market reference remains stronger, these results provide additional evidence that forecasting capabilities learned through Forecast-Dojo can transfer to a different, live information environment, rather than being limited to its offline news corpus.

We have incorporated these updated results into the revised manuscript in Section 5.5 (Table 3), with the prospective evaluation protocol detailed in Appendix C.9.


We thank the reviewer again for the thoughtful feedback. We would greatly appreciate it if you could let us know whether our rebuttal and the updated manuscript address your concerns. We are also happy to address any remaining questions.

Dear Reviewer jfcG,

Thank you again for your thoughtful and detailed review. We have made substantial efforts to address your concerns by clarifying Forecast-Dojo's scientific contributions and information environment, conducting additional corpus coverage analyses, and providing new prospective live-forecasting results on 100 resolved questions. We have also incorporated these updates into the revised manuscript.

As the discussion period is approaching its end, we would be very grateful if you could review our responses and let us know whether your concerns have been addressed or if any questions remain. We would greatly appreciate the opportunity to discuss your feedback further.

Reviewer_NrsaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Forecast-Dojo, an environment for evaluating and training AI agents that allows for the replay of historical forecasting processes. It integrates settled prediction market events with news data—restricted by date—enabling models to retrieve evidence and update their forecasts at various points in the past, with performance subsequently scored against actual outcomes. The dataset comprises 1,568 events; the paper evaluates 12 models and demonstrates the environment's utility for training through supervised fine-tuning.

Critical strengths
  1. The environment design offers practical utility. It supports repeated evaluation of the same event across multiple historical dates, providing immediate feedback—a feature that facilitates the study of forecast updates and the collection of training trajectories.
  2. The experimental and statistical analysis is robust. Covering 12 models and various tool and memory configurations, the study employs repeated runs and event-level confidence intervals, balancing forecast quality against computational costs.
  3. The presentation of results is measured and cautious. The authors explicitly note that models still underperform relative to historical market forecasts, that memory mechanisms have not consistently improved quality, and that some gains from fine-tuning can be attributed to probability calibration.
Critical weaknesses
  1. The evaluation process is quite similar to existing benchmarks; it is recommended to clarify the unique evaluative value provided by features such as multi-timestamp playback.

  2. The paper partitions the training and evaluation sets based on event time; however, it remains unclear whether related events or semantically similar questions appear across both sets.

  3. The rationale behind current context budget settings—and their subsequent impact on outcomes—requires further validation. With the latest models now supporting context lengths well beyond 64k, what are the implications of such a change in this parameter?

Questions for authors
  1. What unique evaluative value does multi-timestamp replay provide beyond existing forecasting benchmarks?
  2. Do related events or semantically similar questions occur across the training and evaluation splits, and how are they identified and controlled?
  3. What motivates the chosen context budget, and how sensitive are the results to larger context budgets, including budgets beyond 64k?
4 comments

We appreciate the reviewer's careful assessment and recognition of Forecast-Dojo's practical utility, comprehensive evaluation, and balanced presentation of findings. We address the questions regarding the value of multi-timestamp replay, cross-split similarity, and context-window settings below.

W1 & Q1. Unique Value of Multi-Timestamp Replay

We clarify both what distinguishes Forecast-Dojo from existing forecasting benchmarks and why multi-timestamp replay is central to its design.

1. Distinction from existing forecasting benchmarks. We acknowledge that prior work has explored temporal forecasting: Autocast uses historical news and forecasts, EvolveCast studies forecast revisions with newly provided evidence, and FutureSim supports autonomous research in a replayed world. Forecast-Dojo is designed from the outset as a unified environment for both evaluating and training forecasting agents, rather than an evaluation-only benchmark. This objective motivates three key design choices: (i) Real-world forecasting with market references: We use resolved prediction-market events and their historical market beliefs, enabling evaluation against both realized outcomes and contemporaneous market expectations. (ii) Controlled and reproducible research: A fixed, date-restricted offline corpus and shared forecasting checkpoints provide identical information boundaries across agents, enabling fair and reproducible comparisons while preserving autonomous research. (iii) Unified evaluation and training: Both share the same agent–environment interaction interface, allowing collected research trajectories to be evaluated immediately against realized outcomes and reused for learning. Importantly, we directly demonstrate this training capability through SFT (§5.5, Table 3), rather than presenting it merely as a potential application.

2. The value of multi-timestamp replay for evaluation and learning. Our motivation is that forecasting is an ongoing process of updating beliefs as new evidence emerges, rather than a single prediction made in isolation. Each forecast date therefore represents a distinct information-conditioned task, even when the question and eventual outcome remain unchanged.

For evaluation, multi-timestamp replay shifts the focus from isolated predictions to how forecasting agents adapt over time. By evaluating the same event at matched historical checkpoints, we can investigate three key capabilities, each demonstrated by our experiments:

  • Forecast adaptation: Can agents improve their predictions as events unfold? Research-enabled forecasts improve substantially from early to late stages (memory-free Brier: 0.670 → 0.606), while no-tools performance remains nearly unchanged (§5.3, Figure 2(a)).
  • Evidence responsiveness: Do agents translate newly available evidence into better forecasts? We find that larger step-to-step forecasting improvements coincide with intervals containing more new evidence (§5.3, Figure 2(b); Appendix C.7).
  • Memory effectiveness: Does retaining past research improve prediction quality, research efficiency, or both? Our memory-on/off comparisons reveal that persistent memory reduces research costs more consistently than it improves forecasting quality (§5.2, §5.4, Figure 3).

For training, multi-timestamp replay also expands what agents can learn from each event. Rather than yielding a single forecasting example, each resolved event provides multiple research-and-forecast interactions under different information states and stages of uncertainty. Each interaction produces a reusable trajectory with outcome-based feedback, without exposing the outcome during forecasting.

Therefore, Forecast-Dojo is not merely a benchmark for repeated forecasting evaluation, but a unified, replayable environment for understanding and improving how forecasting agents research, update beliefs, and adapt to evolving information.

W2 & Q2. Cross-Split Semantic Similarity and Outcome Leakage

We thank the reviewer for raising this concern. While our temporal split ensures that training and evaluation contain disjoint events, semantic similarity does not necessarily imply outcome leakage. For example, recurring questions about quarterly earnings or weekly statistics may share nearly identical wording but concern different future outcomes. The relevant concern is whether a resolved training question could directly reveal the answer to an evaluation question. We conducted an additional cross-split audit to explicitly examine this possibility.

Audit methodology. We considered all 307,740 train–evaluation event pairs (1,338 training × 230 evaluation). To identify potentially overlapping questions, we combined (1) dense embedding retrieval (Qwen3-Embedding-8B, top-25 per evaluation event), (2) BM25 retrieval (top-25), and (3) exhaustive exact/near-exact matching of normalized titles and outcome options. Their union yielded 8,355 candidate pairs, which we examined using Claude Opus 4.6 as judge.

For each pair, the judge compared the full questions, outcome options, resolution criteria, and relevant dates, without access to realized outcomes. We distinguished three cases: (i) same underlying outcome, where the resolved training answer would deterministically establish the evaluation answer; (ii) semantically related but distinct, where questions share entities, topics, or templates but resolve different events; and (iii) unrelated. Ambiguous cases were flagged separately. Importantly, similar wording, shared entities, or correlated outcomes were insufficient to establish an overlap. The judge had to verify the exact resolution target and answer correspondence.

Audit statisticResult
Total train–evaluation pairs screened307,740
Candidate pairs examined by LLM judge8,355
Semantically related but distinct pairs2,293
Unrelated candidate pairs6,062
Confirmed same-outcome pairs0
Evaluation events with confirmed overlap0 / 230
Unresolved / ambiguous pairs0

The audit reveals substantial semantic similarity but no confirmed outcome overlap. Among the 8,355 candidates, 2,293 pairs share related topics or question structures, yet their resolution targets differ. For example, “Will Alphabet (GOOGL) beat quarterly earnings?” appears identically across splits, but the training question concerns Q4 2025 earnings, whereas the evaluation question concerns Q1 2026, with different EPS thresholds. Likewise, recurring weekly activity questions refer to different measurement windows.

Overall, we found no evidence that resolved training outcomes reveal the answers to any of the 230 evaluation events. Importantly, recurring question types across splits represent distinct future outcomes, allowing Forecast-Dojo to evaluate temporal generalization to new events rather than memorization of previously resolved outcomes.

W3 & Q3. Clarification on Context Windows and Output Budgets

We would like to clarify a distinction between context-window size and output-token limits. As specified in Appendix B.2 (Table 12), the reported 65,536-token limit applies to maximum output tokens per model call (including reasoning tokens), not the context window. We do not impose a uniform 64k context-window restriction.

Specifically, no-tools forecasting allows up to 65,536 output tokens because the entire forecast is generated in a single call. In contrast, tool-enabled agents use a 32,768-token per-call output limit but can perform multiple rounds of research and reasoning (up to 120 tool iterations and 400 tool calls per forecast step).

Furthermore, our execution diagnostics in Appendix B.4 show that context overflow accounts for only 0.31% of memory-free forecasts, indicating that hard context limits are not a widespread bottleneck in our evaluation.


We thank the reviewer again for the insightful feedback. Our clarifications highlight the distinctive evaluation and training capabilities enabled by multi-timestamp replay, while the new cross-split audit provides additional evidence for the integrity of our temporal evaluation. We hope these responses resolve the reviewer's concerns and support a more favorable assessment of our work.

Thanks for the detailed responses and additional experiments. These address my main concerns about the value of multi-timestamp replay and cross-split overlap. I also misread the 65,536-token limit as a context-window restriction; thanks for clarifying that it is an output limit. The new robustness analyses strengthen the paper, and I will maintain my rating of 3.

We sincerely appreciate your careful consideration of our response. We are glad that our clarifications of multi-timestamp replay and additional experiments on cross-split overlap addressed your main concerns. We greatly appreciate your constructive feedback, which has helped us strengthen the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair and Reviewers,

We sincerely thank the AC and reviewers for their thoughtful feedback and discussions. We have conducted extensive additional experiments and revised the manuscript to address the main concerns. Below, we summarize our contributions and key rebuttal findings.

Key Contributions of Our Work

  • Replayable Forecasting Environment: A unified environment combining resolved real-world events, dated news, and fixed forecasting checkpoints, enabling reproducible agent interactions under controlled historical information conditions.
  • Longitudinal Evaluation and Analysis: A systematic evaluation of 12 LLMs across research and memory settings, revealing how evidence acquisition, belief updates, and persistent memory affect forecasting quality and research efficiency.
  • Unified Evaluation and Training Interface: A shared interaction framework that supports both benchmarking and agent learning, enabling reusable research trajectories and immediate outcome-based feedback for different training approaches, including SFT and RL.

Major Rebuttal Discussions & Resolutions

  1. Generalization Beyond the Historical Forecasting Environment (Reviewer jfcG). To examine whether the forecasting capabilities learned in Forecast-Dojo generalize to live forecasting, we conducted a prospective evaluation on 100 questions using live web search instead of our historical corpus. Forecast-Dojo SFT improves accuracy from 34.0% to 43.0% and reduces Brier from 0.866 to 0.713, with both paired 95% CIs excluding zero. These results provide preliminary evidence that training in our replayable environment transfers to real-world live forecasting (§5.5, Appendix C.9).
  2. Scientific contribution beyond existing benchmarks (Reviewer jfcG). We clarified that Forecast-Dojo's contribution lies in controlled, reproducible evaluation and training, rather than reproducing live search. Its matched longitudinal design enables systematic analysis of forecasting adaptation, evidence responsiveness, and memory effectiveness. Our experiments distinguish the benefits of acquiring new evidence from those of retaining prior research (§5.2–5.4).
  3. Training utility and outcome feedback (Reviewer yLXv). We added SFT selection ablations showing that Dojo SFT outperforms Random-SFT (+2.2 pp accuracy [95% CI: +0.4, +4.0]; −0.038 Brier [−0.057, −0.020]). We further demonstrated outcome-based RL training, improving accuracy from 42.6% to 45.8% and Brier from 0.753 to 0.661 beyond tool-use SFT (§5.5, Appendix C.5; additional RL experiment in rebuttal).
  4. Longitudinal robustness and data integrity (Reviewers yLXv and Nrsa). We added outcome-agnostic selection, uniform-date evaluations, temporal controls, and an independent corpus-side evidence index. The longitudinal findings remain robust under these alternatives. Reviewer yLXv explicitly acknowledged that these experiments resolved their main concern and that the longitudinal claims are now supported, raising their rating from 2 to 3. We also conducted cross-split overlap and agent-access leakage audits, finding no confirmed outcome overlap or future-dated agent-accessed articles (§5.6, Appendices A.1, C.6–C.7; cross-split audit in rebuttal).

We would especially appreciate the AC's assessment of the evidence addressing Reviewer jfcG's concerns, which has not yet received follow-up feedback. We respectfully ask that the final assessment consider these new results, the clarified scope of our claims, and the subsequent discussion alongside the original reviews.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.