- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
- The paper presents Forecast-Dojo, a replayable environment for evaluating and training LLM forecasting agents. It pairs 1,568 resolved Polymarket events with 18.8M English CC-News articles assigned day-level dates. Each event is replayed at 3–10 fixed forecast dates, one per time bin, chosen to favor days with large market-price moves and high overall news volume. Every agent therefore faces the same dates and the same corpus cutoff. At each date, the agent can search and read only articles dated on or before that day and run sandboxed Python. In a memory-on mode, a structured belief notebook is carried to the next date. Events are split by time into 1,338 training and 230 evaluation events.
- The contributions are the environment, the dataset, and a study of 12 proprietary and open-weight models (4 rollouts each) under three settings: no tools, tools, and tools plus notebook.
- They fine-tune Qwen3-30B-A3B-Thinking-2507 on Qwen3-235B-A22B-Thinking-2507 trajectories. One trajectory per task is selected by a score combining Brier, accuracy and tool-call count. Fine-tuning raises held-out accuracy from 34.8% to 42.8%. Validation uses paired event-level bootstrap CIs, failure accounting under three treatments of unusable outputs, and a no-tools check for outcome recall.
Critical strengths
-
Useful infrastructure that fills a real gap. Fixed forecast schedules, date-restricted retrieval over a large corpus, sandboxed tools and a temporal train/eval split let different agents be compared on the same dates and corpus cutoff. They also allow trajectories with outcome feedback to be collected without waiting for events to resolve. Live benchmarks cannot be replayed, and per the paper’s comparison (Table 1), few prior benchmarks combine agentic research with a shared multi-date schedule. This weighs heavily in my assessment because the main contribution is infrastructure, and the paper links code and data. Its value therefore does not depend on any single empirical finding.
-
Careful, transparent evaluation. The study covers 12 models × 3 settings × 4 rollouts and uses paired event-level bootstrap CIs. It reports failure rates per model and shows Brier under three treatments of unusable outputs (Tables 13, 24), along with cost and tool use per model. Null and fragile results are reported candidly: the memory null, the sensitivity of the κ correlation, and the selection analysis in C.4. This makes the two most robust claims credible: tools help every model, and every configuration trails the market in Brier and Information-α (all 36 CIs exclude zero). The same transparency is what made the weaker claims below auditable.
Critical weaknesses
- I don't think the two longitudinal claims in the abstract are established yet, and those are what set this benchmark apart from earlier work.
The first issue is that event selection uses the realized outcome in several places (Appendix A.2, Table 5). A day only counts as "active" if the market leg for the eventual winner traded enough, which can exclude events won by thinly traded options such as long shots (190 of the 230 test events are multi-option). The a_Q filter drops events where the market already gave the realized outcome near-certain probability. The C_Q filter requires the market's net movement toward the realized outcome to be at least 20% of its total movement, and events that fail it survive only if their total movement is large (V_Q of at least 0.5). To be fair, Table 6b shows that the nontriviality filters (a_Q and C_Q) removed only 45 of 423 evaluation candidates. But the activity filter removed most candidates (7,590 down to 423), and the paper doesn't separate how much of that comes from measuring activity on the winning leg. On top of this, forecast dates within each time bin are picked mostly by market movement (weight 0.7), and same-day articles are visible to the agent.
Table 27 shows why this matters. The entire early-to-late improvement (0.670 to 0.606) comes from the 175 events where the market's Brier fell. On the other 49 events, memory-free agents don't improve at all (+0.005, CI [-0.027, 0.038]). The tools vs. no-tools contrast is real, but as the authors say themselves, the size of the effect depends on how many such events the filters keep. They acknowledge this in Section 5.3 and Appendix C.4, but the abstract states the finding without the caveat, and there are no results under outcome-agnostic selection.
The second issue is that the claim about gains being largest where more new evidence appears relies on a proxy that depends on agent behavior. The index counts other models' self-reported notebook entries dated within the interval, and the authors note that it can miss evidence no model found. The market improves even in the lowest quartile (-0.051) while the agents barely move, which fits that concern. I also couldn't find any control for the number of days between forecast dates or the distance to close. Longer gaps mechanically allow more dated entries, and you'd also expect bigger Brier changes over longer gaps. The related model-level correlation in Fig. 2(c) (Spearman 0.74 across 12 models) drops to 0.14 under worst-case scoring and is close to zero within models (r = -0.017).
Last, the episodes are short. 163 of the 230 events have only 3 forecast dates, and the median time from first forecast to close is 9 days. For most events the notebook is only carried forward twice, so I'm not sure the memory null says much about longer horizons, where memory should matter more.
If these claims don't hold up, what's left is that tools help, markets are better, and memory saves money. Those are useful results, but they're largely in line with earlier retrieval-augmented forecasting work, and the benchmark's distinctive longitudinal angle wouldn't be validated.
- The training half of the paper isn't really demonstrated yet. The SFT experiment shows that a weak student that rarely calls tools gets better when trained on a stronger model's selected trajectories. It doesn't show what the environment's outcome feedback adds.
The teacher's score on the 230 evaluation events isn't reported, so I can't tell how close the student gets to just running the teacher. There's also no ablation of the selection rule, such as taking a random eligible rollout or selecting on tool count alone. That matters because the selection score weights tool-call count at 0.5, next to 0.7 for Brier and 0.3 for accuracy, and tool calls go from 1.4 to 3.7 per forecast. A lot of the gain may simply be the student learning to use the tools.
In terms of probabilistic quality, the trained student is still around uniform level. Its Brier is 0.749 against 0.771 for uniform, and it is actually worse than uniform on log loss (1.89 vs. 1.74) and on binary events (0.539 vs. 0.500). The authors point out that oracle shrinkage of the base model nearly matches the raw Brier gain (0.752 vs. 0.749). The accuracy gain is real (42.8% vs. 34.8%, with uniform at 22.9%), and shrinkage doesn't change it. Only one training run is reported, at the final checkpoint, and neither RL (marked as possible in Fig. 1) nor on-policy distillation is tried.
Training is half the title and one of the main reasons to build a replayable environment instead of a static dataset. A date-restricted corpus does make it possible to research resolved questions without leakage, which is a real advantage. As it stands, though, the experiment doesn't show that the outcome feedback adds anything beyond imitating a stronger model's tool use.
Questions for authors
-
Outcome-agnostic selection. Table 6b gives counts per stage, but the activity stage is measured on the realized outcome's leg. How many evaluation events would pass outcome-agnostic versions of the activity and nontriviality filters (for example, activity on total event volume, uncertainty from the market's maximum probability, and total movement without direction)? Could you rerun the main analyses on that set: tools vs. no tools, the late-minus-early change, Fig. 2(b), and the market gap? A version with uniformly spaced forecast dates would also help. If the longitudinal results hold up there, most of my first concern goes away.
-
Evidence index. Does the Fig. 2(b) gradient survive controls for days between consecutive forecast dates, days to close, step index, domain and number of outcomes? Does it hold with a corpus-side measure that doesn't depend on what agents found, for example relevant articles dated in the interval that fixed event-specific queries retrieve?
-
Training. Could you report (a) the teacher's scores on the evaluation set under the same protocol, (b) SFT on randomly selected or tool-count-only selected rollouts, (c) a non-oracle recalibration baseline for the base student, with shrinkage fit on training events, and (d) the number of evaluation rollouts or seeds behind Table 3? If outcome-based selection doesn't beat random selection, it's unclear what the outcome feedback contributes to the gain.
-
Leakage in what agents actually read. As I understand Appendix A.2, the content-level leakage check covers the 15 documents the judge retrieves per event at close minus 2 days. Agents read many more than that (about 27 full articles per forecast for GPT-5.6 Sol, Table 16). Did you audit a sample of the articles agents actually scraped for information from after the forecast date? Did you look at the event-dates where agents beat the market by the widest margin? And since Appendix A.1 says the copy kept for a URL crawled more than once isn't necessarily the earliest, how often does the kept version postdate its assigned date?
-
The judge's confident_lean field. The judge prompt says this field is there so you can later check whether SUFFICIENT verdicts track correct calibration, but I couldn't find that analysis. How often does it match the realized outcome? If the match rate is very high, late-step questions may be largely answerable from the corpus, which would change how I read the late-step scores.
-
Accuracy vs. market. The abstract says every model trails the market in accuracy, but paired tests are only reported for Brier and Information-α (Table 21). Could you add paired accuracy CIs (best model 57.69% vs. market 64.55%)?
-
Future models. Models trained on data from after June 2026 may know the evaluation outcomes. Is there a plan to refresh the evaluation split, and will the market-wrong recall check from Appendix B.1 be part of the standard protocol? This seems important given the advantage over live benchmarks argued in Section 1.
7 comments
2. Uniformly spaced forecast dates.
Additional experiment. We also agree that prioritizing dates with large market-belief movements could influence longitudinal measurements. To isolate this effect, we replaced the original movement-prioritized schedule with uniformly spaced forecast dates, selecting the midpoint of each temporal bin while preserving the original forecasting horizon constraints and number-of-steps rule. This yields 227 events (789 forecast dates), on which we evaluated the same six models across all three settings.
Results. The same four findings persist:
(i) Research tools improve forecasting. All six models achieve significant Brier improvements, with a pooled memory-free versus no-tools difference of −0.111 [−0.150, −0.073].
(ii) Longitudinal improvements persist. Memory-free and memory-on forecasts improve significantly from early to late stages (−0.051 [−0.074, −0.028] and −0.055 [−0.079, −0.033], respectively), while no-tools remains nearly unchanged (−0.003 [−0.012, +0.005]).
(iii) The evidence gradient persists. The highest-versus-lowest new-evidence quartile contrast in step-to-step Brier change remains negative for memory-free (−0.033) and memory-on (−0.046), compared with −0.006 for no-tools, reproducing the pattern in Figure 2(b).
(iv) The market gap remains. All 18 configurations continue to trail the market in paired Brier comparisons, with positive gap CIs and negative Information-α. Moreover, the market's Brier is essentially unchanged between the original and uniform-date schedules (paired difference −0.001 [−0.010, +0.009]).
Overall, these two ablations reinforce our central findings: research tools improve forecasting, research-enabled agents improve as events unfold, larger improvements are associated with newly available evidence, and historical market forecasts remain stronger. These patterns persist under outcome-agnostic event selection and uniformly spaced forecast dates, supporting their robustness to both design choices.
The revised manuscript includes the full model-wise comparisons and longitudinal plots in §5.6 and Appendix C.6 (Tables 30–31 and Figure 5).
Clarification on episode length and memory. We would like to clarify the scope of our finding. We do not claim that memory is ineffective for longer-horizon forecasting. As stated in our paper (lines 22–24 and 80–82), our conclusion is specific to the evaluated setting (median horizon of nine days): persistent memory consistently reduces research costs but does not yield consistent improvements in forecast quality. We do not extrapolate our findings to longer horizons, where memory may play a different role.
Q2. Evidence Index: Temporal Controls and Independent Corpus-Side Validation
We conducted two additional analyses to assess whether the evidence gradient in Fig. 2(b) is driven by temporal confounders or agent-dependent evidence measurement.
1. Temporal-controlled analysis. We regress consecutive-step Brier changes on the original leave-one-model-out (LOMO) evidence index across 230 evaluation events (567 forecast transitions). Specifically, we include the calendar-day gap between forecasts, days remaining until market close, and forecast step index as temporal covariates, alongside domain indicators, log₂(number of outcomes), and model fixed effects to account for event and model heterogeneity. We use event-balanced estimation, giving each event equal weight, and compute 95% confidence intervals from 4,000 event-level bootstrap resamples.
| Setting | Unadjusted β | Adjusted β [95% CI] | Adjusted Q4−Q1 [95% CI] |
|---|---|---|---|
| Memory-free | −0.044 | −0.046 [−0.071, −0.022] | −0.069 [−0.107, −0.032] |
| Memory-on | −0.048 | −0.047 [−0.073, −0.024] | −0.068 [−0.103, −0.034] |
| No-tools | +0.001 | +0.003 [−0.005, +0.012] | +0.004 [−0.006, +0.016] |
Here, β measures the association between log(1+evidence index) and consecutive-step Brier change (negative indicates improvement). Q4−Q1 compares adjusted Brier changes between the highest and lowest evidence quartiles.
The gradient remains nearly unchanged after adjustment. For memory-free agents, β changes from −0.044 to −0.046, with a substantial adjusted Q4−Q1 difference of −0.069. Memory-on agents show the same pattern, whereas no-tools agents exhibit no comparable association. The results also persist with event fixed effects (adjusted Q4−Q1 = −0.082 for memory-free and −0.075 for memory-on), nonlinear gap-length controls, and worst-case scoring of unusable forecasts.
2. Independent corpus-side evidence index. To eliminate dependence on agent-recorded evidence, we construct a separate index directly from the dated CC-News corpus. For each event, we reuse fixed queries generated during evidence-sufficiency screening, together with the event title, without access to outcomes, market prices, or agent forecasts. Within each forecast interval, we retrieve candidates using dense (Qwen3-Embedding-8B) and BM25 search (top-200 each per query), then count unique articles with a maximum dense cosine similarity ≥ 0.55 across the fixed queries, independently of agent-read evidence.
The corpus-side index agrees strongly with the original LOMO index (Spearman ρ = 0.70), with 89% of transitions assigned to the same or adjacent evidence quartiles. More importantly, it reproduces the evidence gradient under the same temporal controls:
| Evidence index | Memory-free adjusted Q4−Q1 [95% CI] | Memory-on adjusted Q4−Q1 [95% CI] | No-tools adjusted Q4−Q1 |
|---|---|---|---|
| LOMO notebook | −0.069 [−0.107, −0.032] | −0.068 [−0.103, −0.034] | +0.004 |
| Corpus-side | −0.061 [−0.117, −0.014] | −0.076 [−0.130, −0.030] | +0.005 |
The gradient also persists across alternative cosine thresholds (0.55, 0.60, 0.65).
Together, these results support the robustness of our longitudinal finding. The evidence–forecast improvement association persists after controlling for the proposed confounders and when evidence availability is measured independently of agent notebooks. Moreover, the gradient appears consistently in research-enabled settings but not in no-tools forecasting. While these analyses remain observational, they show that the reported pattern is not solely an artifact of temporal spacing or agents' evidence-recording behavior.
We have incorporated these analyses into the revised manuscript in Section 5.6 and Appendix C.7 (Figure 6 and Table 32).
W2 & Q3. Training Ablations and Forecasting Improvements
We thank the reviewer for these constructive suggestions. To better isolate the contribution of outcome-based feedback to demonstration selection, we conducted additional SFT ablations under identical training and evaluation settings. Random-SFT uniformly selects eligible teacher trajectories, Tool-SFT selects those with the most tool calls, and Dojo SFT uses outcome-guided selection. All three use 3,965 demonstrations, with the selected teacher trajectories averaging 3.26, 4.19, and 3.79 tool calls, respectively. All students use identical training configurations and are evaluated on 230 held-out events (797 event–date pairs; four rollouts each; evaluation seed 0). We additionally report teacher performance and a non-oracle recalibration baseline fitted exclusively on training events.
| Model | Acc. (%) ↑ | Brier ↓ | ΔAcc. vs. Base (pp, 95% CI) | ΔBrier vs. Base (95% CI) |
|---|---|---|---|---|
| Base | 34.8 | 0.924 | — | — |
| Random-SFT | 40.6 | 0.788 | +5.8 [3.0, 8.6] | −0.136 [−0.174, −0.101] |
| Tool-SFT | 41.1 | 0.782 | +6.3 [3.4, 9.3] | −0.143 [−0.184, −0.105] |
| Dojo SFT | 42.8 | 0.749 | +8.1 [5.1, 11.2] | −0.176 [−0.214, −0.139] |
| Teacher | 44.3 | 0.721 | +9.5 [6.7, 12.3] | −0.203 [−0.242, −0.164] |
| Base + recalibration | 34.8 | 0.758 | — | −0.166 [−0.205, −0.129] |
All three SFT strategies substantially improve forecasting over the base model, with outcome-guided selection achieving the best average performance. Dojo SFT improves accuracy by 2.2 and 1.7 percentage points over Random-SFT and Tool-SFT, respectively, while reducing Brier by 0.039 and 0.033. Notably, Dojo SFT also uses fewer tools at evaluation than Tool-SFT (3.7 vs. 4.6 calls per forecast), suggesting that outcome feedback provides a useful selection signal beyond simply favoring more tool calls.
Interpreting the SFT gains. We emphasize that uniform forecasting is a conservative no-information reference, not a worst-case Brier baseline. Brier penalizes overconfident errors, so even models with meaningful predictive ability may score near or worse than uniform. This motivates evaluating both probabilistic quality (Brier score) and outcome selection (accuracy). Dojo SFT improves accuracy by 8.1 percentage points and reduces Brier by approximately 19% relative to the base student, with both paired CIs excluding zero. Its accuracy also substantially exceeds uniform (42.8% vs. 22.9%). Moreover, non-oracle recalibration reduces the base Brier to 0.758 but leaves accuracy unchanged at 34.8%. Thus, confidence adjustment alone cannot explain the accuracy improvements achieved through SFT.
We have incorporated these training ablations, teacher comparisons, and non-oracle recalibration results into the revised manuscript in Section 5.5 (Table 3). Additional details on demonstration selection, training configurations, and calibration diagnostics are provided in Appendix C.5 (Tables 28–29 and Figure 4).
We thank the reviewer for the thoughtful and detailed feedback, and for recognizing the value of our replayable infrastructure, rigorous evaluation, and experimental transparency. Below, we address all the concerns and questions.
W1 & Q1. Outcome-Agnostic Event Selection and Forecast-Date Robustness
We thank the reviewer for raising this important concern. We agree that outcome-dependent event filtering and movement-prioritized forecast-date selection may influence the observed longitudinal findings and warrant direct validation. We therefore conducted two targeted robustness experiments addressing these choices separately. We re-evaluated six representative models (GPT-5.6 Sol, GPT-5.5, Claude Opus 4.8, GLM-5, Kimi K2.5, and MiniMax M2.5), covering proprietary and open-weight models, with four rollouts per forecast date across all three evaluation settings.
1. Outcome-agnostic event selection.
Rationale for the original filter. Forecast-Dojo uses historical Polymarket prices as a proxy for contemporaneous market beliefs, making the reliability of these price-based references important for evaluation. In multi-option markets, high overall trading volume does not necessarily imply sufficient activity on every outcome leg. We therefore originally prioritized activity on the realized-outcome leg, since Information-α directly compares the agent's probability of the realized outcome with the market's corresponding probability, and the market's Brier score also depends on this probability as part of its full distribution. The winner-leg filter was intended to prioritize reliable historical market references, although we recognize that it could affect which events are selected.
Additional experiment. Following the reviewer's suggestion, we reconstructed the evaluation set using outcome-agnostic filters based on total event trading activity, uncertainty measured from the market's maximum outcome probability, and undirected market-belief movement. This yields 268 events (907 forecast dates), retaining 192/230 (83.5%) original evaluation events and introducing 76 newly eligible events. We reused the original forecasts for overlapping events and evaluated the 76 new events under the same protocol.
Results. The four main findings remain consistent:
(i) Research tools improve forecasting. All six models retain significant Brier improvements, with a pooled memory-free versus no-tools difference of −0.103 [−0.139, −0.071], compared with −0.119 on the original six-model evaluation.
(ii) Longitudinal improvements persist. The late-minus-early Brier change is −0.069 [−0.091, −0.048] for memory-free and −0.069 [−0.093, −0.048] for memory-on, while no-tools remains nearly flat (−0.003).
(iii) The evidence gradient persists. On the 76 newly included events, more newly recorded evidence remains associated with larger Brier improvements for memory-free (β = −0.043 [−0.085, −0.009]) and memory-on (β = −0.037 [−0.081, −0.001]), but not no-tools.
(iv) The market gap remains. All 18 model–setting configurations continue to trail the historical market in paired Brier comparisons, with positive gap CIs, and retain negative Information-α.
Importantly, the 76 newly included events alone also show significant pooled improvements from research tools (−0.033 [−0.064, −0.004]) and across forecast stages (memory-free: −0.056 [−0.080, −0.032]). Thus, the main patterns are not supported solely by events shared with the original evaluation set.