- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes DeepForecaster, a forecasting framework combining (i) Proactive Belief-State Management (PBM), which periodically compresses search trajectories into structured belief states and resets the context, and (ii) SEIPH, an outcome-blind five-dimensional rubric that provides segment-level process rewards for RL. The authors train 35B Qwen3.6 models with SFT followed by PPO/GRPO and evaluate them on FutureX, ForecastBench, and Polymarket-Sim. DeepForecaster-PPO achieves the best Brier scores on ForecastBench and Polymarket-Sim and outperforms several much larger frontier models, while ablations suggest that PBM itself improves the base model.
Critical strengths
- Strong empirical results with useful ablations.
- A coherent combination of harness design and training signal.
Critical weaknesses
- Dependence on an LLM-based reward judge. SEIPH is scored by GLM-5.2, but there is limited validation of judge reliability. Since the central RL contribution relies on these process rewards, this makes it difficult to determine how much of the improvement comes from genuinely better forecasting/reasoning versus optimizing for the judge's preferences.
- Limited evaluation scale and disentanglement. The main Polymarket-Sim result uses only 126 resolved questions from a short time period, making it unclear whether the reported gains generalize across different forecasting periods and event distributions. Also, the reported gains from PBM, SFT, and RL are somewhat intertwined. Stronger multi-seed and broader temporal evaluation would make the main claims more robust.
Questions for authors
- How reliable is the GLM-5.2 SEIPH judge? Do you have human agreement measurements or an independent evaluation showing that SEIPH scores correlate with human assessments of forecasting-process quality?
- How robust are the reported improvements across seeds and evaluation periods? In particular, do the PBM and PPO gains persist when evaluating on a larger set of questions spanning substantially longer or different time periods?
Thank you for your review. We provide additional evidence on judge reliability and robustness below.
Response to Q1
We report agreement with human judgments and the relationship between rubric rewards and forecasting performance.
(1) Agreement with human judgments. On 50 manually labeled segments, GLM-5.2’s overall process score achieves Pearson r = 0.732, supporting alignment with human assessments of forecasting-process quality.
Table R1: Agreement with human judgments. MAE is measured on the 0–1 scale.
(2) Relationship with forecasting performance. We evaluate eight model configurations on the same set of questions. Mean rubric reward correlates positively with 1-Brier, with Pearson r = 0.798, 95% CI [0.445, 0.910]. This provides additional evidence that the process quality measured by SEIPH is associated with forecasting performance at the model level.
Table R2: Rubric scores and forecasting performance.
We have added the human agreement results to Appendix D.1 (p. 26), and the relationship with forecasting performance to Appendix D.3 (p. 26).
Response to Q2
We added 179 Polymarket-Sim questions from November 8 to 13, 2026, with three inference runs. Together with the original 126 questions from August 31 to September 5, this gives 305 questions across two evaluation periods more than two months apart.
1. PBM and PPO improvements persist across periods and repeated runs.
Table R3: Forecasting performance across two evaluation periods and the combined dataset.
(1) PBM improves Qwen3.6 in both periods, increasing (1-Brier) by 0.0280 and 0.0241, respectively. PPO achieves the highest (1-Brier) in both periods. (2) On the new questions, all three PPO runs outperform all three runs of the strongest ReAct baseline, DeepSeek-V4-Flash.
2. Our evaluation covers different forecast horizons and prediction dates.
Our original evaluation also includes longer-horizon questions from FutureX and ForecastBench, alongside short-horizon questions from Polymarket. FutureX was evaluated over three weekly rounds, and the original Polymarket-Sim evaluation covered six daily rounds. The new November evaluation adds a separate period, providing further evidence that the gains persist across prediction dates.
We have added the November evaluation and repeated-run results to Appendix C.3 (p. 24). The forecast horizons and original evaluation rounds are described in Appendix B.2 (p. 20).