- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces AlphaBet, an RL framework for event forecasting agents that can actively retrieve evidence on different dates and output probabilities for unresolved events. The research topic is very significant and interesting. The main contributions of this paper are: (i) making evidence acquisition part of the learning loop; (ii) highlighting that decisions can be supervised by one resolved event under different information states; (iii) decomposing the forecasting regret into information acquisition gap and reporting gap and providing the theoretical analysis. Experiments on two benchmarks with a single backbone GLM-4.5-Air demonstrate the effectiveness of AlphaBet. The paper is well-organized and well-written.
Critical strengths
- Well-motivated and practically relevant problem. This paper addresses an important limitation of existing forecasting approaches by incorporating evidence acquisition rather than relying on preassembled evidence.
- Empirical and theoretical analysis. This paper provides experiments to demonstrate AlphaBet's effectiveness, along with theoretical analysis.
Critical weaknesses
- Insufficient evaluation. The authors conducted experiments only on a single backbone, GLP-4.5-Air. It remains unclear whether this method can be generalized to other LLMs.
- Unclear contributions about longitudinal learning. The memory-free setting achieves higher scores than the memory-on setting and single-date setting. However, this can not fully clarify the contribution of longitudinal learning.
Questions for authors
- How does AlphaBet perform on other LLM backbones?
- If the forecast date is randomly sampled rather than a fixed single date, how does this setting perform? This would help clarify the effectiveness of longitudinal supervision.
3 comments
W2. Clarifying longitudinal learning
1. Longitudinal training does not depend on explicit cross-date memory.
We would like to clarify an important distinction between longitudinal training across evolving information states and explicit cross-date memory transfer. In particular, our memory-free variant is itself a longitudinal training method, rather than a non-longitudinal baseline. As formalized in Section 2.1 (lines 107–122), the admissible information corpus expands across forecast dates: Thus, different dates of the same event correspond to temporally related information states, where the policy can acquire different evidence and learn to report the corresponding date-conditional posterior distributions (Eqs. 1-2). Our longitudinal objective (Eq. 5) trains the shared policy across these evolving information states, regardless of whether an explicit notebook is carried forward. Removing cross-date memory does not remove the temporal structure of training.
The contribution of longitudinal supervision is directly examined in Section 4.4 (lines 434–446), where multi-date memory-free training improves Forecast-Dojo Brier from 0.668 (single-date) to 0.626 (multi-date) using the same underlying events. We have also incorporated the additional matched-step comparison into Section 4.4 (lines 440–443) and Appendix C.5 (Table 10, lines 1648–1664) of the updated manuscript, showing that the longitudinal advantage on Forecast-Dojo persists at identical training steps.
2. Clarifying the role of memory-on forecasting.
Our central contribution is agentic RL for evidence acquisition and probability reporting across evolving information states, rather than optimizing explicit cross-date memory mechanisms. As stated in the Introduction (lines 49–50), "Cross-date memory is optional." We include memory-on as an extension to investigate whether carrying a belief notebook provides additional benefits beyond longitudinal training, not as a prerequisite for our framework.
We also discuss the limitations of this extension in the original submission. Specifically, our implementation uses immediate per-date rewards without explicit cross-date credit assignment (Section 3.3, lines 297–300), which we identify as a plausible explanation for the observed results and leave for future work (Section 4.2, lines 409–413).
Therefore, the memory-on results characterize the limitations of an optional memory extension, rather than undermining the contribution of longitudinal learning. Our central claim concerns training forecasting agents across evolving, date-specific information states, whose benefits are demonstrated independently of explicit cross-date memory.
Q2. Randomly sampled forecast dates for single-date training
Our single-date baseline already employs randomly sampled forecast dates, rather than assigning a predetermined date to each event. As described in Section 4.4 (lines 434–438) and Appendix B.1 (lines 1214–1218), we first randomly shuffle the 1,028 training events and evenly assign them to early, middle, and late forecast-position groups. For each event, we then uniformly sample one forecast date from its assigned interval, ensuring balanced coverage across forecasting horizons rather than favoring any particular stage of an event.
The sampled date is subsequently held fixed throughout training, so each event contributes exactly one dated information state. This produces 1,028 training states for single-date training, compared with 3,965 for multi-date training. The corresponding results are already reported in Table 1 and Section 4.4, where multi-date training improves Forecast-Dojo Brier from 0.668 to 0.626.
If the reviewer instead refers to resampling a new forecast date at every training iteration, this would effectively introduce longitudinal supervision into the single-date baseline. Over successive iterations, the same event would be encountered at multiple forecast dates, exposing the policy to different information cutoffs and date-conditional forecasting targets. This is essentially a stochastic way of sampling the dated training states used in our memory-free multi-date setting, provided the sampling distributions are matched. As described in Appendix B.1 (lines 1208–1212), our memory-free training likewise treats individual question–date pairs as independent training inputs. We therefore intentionally fix the randomly selected date for each event, ensuring that the single-date baseline genuinely receives only one dated information state per event and allowing us to distinguish its supervision from multi-date training.
We thank the reviewer again for their constructive feedback. The additional cross-backbone experiment, together with our clarifications on longitudinal supervision, optional cross-date memory, and single-date sampling, further strengthens the empirical support for AlphaBet's generalizability and longitudinal training framework. We hope these results and clarifications address the reviewer's concerns.
We appreciate the reviewer’s positive assessment of our work and recognition of the importance of training evidence acquisition for real-world forecasting. We are also pleased that the reviewer acknowledges the value of our theoretical analysis and empirical results in supporting AlphaBet’s effectiveness. Below, we provide additional evidence and clarifications to address the reviewer’s concerns and questions.
W1 & Q1. Generalization across model backbones
To examine whether AlphaBet's improvements generalize beyond GLM-4.5-Air, we additionally train AlphaBet memory-free on Qwen3-30B-A3B-Thinking-2507, a smaller model with different search and reasoning capabilities. Given the limited rebuttal timeframe and the computational cost of large-scale agentic RL, we focus on our main memory-free variant.
The original Qwen3 model rarely uses research tools (approximately 1.4–1.7 calls per forecast). We therefore first perform tool-use SFT using trajectories generated by Qwen3-235B-A22B-Thinking-2507 on earlier, temporally disjoint forecasting questions (June–July 2025). These SFT questions precede our August 2025–February 2026 RL training period and March–May 2026 evaluation period. This initialization increases average tool calls to approximately 3.4 per forecast, establishing basic research capabilities before applying the same AlphaBet memory-free RL training procedure as GLM-4.5-Air with the negative Brier reward.
AlphaBet RL delivers additional forecasting gains on this second backbone. Compared with the SFT initialization, RL reduces Brier from 0.753 to 0.661 and improves accuracy from 42.6% to 45.8%. It also achieves better forecasting performance than the BLF harness baseline (0.661 vs. 0.690 Brier). These results provide further empirical evidence that AlphaBet's agentic RL training benefits extend beyond GLM-4.5-Air, including to a model with substantially weaker initial tool-use capabilities.