acceptodds
Under review as a conference paper at ICLR 2027

AlphaBet: Reinforcement Learning for LLM Event Forecasting Agents

Abstract

Forecasting an unresolved event requires finding relevant evidence and deciding how strongly it supports each outcome. Yet outcome-based training often starts from preassembled evidence at a single prediction date, leaving evidence acquisition outside the learning loop. We introduce **AlphaBet**, a reinforcement learning framework that trains a shared LLM policy to search, reason, and report probabilities across multiple pre-resolution dates. Each dated forecast receives proper-scoring feedback from the eventual outcome, allowing one resolved event to supervise decisions under different information states. We decompose forecasting regret into gaps from evidence acquisition and probability reporting, and show that GRPO standard-deviation normalization can distort the reporting target under multi-class proper scores. On GLM-4.5-Air, **AlphaBet** reduces Brier from 0.731 to 0.626 on Forecast-Dojo and from 0.612 to 0.528 on FutureX. Controlled ablations attribute these gains to longitudinal training and improvements in both acquisition and reporting, while showing that GRPO standard-deviation normalization harms probabilistic forecasting while accuracy remains comparable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, Reinforcement learning, Event forecasting

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_A4xhOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces AlphaBet, an RL framework for event forecasting agents that can actively retrieve evidence on different dates and output probabilities for unresolved events. The research topic is very significant and interesting. The main contributions of this paper are: (i) making evidence acquisition part of the learning loop; (ii) highlighting that decisions can be supervised by one resolved event under different information states; (iii) decomposing the forecasting regret into information acquisition gap and reporting gap and providing the theoretical analysis. Experiments on two benchmarks with a single backbone GLM-4.5-Air demonstrate the effectiveness of AlphaBet. The paper is well-organized and well-written.

Critical strengths
  • Well-motivated and practically relevant problem. This paper addresses an important limitation of existing forecasting approaches by incorporating evidence acquisition rather than relying on preassembled evidence.
  • Empirical and theoretical analysis. This paper provides experiments to demonstrate AlphaBet's effectiveness, along with theoretical analysis.
Critical weaknesses
  • Insufficient evaluation. The authors conducted experiments only on a single backbone, GLP-4.5-Air. It remains unclear whether this method can be generalized to other LLMs.
  • Unclear contributions about longitudinal learning. The memory-free setting achieves higher scores than the memory-on setting and single-date setting. However, this can not fully clarify the contribution of longitudinal learning.
Questions for authors
  1. How does AlphaBet perform on other LLM backbones?
  2. If the forecast date is randomly sampled rather than a fixed single date, how does this setting perform? This would help clarify the effectiveness of longitudinal supervision.
3 comments

We appreciate the reviewer’s positive assessment of our work and recognition of the importance of training evidence acquisition for real-world forecasting. We are also pleased that the reviewer acknowledges the value of our theoretical analysis and empirical results in supporting AlphaBet’s effectiveness. Below, we provide additional evidence and clarifications to address the reviewer’s concerns and questions.

W1 & Q1. Generalization across model backbones

To examine whether AlphaBet's improvements generalize beyond GLM-4.5-Air, we additionally train AlphaBet memory-free on Qwen3-30B-A3B-Thinking-2507, a smaller model with different search and reasoning capabilities. Given the limited rebuttal timeframe and the computational cost of large-scale agentic RL, we focus on our main memory-free variant.

The original Qwen3 model rarely uses research tools (approximately 1.4–1.7 calls per forecast). We therefore first perform tool-use SFT using trajectories generated by Qwen3-235B-A22B-Thinking-2507 on earlier, temporally disjoint forecasting questions (June–July 2025). These SFT questions precede our August 2025–February 2026 RL training period and March–May 2026 evaluation period. This initialization increases average tool calls to approximately 3.4 per forecast, establishing basic research capabilities before applying the same AlphaBet memory-free RL training procedure as GLM-4.5-Air with the negative Brier reward.

ModelDojo Acc.(%)↑Dojo Brier↓Dojo Info-α↑
Qwen3-30B-A3B-Thinking-250734.80.924−1.728
+ Tool-use SFT42.60.753−0.892
+ Tool-use SFT + BLF42.80.690−0.547
+ Tool-use SFT + AlphaBet (memory-free)45.80.661−0.516

AlphaBet RL delivers additional forecasting gains on this second backbone. Compared with the SFT initialization, RL reduces Brier from 0.753 to 0.661 and improves accuracy from 42.6% to 45.8%. It also achieves better forecasting performance than the BLF harness baseline (0.661 vs. 0.690 Brier). These results provide further empirical evidence that AlphaBet's agentic RL training benefits extend beyond GLM-4.5-Air, including to a model with substantially weaker initial tool-use capabilities.

W2. Clarifying longitudinal learning

1. Longitudinal training does not depend on explicit cross-date memory.

We would like to clarify an important distinction between longitudinal training across evolving information states and explicit cross-date memory transfer. In particular, our memory-free variant is itself a longitudinal training method, rather than a non-longitudinal baseline. As formalized in Section 2.1 (lines 107–122), the admissible information corpus expands across forecast dates: I≤τ1⊆I≤τ2⊆⋯⊆I≤τT.I_{\leq \tau_1} \subseteq I_{\leq \tau_2} \subseteq \cdots \subseteq I_{\leq \tau_T}. Thus, different dates of the same event correspond to temporally related information states, where the policy can acquire different evidence and learn to report the corresponding date-conditional posterior distributions (Eqs. 1-2). Our longitudinal objective (Eq. 5) trains the shared policy across these evolving information states, regardless of whether an explicit notebook is carried forward. Removing cross-date memory does not remove the temporal structure of training.

The contribution of longitudinal supervision is directly examined in Section 4.4 (lines 434–446), where multi-date memory-free training improves Forecast-Dojo Brier from 0.668 (single-date) to 0.626 (multi-date) using the same underlying events. We have also incorporated the additional matched-step comparison into Section 4.4 (lines 440–443) and Appendix C.5 (Table 10, lines 1648–1664) of the updated manuscript, showing that the longitudinal advantage on Forecast-Dojo persists at identical training steps.

2. Clarifying the role of memory-on forecasting.

Our central contribution is agentic RL for evidence acquisition and probability reporting across evolving information states, rather than optimizing explicit cross-date memory mechanisms. As stated in the Introduction (lines 49–50), "Cross-date memory is optional." We include memory-on as an extension to investigate whether carrying a belief notebook provides additional benefits beyond longitudinal training, not as a prerequisite for our framework.

We also discuss the limitations of this extension in the original submission. Specifically, our implementation uses immediate per-date rewards without explicit cross-date credit assignment (Section 3.3, lines 297–300), which we identify as a plausible explanation for the observed results and leave for future work (Section 4.2, lines 409–413).

Therefore, the memory-on results characterize the limitations of an optional memory extension, rather than undermining the contribution of longitudinal learning. Our central claim concerns training forecasting agents across evolving, date-specific information states, whose benefits are demonstrated independently of explicit cross-date memory.

Q2. Randomly sampled forecast dates for single-date training

Our single-date baseline already employs randomly sampled forecast dates, rather than assigning a predetermined date to each event. As described in Section 4.4 (lines 434–438) and Appendix B.1 (lines 1214–1218), we first randomly shuffle the 1,028 training events and evenly assign them to early, middle, and late forecast-position groups. For each event, we then uniformly sample one forecast date from its assigned interval, ensuring balanced coverage across forecasting horizons rather than favoring any particular stage of an event.

The sampled date is subsequently held fixed throughout training, so each event contributes exactly one dated information state. This produces 1,028 training states for single-date training, compared with 3,965 for multi-date training. The corresponding results are already reported in Table 1 and Section 4.4, where multi-date training improves Forecast-Dojo Brier from 0.668 to 0.626.

If the reviewer instead refers to resampling a new forecast date at every training iteration, this would effectively introduce longitudinal supervision into the single-date baseline. Over successive iterations, the same event would be encountered at multiple forecast dates, exposing the policy to different information cutoffs and date-conditional forecasting targets. This is essentially a stochastic way of sampling the dated training states used in our memory-free multi-date setting, provided the sampling distributions are matched. As described in Appendix B.1 (lines 1208–1212), our memory-free training likewise treats individual question–date pairs as independent training inputs. We therefore intentionally fix the randomly selected date for each event, ensuring that the single-date baseline genuinely receives only one dated information state per event and allowing us to distinguish its supervision from multi-date training.


We thank the reviewer again for their constructive feedback. The additional cross-backbone experiment, together with our clarifications on longitudinal supervision, optional cross-date memory, and single-date sampling, further strengthens the empirical support for AlphaBet's generalizability and longitudinal training framework. We hope these results and clarifications address the reviewer's concerns.

Reviewer_7APGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AlphaBet, a reinforcement learning framework that trains an LLM to search for evidence and predict how real-world events will turn out. It revisits each event at several dates, using only the information available at the time, and scores each prediction against the final outcome. Training uses GRPO without standard-deviation normalization, motivated by an analysis showing that normalization can distort the probability-reporting objective. The paper also separates forecasting errors into gaps in evidence acquisition and probability reporting. On Forecast-Dojo and FutureX, AlphaBet improves both accuracy and Brier score. Further experiments suggest that these gains come from both finding more useful evidence and making better predictions from the same evidence. Multi-date training outperforms single-date training, although carrying a notebook across dates does not provide further gains.

Critical strengths
  • The training objective is well suited to probabilistic forecasting. The choice to remove GRPO’s standard-deviation normalization is clearly motivated, and the theoretical analysis helps explain why this matters for learning meaningful probabilities.

  • The evidence–reader swap experiments help explain where the gains come from. The normalization experiments are also informative: they show that probability quality can deteriorate even when accuracy remains similar.

Critical weaknesses
  • All training experiments use GLM-4.5-Air, leaving it unclear whether the gains would hold for models with different search and reasoning capabilities. The reported bootstrap confidence intervals measure variation across evaluation examples for the selected checkpoints, rather than variation across independent training runs. They therefore do not establish how consistently the gains can be reproduced.

  • The OpenForecaster comparison differs in training data, rewards, and answer format, so it does not cleanly isolate the benefit of training the search process. The performance gap may reflect several differences between the training setups.

Questions for authors

Were the main results reproduced across independent training seeds? If so, how much did the gains vary? The reported bootstrap intervals capture variation across evaluation examples, but do not show training stability.

1 comment

We appreciate the reviewer's careful assessment and encouraging evaluation of our work. In particular, we are glad that the reviewer recognizes the consistent forecasting gains across both benchmarks and the value of our controlled evidence–reader comparisons and reward-normalization analyses. Below, we address the reviewer's concerns and questions.

W1. Generalization across model backbones

To address the cross-backbone generalization concern, we additionally train AlphaBet on Qwen3-30B-A3B-Thinking-2507, a substantially smaller model with different search and reasoning capabilities. The results show that AlphaBet RL further reduces Forecast-Dojo Brier from 0.753 to 0.661 and improves accuracy from 42.6% to 45.8% over the tool-use SFT initialization, providing additional evidence that our training gains extend beyond GLM-4.5-Air. Please refer to our response to Reviewer A4xh, W1, for the detailed experimental setup and results.

W1 & Q1. Reproducibility across independent training runs

To examine whether our improvements are reproducible across training runs, we additionally train AlphaBet memory-free with three independent random seeds (42, 2026, and 0). Given the substantial computational cost of agentic RL training with GLM-4.5-Air (106B-A12B), we prioritize this evaluation for our main memory-free variant, which achieves the strongest overall performance in our submission.

Model / Training SeedDojo Acc. ↑Dojo Brier ↓Dojo −Info-α ↓FutureX Acc. ↑FutureX Brier ↓
GLM-4.5-Air (Base)41.60.7310.72651.30.612
AlphaBet (Seed 42, original)50.50.6260.39957.90.528
AlphaBet (Seed 2026)49.80.6290.40557.70.533
AlphaBet (Seed 0)50.80.6250.39657.30.525
Mean ± Std.50.4 ± 0.50.627 ± 0.0020.400 ± 0.00557.6 ± 0.30.529 ± 0.004

Standard deviations are computed across the three independent training runs.

The improvements are highly consistent across training seeds. Forecast-Dojo Brier ranges narrowly from 0.625 to 0.629, while FutureX Brier ranges from 0.525 to 0.533. All three independently trained models substantially outperform the GLM-4.5-Air base model on both benchmarks. These results provide additional empirical evidence that AlphaBet's forecasting gains are reproducible across training runs, rather than being specific to a favorable random seed.

W2. OpenForecaster comparison and evidence-acquisition training

We would like to clarify that our OpenForecaster comparison is intended as a method-level baseline rather than a controlled isolation of evidence-acquisition training. OpenForecaster primarily focuses on scaling forecasting RL through synthetic question generation and preassembled evidence. We therefore preserve its defining training methodology, including its original reward and answer representation, rather than replacing these components with AlphaBet's training choices, which would substantially change the method being reproduced.

To more directly address the reviewer's concern, we additionally conduct a controlled ablation comparing preassembled-evidence RL and agentic forecasting RL. Both approaches use the same backbone, training questions and dates, reward, answer format, GRPO settings, and evaluation protocol, with the key difference being whether evidence is preassembled offline or actively acquired through policy-controlled research during training. At the same training step (103), agentic RL reduces Forecast-Dojo Brier from 0.665 to 0.626 (paired 95% CI for the reduction: [0.021, 0.057]) and improves accuracy from 44.8% to 50.5%. On FutureX, Brier decreases from 0.551 to 0.528, while accuracy increases from 53.9% to 57.9%. These results provide direct empirical evidence that training the agentic evidence-acquisition process yields additional forecasting gains beyond preassembled-evidence training, even after controlling for the major differences in the original OpenForecaster comparison.

We have incorporated this controlled ablation into Table 1 and Section 4.2 (lines 378–385), with detailed experimental settings provided in Appendix C.5 (lines 1666–1675) of the updated manuscript. Please also refer to our response to Reviewer fCa2, W2, for the full experimental setup and results.


We appreciate the reviewer's constructive feedback. The additional cross-backbone experiments, independent training-seed replications, and controlled evidence-acquisition ablation provide stronger empirical support for AlphaBet's generalizability, reproducibility, and the benefits of agentic RL training. We hope these new results and clarifications address the reviewer's concerns and support a favorable assessment of our work.

Reviewer_fCa2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AlphaBet trains an agentic/tool use LLM to forecast event outcomes from evidence available at multiple historical dates. It uses Brier rewards and GRPO without standard dev normalization, comparing independent forecasts with a notebook-based memory variant. Contributions include multi-date training, a decomposition of forecasting regret into acquisition and reporting terms, and a multiclass analysis of normalization-induced probability distortion. Results are reported through Forecast-Dojo and an adaptation of FutureX thats offline. There are also ablations of training dates, normalization, and evidence collection. The strongest, memory-free variant improves Brier scores over the same backbone from 0.731 -> 0.626 and 0.612 -> 0.528, on the two benchmarks.

Critical strengths
  • There are clearly some improvements over the same backbone with the methodology. Gains across two forecasting datasets are presented and supported by disjoint events evaluation and dated evidence access.
  • Informative diagnostic experiments. Crossing evidence collectors with different readers provides evidence that training improves both retrieved evidence and its use. The normalization ablations also examine probability quality and invalid outputs, strengthening the interpretation of the reported gains.
Critical weaknesses
  • The presentation of the benefits of multi-date training is a bit murky, I would suggest that it needs a clearer compute-matched comparison. Section 4.4 matches training budgets per step, but Appendix C.5 reports validation-selected checkpoints with different update counts. Previous literature would suggest that such multi-date contribution is particularly important to establish. This could also help clarify how much of the gain comes from longitudinal supervision.
  • I think the presentation of baselines could be presented in a clearer way. For example, the OpenForecaster comparison seems to alter the training data, reward, and output representation, which makes it difficult to attribute the difference specifically to grokking evidence acquisition.
Questions for authors
  • How many FutureX categorical reports contribute to each model's Brier score? Does the improvement hold when all 331 categorical questions are scored using the same treatment of invalid outputs?
  • [not a clarification request] Have the authors considered multi-agent forecasting, for example by assigning agents different subquestions or having them make intermediate predictions?
3 comments

We thank the reviewer for the time and effort devoted to evaluating our work. We appreciate their recognition of AlphaBet's improvements across two forecasting benchmarks, as well as the informative evidence–reader swap experiments and GRPO normalization ablations that support our findings. Below, we address the reviewer's concerns with additional experiments and clarifications.

W1. Compute-matched comparison of single-date and multi-date training

1. Clarifying the original comparison protocol.

Our original comparison uses the same 1,028 training events, model initialization, reward, optimizer, and rollout configuration for both single-date and multi-date training (Section 4.4, lines 434–439). The key distinction is that multi-date training provides 3,965 event–date information states, whereas single-date training retains only 1,028. This difference is inherent to longitudinal supervision: multiple forecast dates allow the same event to provide training signals under different information states. We intentionally preserve the same underlying event pool rather than artificially matching the number of distinct event–date samples. Doing so would require introducing additional events into single-date training, thereby changing the underlying training data and introducing an additional confounding factor.

Given this inherent difference, we use the same maximum training-step budget and validation-based checkpoint-selection protocol, allowing each method to select its best validation checkpoint within the shared budget. This is intentional: under the same rollout budget, the smaller single-date training set is revisited more frequently, potentially leading to earlier validation saturation, while multi-date training can continue learning from a larger set of distinct information states. Our original comparison therefore evaluates the best forecasting performance achieved by each approach under a common training-budget constraint, rather than requiring the selected checkpoints to occur at identical training steps.

2. Additional matched-update comparison.

To directly address the reviewer's concern, we additionally evaluate both methods at identical training steps, with matched rollout-group and optimization budgets. Specifically, we compare the models at step 83 (the single-date validation-selected checkpoint) and step 103 (the multi-date validation-selected checkpoint). The results are shown below, with the validation-selected checkpoints highlighted in bold.

StepModelDojo Acc. ↑Dojo Brier ↓Dojo −Info-α ↓FutureX Acc. ↑FutureX Brier ↓
83Single-date (selected)47.20.6680.50753.20.540
83Multi-date48.50.6440.46954.80.539
103Single-date46.50.6670.58852.10.555
103Multi-date (selected)50.50.6260.39957.90.528

The matched-step results consistently favor multi-date training on Forecast-Dojo. Multi-date reduces Brier by 0.024 at step 83 (0.668 → 0.644) and 0.041 at step 103 (0.667 → 0.626), with corresponding improvements in accuracy and −Info-α. On FutureX, multi-date also improves accuracy at both checkpoints, while Brier is nearly identical at step 83 and improves at step 103. These results demonstrate that the benefits of longitudinal supervision persist at matched optimization steps, rather than arising solely from differences in validation-selected checkpoint timing.

We have incorporated this comparison into Section 4.4 (lines 440–443) and Table 10 in Appendix C.5 (lines 1648–1664) of the updated manuscript.

W2. OpenForecaster comparison and the contribution of agentic evidence acquisition

1. Clarifying the OpenForecaster comparison.

Our OpenForecaster comparison is intended as a method-level baseline rather than a controlled isolation of evidence-acquisition training. OpenForecaster's central contribution lies in scaling forecasting RL through large-scale synthetic question generation and offline preassembled evidence. Accordingly, we preserve its defining training methodology, including synthetic training questions, fixed retrieval contexts, and its original reward and answer representation. Replacing these components with AlphaBet's training choices would substantially change the methodology being reproduced, rather than provide a faithful comparison of the existing approaches.

2. Additional controlled ablation: Preassembled-evidence vs. agentic forecasting RL training.

To directly examine the contribution of training evidence acquisition, we conduct an additional controlled ablation comparing two forecasting RL training paradigms: learning to forecast from preassembled evidence versus jointly learning evidence acquisition and probability reporting through an agentic research loop.

Following the preassembled-evidence training paradigm of OpenForecaster, we use a frozen GLM-4.5-Air to generate search queries, retrieve relevant news evidence in advance using the same retrieval infrastructure as AlphaBet, and train the model to forecast from this fixed evidence without interactive tool use. In contrast, AlphaBet learns to actively acquire evidence during RL training. Both approaches use identical training questions, forecast dates, backbone, reward, answer format, and GRPO settings, and are compared at the same training step (103) under the same agentic evaluation protocol.

Model (Step 103)Dojo Acc. ↑Dojo Brier ↓Dojo −Info-α ↓FutureX Acc. ↑FutureX Brier ↓
AlphaBet (Preassembled-Evidence RL)44.80.6650.51153.90.551
AlphaBet (Memory-Free Agentic RL)50.50.6260.39957.90.528

The results consistently favor agentic RL training. On Forecast-Dojo, agentic training reduces Brier by 0.039 (paired 95% CI for the reduction: [0.021, 0.057]) and improves accuracy by 5.7 percentage points. On FutureX, it improves accuracy by 4.0 percentage points and reduces Brier by 0.023. These results provide direct evidence that training with interactive evidence acquisition yields additional forecasting gains over preassembled-evidence RL under otherwise matched training and evaluation settings.

We have incorporated this ablation into Table 1 and Section 4.2 (lines 378–385), with implementation details in Appendix C.5 (lines 1648–1675) of the updated manuscript.

Q1. FutureX categorical report counts and invalid-output treatment

To address this question, we additionally recompute Brier scores over all 331 categorical FutureX questions (138 binary and 193 multiple-choice) under a consistent treatment of invalid or missing probability reports. Specifically, we consider two alternatives: (1) assigning a uniform distribution over the KK outcomes (Brier =1−1/K= 1-1/K), and (2) assigning the worst-case Brier loss of 2.02.0, consistent with our Forecast-Dojo evaluation protocol. We also report the number of valid categorical reports contributing to the original Brier scores.

ModelValid / Attempted ReportsOriginal Brier ↓Invalid → Uniform ↓Invalid → 2.0 ↓
GLM-4.5-Air1,309 / 1,3240.6120.6140.628
OpenForecaster1,231 / 1,3240.5830.5900.634
AlphaBet single-date1,247 / 1,3240.5400.5520.625
AlphaBet memory-free1,312 / 1,3240.5280.5300.541

The improvement remains robust under both invalid-output treatments. AlphaBet memory-free achieves the highest valid-report rate (99.1%) and the lowest Brier score under both uniform imputation (0.530) and worst-case penalization (0.541). These results show that AlphaBet's FutureX improvements persist under a common invalid-output treatment, rather than being an artifact of excluding invalid probability reports.

Q2. Multi-agent forecasting

Multi-agent forecasting is a promising direction, particularly for decomposing forecasting questions into specialized research tasks and aggregating intermediate predictions. However, our work focuses on training a single shared forecasting policy through outcome-based RL, jointly optimizing evidence acquisition and probability reporting across evolving information states (Section 3.1, lines 141–158).

Extending this framework to multi-agent forecasting would introduce a distinct training problem, involving agent coordination, communication, aggregation of intermediate predictions, and credit assignment across agents. These additional design and optimization challenges are beyond the scope of our current study, which aims to establish the effectiveness of agentic RL training within a single-policy framework. We view multi-agent forecasting and its associated RL training formulation as an interesting direction for future work.


We thank the reviewer again for their constructive feedback. The additional matched-update comparisons, controlled evidence-acquisition training ablation, and FutureX evaluation robustness analysis further strengthen the empirical support for AlphaBet's longitudinal supervision and agentic RL training. We hope these new results and clarifications address the reviewer's concerns and support a favorable assessment of the paper.

Reviewer_YAqiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes an RL framework for agent event forecasting. It trains agents to search, gather evidence, reason, and produce probabilistic forecasts across multiple dates before an event resolves. The framework uses the eventual outcome to provide rewards and adopts GRPO to preserve probability calibration. The authors also introduce an optional belief notebook for cross-date information transfer. Experiments on Forecast-Dojo and FutureX demonstrate improvements over the base model and forecasting baselines.

Critical strengths
  1. Unlike approaches that optimize forecasting from pre-retrieved evidence, this work jointly trains evidence acquisition and probability reporting. Using multiple dated information states provides a practical way to expand outcome-based RL supervision.

  2. Strong theoretical motivation and empirical validation. The paper provides a principled decomposition of forecasting regret into information acquisition and reporting gaps, and analyzes how GRPO reward normalization can distort proper-scoring objectives. The experiments include meaningful baselines and controlled ablations that support the effectiveness of the proposed training framework.

Critical weaknesses
  1. The first concern is that, the current reward design encourages forecasting results at each date close to the ground truth. However, this is not how events develop in real-world. For human markets, the probability towards the final outcome can be very low at the beginning and increase by time.

  2. Another concern is that the usage of date or temporal relationship is not fully explored in this work. For memory-free version, date serves to provide diverse training samples (with different evidence cut-offs). For memory-on version, memory are supposed to provide useful information to guide the forecasting, i.e. how to narrow down the search scope and adjust the forecasting properly. However, the model trained with memory-on is worse than memory-free, which is quite counter-intuitive.

Questions for authors

For the 1st weakness, is it possible to refer the human market data to redesign the reward signal?

For the 2nd weakness, the paper exposes a research question that it doesn't aim to solve: how can forecasting agents use temporal relationship to improve forecasting result? In current training design, memory is harming the performance. I would suggest the authors to reframe the paper.

4 comments

We thank the reviewer for the time and effort invested in evaluating our work and for their constructive feedback. We appreciate their recognition of AlphaBet's contributions, particularly the joint training of evidence acquisition and probability reporting, as well as the theoretical analysis and comprehensive empirical validation. Below, we address the reviewer's concerns and questions.

W1 & Q1. Outcome-based rewards and time-varying forecasting probabilities

We would like to clarify an important distinction in our reward formulation: the realized outcome is used to evaluate each forecast, but it is not the probability target that our policy is encouraged to report at every date. Instead, our objective encourages the policy to report the probability distribution supported by the information available at each forecast date. The reviewer's example—where an event initially has a low probability of occurring and becomes increasingly likely as new evidence arrives—is precisely the type of evolving uncertainty that motivates our longitudinal forecasting framework.

We explicitly formalize this distinction in Section 2.1, Eqs. 1-2. For each forecast date τt\tau_t, we define the acquired-context posterior as

qQ,tH(k)=Pr⁡(YQ=k∣Q,τt,Ht).q^H_{Q,t}(k) = \Pr(Y_Q = k \mid Q, \tau_t, H_t).

Although all forecast dates share the same eventual outcome YQY_Q, the available evidence HtH_t changes over time, and thus the optimal forecast distribution qQ,tHq^H_{Q,t} can differ across dates. As established by the strict properness of our scoring rule (Section 2.2, Eq. 3), the expected reward is uniquely maximized by this date-conditional posterior, rather than by assigning probability one to the eventual outcome.

To illustrate, consider an event that eventually occurs but has only a 10% probability of occurring given the information available at an earlier date (latent ground truth posterior). Under our negative Brier reward, reporting a 10% probability yields an expected reward of

0.1(−1.62)+0.9(−0.02)=−0.18,0.1(-1.62) + 0.9(-0.02) = \mathbf{-0.18},

whereas reporting 100% yields

0.1(0)+0.9(−2)=−1.80.0.1(0) + 0.9(-2) = \mathbf{-1.80}.

Thus, prematurely assigning 100% probability to the eventual outcome is penalized in expectation, rather than encouraged by our objective. If subsequent evidence raises the conditional probability to 80%, the same reward then favors reporting 80%. The realized outcome remains unchanged. What changes is the information available to the forecaster and, consequently, the optimal probability report.

This property is further established in Proposition 1 (Eq. 7), which defines the reporting gap relative to the date-conditional posterior qQ,tHq^H_{Q,t}, rather than the realized outcome. We also address this point explicitly in Appendix A.2 (lines 853–857), "Why one resolved outcome can supervise every forecast date," explaining that a single realized outcome can supervise multiple dated forecasts without imposing the same target distribution across dates.

Human-market probabilities as training targets. We would like to highlight that market probabilities are estimates of the underlying conditional outcome distribution, rather than ground-truth probabilities. They may contain estimation noise and systematic biases. In contrast, our outcome-based reward is a strictly proper scoring rule, whose expected value is uniquely maximized by the true date-conditional posterior. Specifically, for the negative Brier score, reporting a market distribution mtm_t instead of the true conditional posterior qtq_t (under the same information set) incurs an expected score gap of

= \|m_t-q_t\|_2^2 \geq 0.$$ Thus, directly supervising the model to match market probabilities would introduce an additional proxy target that is not guaranteed to coincide with the optimal distribution. **Our current formulation avoids this dependence on potentially noisy market-probability targets while already capturing the time-varying forecasting behavior described in the review.** Market probabilities remain useful as evaluation references, as reflected in our Information Alpha metric, but are not necessary for defining the theoretically optimal forecasting objective.

W2 & Q2. Longitudinal forecasting and the role of cross-date memory

1. Clarifying how temporal information is incorporated into longitudinal training.

We would like to clarify an important distinction between longitudinal training across evolving information states and explicit cross-date memory transfer. Our framework focuses on the former, while the latter is explored as an optional extension (lines 49–50).

In particular, memory-free forecasting is not simply an augmentation of unrelated training samples. As formalized in Section 2.1 (lines 107–122), the information available to the policy evolves with the forecast date. Specifically, the admissible information corpus expands over time:

I≤τ1⊆I≤τ2⊆⋯⊆I≤τT.I_{\leq \tau_1} \subseteq I_{\leq \tau_2} \subseteq \cdots \subseteq I_{\leq \tau_T}.

Thus, the forecast date directly determines which evidence the policy can search and reason over. Different dates of the same event correspond to temporally related information states, rather than independent forecasting questions. Although these dated tasks share the same eventual outcome, their available information differs, and consequently their optimal date-conditional posterior distributions can differ (Eqs. 1-2). Our longitudinal objective (Eq. 5) trains the shared policy to acquire evidence and report probabilities across these evolving information states.

Removing cross-date memory does not remove this temporal structure. As described in Section 3.1 (lines 160–164), the admissible corpus continues to expand across forecast dates regardless of whether a belief notebook is carried forward. Memory-free forecasting therefore retains the longitudinal training formulation while allowing each dated forecast to begin independently.

2. Clarifying our contribution, scope, and memory-on results.

Our central research question, as stated in the Introduction (lines 42–43), is how outcome feedback can train evidence-seeking and probability-reporting behavior across changing information states. Accordingly, our contribution is an agentic RL framework that jointly trains evidence acquisition and probability reporting at multiple pre-resolution dates, rather than a method for optimizing cross-date memory utilization. We explicitly state in the Introduction (lines 49–50) that "Cross-date memory is optional." The memory-on variant is included as an extension to examine whether carrying a belief notebook provides additional benefits beyond longitudinal training, not as a prerequisite for our framework or a central claim of our work.

We also acknowledge the limitations of this design in the original submission. Specifically, our implementation uses immediate per-date rewards without explicit cross-date credit assignment, meaning that earlier notebook actions are not directly credited for their effects on later forecasts (Section 3.3, lines 297–300). We discuss this as a plausible explanation for the observed suboptimal results and identify explicit cross-date credit assignment as future work (Section 4.2, lines 409-411; Appendix A.5). Therefore, the memory-on results characterize the limitations of an optional extension, rather than contradicting the effectiveness of longitudinal training.


We thank the reviewer again for the constructive feedback. We hope our clarifications on the proper-scoring objective, longitudinal training, and the role of optional cross-date memory, together with the theoretical and empirical evidence already presented, address the concerns and support consideration for acceptance.

Thank you for the clarification on the reward design. It addresses my concern in W1, and I have raised my score from 2 to 3.

Regarding W2: the authors state that the research question is how outcome feedback can train evidence-seeking and probability-reporting behavior across changing information states. Recent memory-based approach [1] shares a similar goal without updating model weights. ForecastCompass distills retrospective analyses of resolved event into reusable factors and calibration rules. A brief discussion of how outcome driven memory relates to outcome driven RL will help position the paper.

As for the memory-on result, ALPHABET’s notebook carries event-specific beliefs across dates of the same question, whereas memory in [1] stores knowledge transferable across questions. The negative memory-on result may caused by anchoring on earlier, less-informed beliefs rather than a limitation of memory in general. Could the authors comment on this distinction? If feasible, it would be informative to see whether cross event memory is complementary to ALPHABET.

[1] Chang, Yurui, et al. "ForecastCompass: Guiding Agentic Forecasting with Adaptive Factor Memory."

We sincerely thank you for acknowledging that our clarification addresses W1 and raising the score. We also appreciate the reference to ForecastCompass and address your question below.

1. Outcome-driven memory versus outcome-driven RL.

ForecastCompass and AlphaBet share the broader goal of improving forecasting through feedback from resolved events, but differ in what is learned and how that learning is transferred. ForecastCompass distills retrospective analyses of resolved forecasts into reusable predictive-factor and calibration memories, which guide future forecasting through inference-time retrieval without updating model weights. In contrast, AlphaBet directly updates the shared forecasting policy through proper-scoring RL, training evidence acquisition and probability reporting across date-specific information states. Consequently, AlphaBet transfers experience across events through learned policy parameters, whereas ForecastCompass transfers experience through explicit external memory. These approaches are not mutually exclusive: external memory can provide reusable forecasting guidance, while RL can improve how the underlying policy acquires and uses evidence.

2. Clarifying the scope and interpretation of our memory-on results.

We appreciate the reviewer's observation regarding the distinction between within-event belief tracking and cross-event transferable memory. We would like to clarify that our paper does not claim that memory is harmful or ineffective for forecasting. More importantly, the effectiveness of memory mechanisms is not the central research question of this paper. We include the memory-on variant as an additional design extension to explore whether carrying an event-specific belief notebook across forecast dates provides benefits. Our interpretation (Section 4.2, lines 409–412) is that this particular memory-on implementation does not yield additional improvements over our memory-free variant. We do not interpret these results as a general conclusion about the effectiveness of memory mechanisms in forecasting.

We agree that anchoring on earlier, less-informed beliefs is one plausible explanation for the observed memory-on results. Other potential factors include the absence of explicit cross-date credit assignment during training (Section 3.3, lines 297–300), the design of the belief notebook, and how accumulated information is incorporated into subsequent forecasts. However, a systematic investigation of these memory mechanisms goes beyond the central focus of our work. We also see cross-event transferable memory, as explored in ForecastCompass, as potentially complementary to AlphaBet's RL-trained policy, allowing knowledge distilled from resolved events to inform future forecasting decisions. We view both improving within-event memory mechanisms and integrating cross-event transferable knowledge with agentic RL as valuable directions for future work.

To make this distinction explicit, we have updated Appendix D (Limitations) to clarify that our memory-on results are specific to the memory design studied in this work and that a systematic investigation of memory effects in agentic forecasting RL is left for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.