- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how agents can self-improve their predictive capabilities through online reinforcement learning as real-world events continue to unfold. The authors propose a real-time feedback framework that saves the agent's search trajectories at prediction time. The reward is calculated by comparing the agent's predicted probability of an event with the actual ground-truth outcome retrieved later, allowing the agent to update its policy. Unlike prior work, this paper combines live question generation, active agent search, real-world outcome rewards, and model parameter updates into a single, continuous training loop. Experiments are conducted by training multiple models at a relatively large scale, with comparisons against retrieval and supervised fine-tuning baselines. Results show that the agents improve across all metrics.
Critical strengths
- The paper proposes a live RL environment for future prediction. It combines live question generation, agentic information retrieval, trajectory storage, ground-truth retrieval, and policy updates into a complete training loop. Unlike previous methods that use immediate rewards or static historical questions, FutureWorld stores prediction-time rollouts and assigns rewards after real-world outcomes become available. This allows agents to learn directly from the outcomes of their own predictions. I think this is a meaningful contribution to training agents with real-world feedback.
- The paper provides experiments on multiple open-source LLMs and shows improvements across consecutive training days. The authors also compare RL with SFT, evaluate the effect of web search using placebo retrieval, and test generalization across different domains and prediction tasks. Additional experiments include evaluation on ForecastBench, 28-day training with Qwen3-32B, and offline training on historical questions. These experiments provide evidence that FutureWorld training can improve prediction performance, especially in terms of Brier score and calibration.
Critical weaknesses
- Section 3.9 states that the agent generates prediction trajectories at 20:00 each day and uses them for training after retrieving the outcomes at 20:30 on the following day. Since trajectory generation and model updates are interleaved across days, some trajectories may have been generated by an earlier policy.The paper does not clearly explain how policy versions are managed or how GRPO handles a possible mismatch between the policy used for rollout generation and the policy used for training. Since delayed trajectory replay is a key part of the proposed framework, the authors should clarify the training schedule and how policy updates are handled.
- The paper uses the negative Brier score of the final prediction as the reward for the entire rollout, including the search actions. Although this reward can be used to train the search policy, it does not directly measure the quality or efficiency of individual search actions.The experiments mainly show improvements in final prediction performance, but do not directly examine how the agent's search behavior changes after training. It is therefore unclear how much of the improvement comes from better information retrieval, rather than better use of retrieved information or better probability estimation.
Questions for authors
- According to the schedule in Section 3.9, the model may be updated using the previous day's trajectories while the current day's trajectories are still waiting for their outcomes. Could the authors clarify the order of rollout generation and policy updates? Are the token log probabilities from the rollout policy stored and used during training? If the policy has changed between rollout generation and training, how is this handled in GRPO?
- Could the authors report how the number of search steps and the agent's search behavior change before and after training? Under the same search budget, does the trained model retrieve more relevant information? Also, how much do the rewards typically vary across the four rollouts sampled for each question?
2 comments
Response to Reviewer WJxv (2 of 2)
Question 1: According to the schedule in Section 3.9, the model may be updated using the previous day's trajectories while the current day's trajectories are still waiting for their outcomes. Could the authors clarify the order of rollout generation and policy updates? If the policy has changed between rollout generation and training, how is this handled in GRPO?
Reply:
Please see our reply to Weaknesses 1.
Question 2: Are the token log probabilities from the rollout policy stored and used during training?
Reply:
No. In our experiments, we do not store or reuse the token log probabilities. When the outcomes become available, the trainer replays the saved token sequences and computes their log probabilities with the training actor before the update. The GRPO policy loss uses training-side log probabilities.
Question 3: Could the authors report how the number of search steps and the agent's search behavior change before and after training? Under the same search budget, does the trained model retrieve more relevant information?
Reply:
Please see our reply to Weaknesses 2. Under the same search budget, the trained model submitted 46.9% fewer queries per prediction on average and achieved a lower mean Brier score. Based on these results, we believe that the trained model retrieves more relevant information.
Question 4: How much do the rewards typically vary across the four rollouts sampled for each question?
Reply:
In our 28-day Qwen3-32B LoRA experiment, we analyzed all resolved training questions. For each question, we computed the sample standard deviation of the four raw rewards. The median within-question standard deviation was 0.280, and the mean was 0.268. The median gap between the highest and lowest reward was 0.603, and 4.8% of questions had identical rewards across all four rollouts.
Response to Reviewer WJxv (1 of 2)
Weaknesses 1: Section 3.9 states that the agent generates prediction trajectories at 20:00 each day and uses them for training after retrieving the outcomes at 20:30 on the following day. Since trajectory generation and model updates are interleaved across days, some trajectories may have been generated by an earlier policy.The paper does not clearly explain how policy versions are managed or how GRPO handles a possible mismatch between the policy used for rollout generation and the policy used for training. Since delayed trajectory replay is a key part of the proposed framework, the authors should clarify the training schedule and how policy updates are handled.
Reply:
We would like to clarify the ordering of rollout generation and parameter updates in our training loop. In Section 3.9, 20:00 is when we generate the new prediction questions. At 20:30, we attempt to retrieve the ground truth for the previous day's questions. For the questions whose outcomes we obtain, we backfill rewards into their stored trajectories and use those trajectories to update the model with GRPO. Once the update is complete, we use the newly updated model to generate trajectories for the questions created at 20:00 that evening. We first complete the parameter update for the previous round and then use the updated model to generate the next batch of prediction trajectories. These trajectories are stored, and we perform no parameter updates while waiting for their corresponding outcomes. To maintain this fixed update schedule, we discard questions whose outcomes cannot be retrieved at the scheduled time. So the policy staleness caused by such updates does not arise under our current training schedule. Although we describe our training loop in the original manuscript, we do not sufficiently emphasize that model parameters remain unchanged while waiting for the corresponding outcomes.
An asynchronous training loop could allow the model to keep updating as shorter-horizon questions resolve while longer-horizon forecasts are still pending. By the time those outcomes become available, the corresponding trajectories may have been generated several policy updates earlier. Extending our current synchronous training loop to support asynchronous updates, and studying how to learn reliably from trajectories generated by earlier policies, are directions for future work.
Weaknesses 2: The paper uses the negative Brier score of the final prediction as the reward for the entire rollout, including the search actions. Although this reward can be used to train the search policy, it does not directly measure the quality or efficiency of individual search actions.The experiments mainly show improvements in final prediction performance, but do not directly examine how the agent's search behavior changes after training. It is therefore unclear how much of the improvement comes from better information retrieval, rather than better use of retrieved information or better probability estimation.
Reply:
We compared Qwen3-32B Base and the final LoRA checkpoint on the same 1,000 forecasting questions. Both used the same prompts and search budget of up to three queries per question.
In this evaluation, the LoRA checkpoint improved forecast quality while reducing inference resource use.
We agree that improvements in forecasting performance may come from finding more relevant information, reasoning more effectively about the retrieved evidence, or estimating probabilities more accurately. A simple ablation cannot cleanly separate these contributions because the three abilities interact within the same rollout.
For example, effective search involves both formulating precise queries and making good use of the results already retrieved. When an agent can search more than once, how it interprets the first set of results affects whether it searches again and how it formulates the next query. If the agent realizes that a missing fact could substantially change its probability estimate, it can search specifically for that fact. The new results can then change its reasoning and final probability estimate. In this way, better use of retrieved information can itself lead to more effective search. Search, evidence use, and probability estimation are interdependent parts of the forecasting capability, and their contributions to the observed improvement are difficult to disentangle.