acceptodds
Under review as a conference paper at ICLR 2027

FutureWorld: A Live Reinforcement Learning Environment for Predictive Agents with Real-World Outcome Rewards

Abstract

Live future prediction refers to the task of making predictions about real-world events before they unfold. This task is increasingly studied using large language model-based agent systems, and it is important for building agents that can continually learn from the real world. It can provide a large number of prediction questions grounded in diverse real-world events, while preventing answer leakage. To leverage the advantages of future prediction, we present FutureWorld, a live agentic reinforcement learning environment that closes the training loop between prediction, outcome realization, and parameter updates. Specifically, we modify and extend verl-tool, resulting in a new framework that we call verl-tool-future. Unlike standard RL training frameworks that rely on immediate rewards, verl-tool-future stores prediction-time rollouts, backfills rewards after real-world outcomes become available, and then replays the completed trajectories for policy update. Across three open-source agents, successive FutureWorld training rounds lead to consistent improvements in prediction accuracy, probabilistic scoring, and calibration, demonstrating that delayed real-world outcome feedback can serve as an effective RL signal for predictive agents.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
live future prediction, agentic reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_WJxvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how agents can self-improve their predictive capabilities through online reinforcement learning as real-world events continue to unfold. The authors propose a real-time feedback framework that saves the agent's search trajectories at prediction time. The reward is calculated by comparing the agent's predicted probability of an event with the actual ground-truth outcome retrieved later, allowing the agent to update its policy. Unlike prior work, this paper combines live question generation, active agent search, real-world outcome rewards, and model parameter updates into a single, continuous training loop. Experiments are conducted by training multiple models at a relatively large scale, with comparisons against retrieval and supervised fine-tuning baselines. Results show that the agents improve across all metrics.

Critical strengths
  1. The paper proposes a live RL environment for future prediction. It combines live question generation, agentic information retrieval, trajectory storage, ground-truth retrieval, and policy updates into a complete training loop. Unlike previous methods that use immediate rewards or static historical questions, FutureWorld stores prediction-time rollouts and assigns rewards after real-world outcomes become available. This allows agents to learn directly from the outcomes of their own predictions. I think this is a meaningful contribution to training agents with real-world feedback.
  2. The paper provides experiments on multiple open-source LLMs and shows improvements across consecutive training days. The authors also compare RL with SFT, evaluate the effect of web search using placebo retrieval, and test generalization across different domains and prediction tasks. Additional experiments include evaluation on ForecastBench, 28-day training with Qwen3-32B, and offline training on historical questions. These experiments provide evidence that FutureWorld training can improve prediction performance, especially in terms of Brier score and calibration.
Critical weaknesses
  1. Section 3.9 states that the agent generates prediction trajectories at 20:00 each day and uses them for training after retrieving the outcomes at 20:30 on the following day. Since trajectory generation and model updates are interleaved across days, some trajectories may have been generated by an earlier policy.The paper does not clearly explain how policy versions are managed or how GRPO handles a possible mismatch between the policy used for rollout generation and the policy used for training. Since delayed trajectory replay is a key part of the proposed framework, the authors should clarify the training schedule and how policy updates are handled.
  2. The paper uses the negative Brier score of the final prediction as the reward for the entire rollout, including the search actions. Although this reward can be used to train the search policy, it does not directly measure the quality or efficiency of individual search actions.The experiments mainly show improvements in final prediction performance, but do not directly examine how the agent's search behavior changes after training. It is therefore unclear how much of the improvement comes from better information retrieval, rather than better use of retrieved information or better probability estimation.
Questions for authors
  1. According to the schedule in Section 3.9, the model may be updated using the previous day's trajectories while the current day's trajectories are still waiting for their outcomes. Could the authors clarify the order of rollout generation and policy updates? Are the token log probabilities from the rollout policy stored and used during training? If the policy has changed between rollout generation and training, how is this handled in GRPO?
  2. Could the authors report how the number of search steps and the agent's search behavior change before and after training? Under the same search budget, does the trained model retrieve more relevant information? Also, how much do the rewards typically vary across the four rollouts sampled for each question?
2 comments
AuthorsOpenReview ↗

Response to Reviewer WJxv (1 of 2)

Weaknesses 1: Section 3.9 states that the agent generates prediction trajectories at 20:00 each day and uses them for training after retrieving the outcomes at 20:30 on the following day. Since trajectory generation and model updates are interleaved across days, some trajectories may have been generated by an earlier policy.The paper does not clearly explain how policy versions are managed or how GRPO handles a possible mismatch between the policy used for rollout generation and the policy used for training. Since delayed trajectory replay is a key part of the proposed framework, the authors should clarify the training schedule and how policy updates are handled.

Reply:

We would like to clarify the ordering of rollout generation and parameter updates in our training loop. In Section 3.9, 20:00 is when we generate the new prediction questions. At 20:30, we attempt to retrieve the ground truth for the previous day's questions. For the questions whose outcomes we obtain, we backfill rewards into their stored trajectories and use those trajectories to update the model with GRPO. Once the update is complete, we use the newly updated model to generate trajectories for the questions created at 20:00 that evening. We first complete the parameter update for the previous round and then use the updated model to generate the next batch of prediction trajectories. These trajectories are stored, and we perform no parameter updates while waiting for their corresponding outcomes. To maintain this fixed update schedule, we discard questions whose outcomes cannot be retrieved at the scheduled time. So the policy staleness caused by such updates does not arise under our current training schedule. Although we describe our training loop in the original manuscript, we do not sufficiently emphasize that model parameters remain unchanged while waiting for the corresponding outcomes.

An asynchronous training loop could allow the model to keep updating as shorter-horizon questions resolve while longer-horizon forecasts are still pending. By the time those outcomes become available, the corresponding trajectories may have been generated several policy updates earlier. Extending our current synchronous training loop to support asynchronous updates, and studying how to learn reliably from trajectories generated by earlier policies, are directions for future work.

Weaknesses 2: The paper uses the negative Brier score of the final prediction as the reward for the entire rollout, including the search actions. Although this reward can be used to train the search policy, it does not directly measure the quality or efficiency of individual search actions.The experiments mainly show improvements in final prediction performance, but do not directly examine how the agent's search behavior changes after training. It is therefore unclear how much of the improvement comes from better information retrieval, rather than better use of retrieved information or better probability estimation.

Reply:

We compared Qwen3-32B Base and the final LoRA checkpoint on the same 1,000 forecasting questions. Both used the same prompts and search budget of up to three queries per question.

MetricBaseLoRA
Mean generated tokens per prediction1,982.21,075.2
Mean submitted search queries per prediction2.2411.191
Mean generation time per prediction246.9 s136.1 s
GPU-hours23.9715.03

In this evaluation, the LoRA checkpoint improved forecast quality while reducing inference resource use.

We agree that improvements in forecasting performance may come from finding more relevant information, reasoning more effectively about the retrieved evidence, or estimating probabilities more accurately. A simple ablation cannot cleanly separate these contributions because the three abilities interact within the same rollout.

For example, effective search involves both formulating precise queries and making good use of the results already retrieved. When an agent can search more than once, how it interprets the first set of results affects whether it searches again and how it formulates the next query. If the agent realizes that a missing fact could substantially change its probability estimate, it can search specifically for that fact. The new results can then change its reasoning and final probability estimate. In this way, better use of retrieved information can itself lead to more effective search. Search, evidence use, and probability estimation are interdependent parts of the forecasting capability, and their contributions to the observed improvement are difficult to disentangle.

AuthorsOpenReview ↗

Response to Reviewer WJxv (2 of 2)

Question 1: According to the schedule in Section 3.9, the model may be updated using the previous day's trajectories while the current day's trajectories are still waiting for their outcomes. Could the authors clarify the order of rollout generation and policy updates? If the policy has changed between rollout generation and training, how is this handled in GRPO?

Reply:

Please see our reply to Weaknesses 1.

Question 2: Are the token log probabilities from the rollout policy stored and used during training?

Reply:

No. In our experiments, we do not store or reuse the token log probabilities. When the outcomes become available, the trainer replays the saved token sequences and computes their log probabilities with the training actor before the update. The GRPO policy loss uses training-side log probabilities.

Question 3: Could the authors report how the number of search steps and the agent's search behavior change before and after training? Under the same search budget, does the trained model retrieve more relevant information?

Reply:

Please see our reply to Weaknesses 2. Under the same search budget, the trained model submitted 46.9% fewer queries per prediction on average and achieved a lower mean Brier score. Based on these results, we believe that the trained model retrieves more relevant information.

Question 4: How much do the rewards typically vary across the four rollouts sampled for each question?

Reply:

In our 28-day Qwen3-32B LoRA experiment, we analyzed all resolved training questions. For each question, we computed the sample standard deviation of the four raw rewards. The median within-question standard deviation was 0.280, and the mean was 0.268. The median gap between the highest and lowest reward was 0.603, and 4.8% of questions had identical rewards across all four rollouts.

Reviewer_EsAfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents FutureWorld, a live reinforcement learning environment for forecasting real-world events. Agents search for information and make predictions before events resolve. The system stores their trajectories, retrieves outcomes later, and uses outcome-based rewards for training. Experiments include three smaller models and a separate 28-day Qwen3-32B LoRA experiment. The paper reports improvements in forecasting quality and calibration, with additional baseline and external benchmark evaluations.

Critical strengths

The paper implements a live training loop and reports improved forecasting quality across several models.

Critical weaknesses
  1. The practical cost-benefit evidence is incomplete. Appendix E compares prediction quality against base models with web search, but does not report the additional training compute, search/API usage, or per-prediction latency and resource usage. Hardware and training settings are provided, but these do not establish the total cost. Therefore, it remains unclear whether the gains justify the additional resources compared with simply using the base model with search.
  2. A secondary limitation is the lack of a controlled model-scaling analysis. The Qwen3-32B experiment provides useful evidence of benefits on a larger model. However, its training method, duration, daily question count, and evaluation set differ from the smaller-model experiments. These results therefore do not show how the additional benefit changes with model size or baseline capability. A controlled comparison within one model family, with clearly stated training and search budgets and corresponding costs, would strengthen this evidence.
Questions for authors

Could the authors report the additional training compute and search/API usage, together with inference latency and tool usage, relative to the base-model-with-search baseline? This would help assess the practical value of the improvements.

1 comment
AuthorsOpenReview ↗

Response to Reviewer EsAf

Weaknesses 1: The practical cost-benefit evidence is incomplete. Appendix E compares prediction quality against base models with web search, but does not report the additional training compute, search/API usage, or per-prediction latency and resource usage. Therefore, it remains unclear whether the gains justify the additional resources compared with simply using the base model with search.

Reply

We compared Qwen3-32B Base and the final LoRA checkpoint on the same 1,000 forecasting questions. Both used the same prompts and search budget of up to three queries per question.

MetricBaseLoRA
Mean generated tokens per prediction1,982.21,075.2
Mean submitted search queries per prediction2.2411.191
Mean generation time per prediction246.9 s136.1 s
GPU-hours23.9715.03

In this evaluation, the LoRA checkpoint improved forecast quality while reducing inference resource use.

Training the final Qwen3-32B LoRA checkpoint used approximately 238 A100 GPU-hours, including 205 for collecting training rollouts and 33 for updating the adapter. During training, we generated four rollouts per question and averaged 5.07 completed searches per question across those rollouts (1.27 per rollout).

Weaknesses 2: A secondary limitation is the lack of a controlled model-scaling analysis. The Qwen3-32B experiment provides useful evidence of benefits on a larger model. However, its training method, duration, daily question count, and evaluation set differ from the smaller-model experiments. These results therefore do not show how the additional benefit changes with model size or baseline capability. A controlled comparison within one model family, with clearly stated training and search budgets and corresponding costs, would strengthen this evidence.

Reply

We agree that a controlled comparison is needed to determine how the gains from FutureWorld training vary with model size. In our preliminary experiments, larger models appeared to train more stably, with slower improvements in forecasting performance. Larger model achieved better forecasting performance. These observations are preliminary and still need to be verified.

The slower gains we observed for larger models suggest that short runs may underestimate how much they benefit from training. A comparison over a longer training horizon would take more time than is available during the rebuttal period. We will report the controlled scaling results in the camera-ready version, including the gains relative to each model's starting checkpoint, the training and search budgets, and the corresponding costs.

Question 1: Could the authors report the additional training compute and search/API usage, together with inference latency and tool usage, relative to the base-model-with-search baseline? This would help assess the practical value of the improvements.

Reply:

Please see our reply to Weaknesses 1.

Reviewer_9AZZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary
  • The paper presents FutureWorld, a live reinforcement learning environment for training LLM agents to forecast real-world events. Each day the system:
    • generates about 2,000 binary questions from 72 public sources;
    • filters them with an LLM and resamples them by clustering to balance domains;
    • has the agent search the web and output a probability;
    • retrieves the actual outcome the next day and assigns the reward.
  • To support this delayed reward, the authors extend verl-tool into verl-tool-future, which stores the agent's rollouts, fills in the rewards once outcomes are available, and then updates the policy using GRPO with a negative Brier score reward.
  • Validation:
    • three small open models (Qwen2.5-3B, Qwen3-4B, DeepSeek-R1-0528-Qwen3-8B) trained for 8 days, with accuracy, Brier score and ECE improving on 500 FutureWorld questions;
    • a 28-day LoRA run with Qwen3-32B;
    • transfer to a multi-format daily benchmark and to ForecastBench;
    • a supervised fine-tuning baseline, offline training on historical questions, and an AI-assisted audit of the outcome labels.
Critical strengths
  • Relevant problem, practical infrastructure. Forecasting future events matters for real decision-making. Using realized outcomes as the reward provides supervision that scales without annotation and avoids answer leakage by design. The end-to-end daily pipeline and verl-tool-future are a useful engineering contribution that other groups could build on.
  • Clarity and data quality control. The paper is self-contained and easy to follow. The authors validate their data unusually carefully: an item-level audit reports 96.39% of outcome labels verified correct (Table 8), and Appendix D analyses which questions are dropped as unresolved. This makes the reported training signal credible.
Critical weaknesses
  • Limited novelty; the central claim is not substantiated.
    • Outcome-based RL for forecasting has already been shown (Turtel et al., 2025; Jeen et al., 2026). This paper uses the same reward (negative Brier score), the same algorithm (GRPO) and the same binary question format.
    • The claimed distinction is that live training with search inside the loop is necessary. This is never tested against offline training with date-filtered search at equal compute. The paper's own offline experiment (Appendix H) also shows improvement.
    • There is no analysis of what the agent learns. Questions are partly drawn from Polymarket, and prediction-market sites are not excluded from search (Appendix I, Limitations). The agent may simply learn to look up crowd prices.
    • As a result, the paper contributes infrastructure but little new, rigorously tested knowledge.
  • Shallow experimental design and weak evidence.
    • The prediction horizon is about 24 hours (Sec. 3.9). Questions are dominated by short-term numeric quantities such as temperature, exchange rates and commodity prices. This is closer to nowcasting than to the long-horizon forecasting the paper uses as motivation, so the delayed-reward mechanism is barely exercised.
    • The agent cannot act on the environment: its actions affect neither the outcomes nor later states. The setting is therefore a one-step bandit problem with delayed reward rather than an RL environment, and the comparison with interactive environments (Fig. 2) is overstated.
    • Reported gains are small or inconsistent:
      • Qwen3-32B gains only +0.31% accuracy after 28 days (Table 1).
      • Several ForecastBench sub-scores get worse (Table 7).
      • The daily benchmark has at most 50 questions per day, too few to support claims of outperforming GPT-5.4.
    • No base-rate (constant-prior) baseline is reported, a single seed is used, and 35.65% of questions are discarded as unresolved, which may bias the data.
Questions for authors

Clarification

  1. What is the base rate of positive outcomes in the training and evaluation questions? How does a constant-prior predictor score on accuracy, Brier score and ECE?
  2. Are the 500 evaluation questions in Sec. 4.2.1 disjoint from the training questions in time and template? On which day were they issued?
  3. What share of the agent's searches retrieve prediction-market or odds-aggregator pages? Does the improvement remain when these sites are excluded?
  4. Why does real web search worsen the base model's Brier score compared with placebo retrieval for Qwen3-4B and Qwen2.5-3B (Table 6)?
  5. Please provide the training prompt, the list of source websites, the question templates, the filter prompts, the maximum number of search turns and context length, and the KL coefficient. Will the code and a frozen snapshot of questions and search results be released?
  6. Appendix H gives a training period in July 2025 but holdout questions resolved on June 30, 2026. Is this a typo?

Suggestions

  1. An equal-compute comparison between live training and offline training with date-filtered search would directly test the central claim.
  2. Evaluating longer horizons (weeks to months) and addressing the resulting staleness of the policy between prediction and reward would strengthen the motivation for a delayed-reward framework.
8 comments
AuthorsOpenReview ↗

Response to Reviewer 9AZZ (1 of 7)

Weaknesses 1: Outcome-based RL for forecasting has already been shown (Turtel et al., 2025; Jeen et al., 2026). This paper uses the same reward (negative Brier score), the same algorithm (GRPO) and the same binary question format.

Reply:

We do not claim novelty in the negative Brier-score reward, the GRPO algorithm, or the binary question format. Our contribution is to establish live future prediction as an online learning environment in which agents can continually learn from real-world outcomes. It allows the agent to keep learning as events and available information change. FutureWorld's pool of sources can be expanded by adding new websites, which allows us to generate more prediction questions and cover a broader range of real-world events and domains. The realized outcomes provide ground-truth labels without requiring human annotation, keeping the cost of obtaining feedback very low. This combination of extensible question sources and low-cost outcome feedback makes it practical to scale online training to many more questions. To support this setting, we develop verl-tool-future, which decouples prediction-time rollout collection from subsequent outcome retrieval, reward backfilling, and policy updates using the stored trajectories.

AuthorsOpenReview ↗

Response to Reviewer 9AZZ (2 of 7)

Weaknesses 2: The claimed distinction is that live training with search inside the loop is necessary. This is never tested against offline training with date-filtered search at equal compute. The paper's own offline experiment (Appendix H) also shows improvement.

Reply:

We agree that offline training can improve forecasting performance, and the results in Appendix H support this. Because historical outcomes are already available, the offline experiment lets us quickly assess the effect of training. The experiment simulates live training by approximating the information the agent could have accessed at each historical prediction time.

Reconstructing the information conditions of a real live online experiment poses two challenges for offline training. First, enforcing a strict information cutoff in offline forecasting evaluations is difficult. Future information leakage is a common limitation of current search engines. As the table below shows, even when given an explicit cutoff date, search engines can return results containing information that became public only after that date. Second, it can be hard to recover the information that was available at the time of a historical forecast. For example, if a webpage was updated after the forecast date, researchers may only be able to access the updated version.

After carefully considering these limitations and the need for a reliable offline experiment, we ultimately chose to use only questions from Manifold and Polymarket, drawing on far fewer sources than in our live experiments.

Despite our engineering efforts to mitigate future information leakage in the offline experiment, some leakage remained in the returned search results. To assess its extent, we use GPT-6.1 Sol with reasoning effort set to Max to audit the prediction trajectories generated by each offline-trained model on the holdout set. GPT-6.1 checked the search results one by one. If a result contained information that only became available after the simulated prediction time, we flagged it for future information leakage. For each model, we report the percentage of audited search results flagged for future information leakage.

Modelleakage rateUnverified rate
Qwen3-4B33.39%31.48%
Qwen2.5-3B34.00%30.87%
DeepSeek-R1-0528-Qwen3-8B17.90%27.50%

Both percentages use all search results for each model as the denominator. We mark a result as unverified when the available evidence is insufficient to determine whether it contained information first made public after the search cutoff.

The substantial future information leakage observed in our offline experiments makes the reported gains less reliable as evidence of improved forecasting. The gains could reflect better probability calibration or better use of evidence available at the prediction time. It is also possible that the models learned shortcuts during offline training, such as issuing queries that surface information published after the cutoff and using it to make their predictions. Offline training used the same date-filtered retrieval mechanism, so the training trajectories were subject to the same leakage risk.

An equal-compute comparison would control the training budget. A major difference would remain in the information available to the models when they make predictions. Offline search can surface information that appeared after the historical cutoff and miss information that was available at the time. Some pages that were accessible at the prediction time have since been removed or changed, and no snapshots of their earlier versions survive. Because we could not reliably reconstruct what was retrievable at the time, we did not use GPT-6.1 Sol to assess how much of that information the offline searches missed. Under these conditions, a performance difference at equal compute would be difficult to attribute specifically to the choice between online and offline training.

We therefore treat the offline experiment as a simulation that provides a preliminary indication of what outcome-based RL might achieve. To see whether training actually improves forecasting, we need to test the model in real time. It makes each prediction using the information available at that moment, and we score the prediction once the outcome is known. This is precisely why we study online future prediction in FutureWorld, even though some prior works have already explored forecasting through offline simulations.

AuthorsOpenReview ↗

Response to Reviewer 9AZZ (3 of 7)

Weaknesses 3: Questions are partly drawn from Polymarket, and prediction-market sites are not excluded from search (Appendix I, Limitations). The agent may simply learn to look up crowd prices.

Reply:

We run a new retrieval ablation comparing each evaluated Base model with its FutureWorld RL checkpoint after blocking specific prediction-market and odds-aggregation websites from search results.

The evaluation is constructed from a FutureWorld pool of around 2,000 forecasting questions. Accuracy gain is RL minus Base; Brier and ECE improvements are Base minus RL, so positive values consistently favor RL.

Qwen3-4B

Search conditionAccuracy gainBrier improvementECE improvement
Unblocked+19.07+7.43+10.62
Blocked+14.43+2.95+3.77

Qwen2.5-3B

Search conditionAccuracy gainBrier improvementECE improvement
Unblocked+5.54+6.46+8.78
Blocked+5.33+5.94+7.78

Across both Qwen3-4B and Qwen2.5-3B, FutureWorld RL increases accuracy and reduces both Brier score and ECE under the unblocked and blocked conditions. The gains remain positive after prediction-market and odds-aggregation websites are blocked, suggesting that the improvements do not depend solely on retrieving information from these sources.

Weaknesses 4: The prediction horizon is about 24 hours (Sec. 3.9). Questions are dominated by short-term numeric quantities such as temperature, exchange rates and commodity prices. This is closer to nowcasting than to the long-horizon forecasting the paper uses as motivation.

Reply:

FutureWorld covers a wide range of prediction targets. The Supplementary Material submitted with our paper includes some FutureWorld question batches. In the April 21, 2026 batch alone (FutureWorld/FutureWorld_v1.1.0/questions/2026-04-21.json), at least 207 of the 500 questions (41.4%) ask about categorical outcomes in non-numeric domains. For example, the batch asks whether Crystal Palace and West Ham United would draw, whether both teams would slay Baron Nashor in the first game of a League of Legends match, whether Anaconda would top the U.S. Netflix movie chart that week, and whether Claude would rank first among free apps in the U.S. Apple App Store on April 21.

Our evaluation also covers longer-horizon forecasting through ForecastBench. Specifically, the 2026-03-01 round used in our experiments includes prediction horizons of 7, 30, 90, and 180 days in its Dataset component. As reported in the appendix, all three trained checkpoints achieve lower Overall Brier scores than their corresponding base models.

We use the question set and prediction horizons provided by ForecastBench. Although we cite the benchmark, our original manuscript did not explicitly describe these longer prediction horizons, which may have left the scope of our evaluation unclear to readers unfamiliar with ForecastBench.

Weaknesses 5: The agent cannot act on the environment: its actions affect neither the outcomes nor later states. The setting is therefore a one-step bandit problem with delayed reward rather than an RL environment, and the comparison with interactive environments (Fig. 2) is overstated.

Reply:

As described in Section 3.2 and Figure 4, an agent receives a prediction question, interacts with external information sources through search actions and observations, and then produces a probability estimate. A search query is a policy action, while the returned evidence is an observation. Different queries retrieve different evidence, and the policy can condition subsequent queries and the final forecast on the evidence already obtained. Thus, although the agent does not control the event outcome, its actions affect the observation history on which its subsequent decisions depend.

Crucially, these intermediate search actions are part of the policy being optimized, rather than a fixed preprocessing step. Section 4.1 explicitly states that search queries are treated as policy actions, while tool observations are masked in the policy loss because they are generated by the external environment. The terminal, outcome-based reward therefore supervises a policy that performs information acquisition as well as probability prediction.

Reviewer_Bs51OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FutureWorld, a live RL environment for training LLM agents to predict events, with the reward appearing after the event actually happens (contribution #1). Their pipeline collects events from a pool of public websites, turns them into binary questions, filters them, and resamples them to balance the proportion of questions across domains. The agent then searches the web at least once and outputs a probability that reflects confidence level for each prediction. The next day, outcomes appear (although a notable proportion doesn't, see my W1) and the policy is updated with GRPO. verl-tool\texttt{verl-tool} is extended to verl-tool-future\texttt{verl-tool-future} to support this delay (contribution #2). As for the validation, they train three small open-source models for 8 days and show day-to-day improvements in accuracy, Brier score, and calibration (ECE) in most domains. On a separate daily benchmark, they extend validation to other types of questions like numeric or multiple-choice, where we can also see improvements wrt the before training\textit{before training} model.

Critical strengths

S1: Both contributions of the paper address a clear gap in the literature. FutureWorld is, as illustrated in Fig. 2, the first learning environment for future prediction, as others like WebShop or ALFWorld are static and closed, i.e. the world only changes when the agent acts, and the reward is available instantly. This is very relevant now that agents are increasingly expected to act and learn in a world that keeps changing. Their second methodological contribution, verl-tool-future\texttt{verl-tool-future}, is a natural extension that makes delayed feedback possible, something that the standard RL frameworks don't support.

S2: The authors were careful about data quality (they introduced three filters so that each selected event is resolvable, meaningful, and safe for public release), and included a comprehensive Appendix with supplementary material that makes it mostly reproducible, apart from some missing details (see my comments below).

Critical weaknesses

W1: The main weakness I identified is the substantial amount of events discarded ("35.65% of questions do not have retrievable ground truth at the scheduled retrieval time"). This makes a lot of computation wasted, and I strongly recommend moving this limitation from the Appendix I to the main text (within the Conclusion section). I recommend discussing there as well future directions on how to solve this waste. Also, I'm a bit concerned about the balance of domains after\textbf{after} discarding. In Figure 1c, the domains look nicely balanced, but what about after discarding more than 1/3 questions because they couldn't be resolved? The actual training set may end up less balanced.

Questions for authors

I've the following questions regarding different aspects of the paper:

Q1: On the FutureWorld daily benchmark, Appendix C

  • Table 5 shows a lot of variance in the SbinS_{bin} column, which makes sense given there are at most 5 binary-choice questions per day (Table 2). However, I see that Eq. 8 under C.2.3 gives SbinS_{bin} the same importance as the other three types, so a single binary question can shift SoverallS_{overall} by several points. Did you consider a different weighting system to make the overall score more robust? Or having more than 5 binary-choice questions per day?
  • The model is only trained probabilistically on binary questions, while the daily benchmark also includes e.g. multiple choice or numeric questions. Have you considered training probabilistically on these types too?

Q2: On reproducibility

  • In Appendix A, you state "the values of MdM_d are chosen to make the domain distribution as balanced as possible", but it's not completely clear to me how exactly the budget MM is split across domains. Is it something like small domains capped at NdN_d and the leftovers redistributed? A short description would help.
  • The prompts in Appendix M are for the daily benchmark and ask for discrete answers, but Section 3.7 says a template is used to "convert each retained binary question into a probability estimation question". Could you include this training prompt as well to ensure reproducibility?
  • The three quality filters (resolvable, meaningful, safe) make a lot of sense, but I'd like to know more about how the filtering is actually done. What are the filter prompts send to Seed-1.6? Did you check its judgments against human labels? How many questions each filter removes?

A last minor question:

  • In Figure 1, "Entertainment" goes from the second-smallest category in (a) to the largest in (b). Is it because a few entertainment websites generate many questions each day?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Bs51 (1 of 5)

Weaknesses 1: The main weakness I identified is the substantial amount of events discarded ("35.65% of questions do not have retrievable ground truth at the scheduled retrieval time"). This makes a lot of computation wasted, and I strongly recommend moving this limitation from the Appendix I to the main text (within the Conclusion section). I recommend discussing there as well future directions on how to solve this waste.

Reply:

Thank you for highlighting this limitation. We agree that discarding questions wastes the resources already spent on prediction and web searches. We greatly appreciate your suggestion and agree that this limitation should be discussed in the main text.

An asynchronous training loop could allow the model to keep updating as shorter-horizon questions resolve while longer-horizon forecasts are still pending. By the time those outcomes become available, the corresponding trajectories may have been generated several policy updates earlier. Extending our current synchronous training loop to support asynchronous updates, and studying how to learn reliably from trajectories generated by earlier policies, are directions for future work.

Weaknesses 2: I'm a bit concerned about the balance of domains after discarding. The actual training set may end up less balanced.

Reply:

As you anticipated, discarding unresolved questions does make the effective training set less balanced across domains. We show the domain distribution of the retained training questions. Nevertheless, the extent of this imbalance is limited.

DomainBefore discardingAfter discardingChange
Financial Markets14.91%16.62%+1.71
Agriculture7.46%5.09%−2.36
Options Volatility7.89%6.43%−1.46
Exchange Rates8.33%9.12%+0.78
Industry9.21%7.24%−1.97
Weather7.46%1.88%−5.58
Logistics5.92%7.24%+1.32
Entertainment20.61%24.13%+3.51
Politics18.20%22.25%+4.05

Question 1: Table 5 shows a lot of variance in the SbinS_{bin} column, which makes sense given there are at most 5 binary-choice questions per day (Table 2). However, I see that Eq. 8 under C.2.3 gives SbinS_{bin} the same importance as the other three types, so a single binary question can shift SoverallS_{overall} by several points. Did you consider a different weighting system to make the overall score more robust? Or having more than 5 binary-choice questions per day?

Reply:

As shown in Table 2, we allocate different numbers of questions to the four question types. This allocation is motivated by our preliminary experiments, which revealed different levels of difficulty across these types. We therefore assign more questions to the more challenging types. The easier types also exhibited more consistent performance, with 100% accuracy observed in many of the reported evaluations in Table 5.

Moreover, numeric questions have the largest daily sample size among the four question types. Yet even when we restrict the comparison to this type, the relative rankings of the evaluated agent frameworks still vary across days. The observations suggest that results from a single day provide a limited basis for comparing the evaluated frameworks. This consideration also motivates the five-day comparison in Figure 7, rather than relying on a single day's results.

To provide a more representative summary of performance over the nine-day evaluation period in Table 5, we now report results averaged across all nine days. This reduces dependence on any individual day's question set and helps address your concern. The nine-day averages are presented below.

MethodSbinS_{\mathrm{bin}}SsmcS_{\mathrm{smc}}SdmcS_{\mathrm{dmc}}SnumS_{\mathrm{num}}SoverallS_{\mathrm{overall}}
smolagents84.3848.8643.4548.7256.36
Flash-Searcher80.5648.1851.6760.2260.20
OpenClaw80.0039.9644.8024.8947.42
Hermes77.8644.0647.2754.6755.96
GPT-5.4 (web search)51.2534.7213.8536.7134.13
AuthorsOpenReview ↗

Response to Reviewer Bs51 (2 of 5)

Question 2: The model is only trained probabilistically on binary questions, while the daily benchmark also includes e.g. multiple choice or numeric questions. Have you considered training probabilistically on these types too?

Reply:

Thank you for this insightful and constructive suggestion. Extending our approach to other question types calls for careful consideration of the reward design, rather than directly reusing our current binary Brier-score reward. We view the design and evaluation of suitable rewards for these question types as an important direction for future work.

One approach we plan to explore is to turn a numeric question into a prediction over intervals. For example, an agent forecasting tomorrow's temperature would assign probabilities to a set of non-overlapping temperature intervals that cover all possible outcomes. Once the actual temperature is available, we identify the interval it falls into and use the negative multiclass Brier score as the reward. Narrower intervals would allow us to train for finer-grained predictions. The same reward could be used for multiple-choice questions with a single correct answer by assigning a probability to each option. This extension would preserve the FutureWorld training loop.

Question 3: In Appendix A, you state "the values of MdM_d are chosen to make the domain distribution as balanced as possible", but it's not completely clear to me how exactly the budget MM is split across domains. Is it something like small domains capped at NdN_d and the leftovers redistributed? A short description would help.

Reply:

Yes, that is how we allocate the budget. We start with a quota of zero for each non-empty domain and make repeated passes through the domains, ordered from fewest to most available question–description pairs. On each pass, we give one slot to each domain that has yet to reach its capacity NdN_d. Once a domain reaches capacity, we skip it in subsequent passes, so the remaining slots go to domains that still have candidates. If the budget MM runs out partway through a pass, the remaining slots go to the first eligible domains in that order.

For example, suppose three domains have 2, 6, and 10 candidate pairs and the total budget is 12. After two passes, each domain has two slots. The smallest domain is now at capacity, so the remaining six slots are split evenly between the other two domains, giving final quotas of 2, 5, and 5.

These quotas are the targets for the within-domain clustering step. A pair assigned to multiple domains is retained only once. If overlaps leave any slots unfilled, we first try to fill the affected domain quotas from their remaining candidates, then fill any remaining slots from the full pool of unselected pairs.

Question 4: The prompts in Appendix M are for the daily benchmark and ask for discrete answers, but Section 3.7 says a template is used to "convert each retained binary question into a probability estimation question". Could you include this training prompt as well to ensure reproducibility?

Reply:

[System]
You are a careful forecasting assistant with access to the search tool. Follow the search protocol exactly.

[User]
Answer the given question. Before the final forecast, make at least one and at most three search calls. Normally use three searches to cross-check different sources or aspects of the evidence; finish early only when the evidence is already decisive. After the third result, do not search again; forecast from the available evidence, even if the result is empty or inconclusive. Emit each query exactly as <search>your query</search>; the tool replies inside <result> and </result> tags. Prefer current primary sources, the exact event definition, and the specified time window. For a binary question, the forecasted event is option A, Yes, Up, Over, Higher, or the first listed outcome. Give a concise rationale weighing evidence for and against the event. On every search turn, emit exactly one <search>...</search> block containing a specific query about the actual question. Do not repeat these instructions and do not generate <result> tags yourself. Estimate the probability that the event specified in the question will occur. Make the final non-empty line the only boxed expression. The boxed expression must start with \boxed{, contain only that probability expressed as a decimal number between 0 and 1, and end with }. Do not put p, A, B, a percent sign, or explanatory text inside the braces, and do not write anything after that line.

Question: {question}
AuthorsOpenReview ↗

Response to Reviewer Bs51 (3 of 5)

Question 5: The three quality filters (resolvable, meaningful, safe) make a lot of sense, but I'd like to know more about how the filtering is actually done. What are the filter prompts send to Seed-1.6?

Reply (1 of 2):

We use the three prompts below with Seed‑1.6. For each call, we replace <QUESTION> with the question text, append the supplementary description when available, and send the result as a single user message. The filters run in the order Meaningful → Resolvable → Safe.

Resolvable

You are an agent that can evaluate whether a proposed prediction question is objectively resolvable. The question to be evaluated:
"""
<QUESTION>
"""

Your goal is NOT to predict the outcome of this question. Your goal is to judge whether this question can be objectively resolved to a publicly verifiable and widely credible ground truth by the specified time.

If the outcome relies on secret/insider information, private personal facts, rumors, inherently unconfirmable claims, etc., you MUST judge it as NOT objectively resolvable.

If the question is multiple-choice (single- or multi-select) and the answer choices are sloppily specified (ambiguous, conflicting, overlapping, etc.), you MUST judge it as NOT objectively resolvable.

Leniency rules:
1. Be moderately permissive—judge NOT objectively resolvable only if the ambiguity/verification barrier would likely prevent reasonable third parties from reaching the same publicly verifiable ground truth.
2. If there exists a credible public resolution source or a reasonable, common-sense interpretation, judge it objectively resolvable.
3. Implicit domain conventions: If the question uses terminology, abbreviations, or phrasing that is not explicitly defined but has a widely recognized, industry-standard meaning among relevant practitioners (i.e., established domain convention or jargon), interpret it according to that conventional meaning. Only judge the question as NOT objectively resolvable if such implicit conventions are not sufficiently consistent or publicly verifiable to allow reasonable third parties to converge on the same ground truth.
4. Reasonable default assumptions for missing parameters: If the question omits certain parameters (e.g., currency unit, geographic scope, time zone, reference benchmark), but there exist widely accepted default conventions that allow a reasonable, publicly verifiable interpretation, apply those defaults. Only judge the question as NOT objectively resolvable if (i) multiple competing defaults are common in practice, or (ii) the omitted parameter would materially change the outcome and cannot be pinned down via credible public conventions or sources.
5. High-likelihood verifiability: If strict verification cannot be guaranteed, but the event is very likely to become publicly verifiable via a widely credible source by the resolution time (based on standard practice, recurring publication patterns, established institutional reporting, etc.), treat the question as objectively resolvable.
6. Finance/Market Default-Resolvable Bias: If the question concerns finance, cryptocurrency, securities/FX/futures, market prices/indices, or product sales/market volume, etc., then you should lean toward judging it objectively resolvable.

NOTE: If you notice a potential time conflict in the question, do not worry—this may be due to different time zones, and it can still refer to the same moment in time.

IMPORTANT: Your final answer MUST end with this exact format:
\boxed{Yes} or \boxed{No}

Output \boxed{Yes} if you judge the question is objectively resolvable, otherwise output \boxed{No}.

Do not use any other format. Do not refuse to make an evaluation. Do not say "I cannot evaluate the question." You must make a clear evaluation, using the box format specified above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.