acceptodds
Withdrawn from ICLR 2027

Agentic Forecasting with Structured Linguistic Beliefs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We propose the Bayesian Linguistic Forecaster (BLF), which is a search-based ReAct style agent for (binary) forecasting that augments its context with a linguistic 'belief state', consisting of a probability estimate and a structured natural-language evidence summary. (The belief update is 'Bayesian-inspired' rather than exact Bayesian inference, although we do include a comparison to a more explicit Bayesian approach using LLM-elicited likelihood surrogates.) On 400 historical ForecastBench question-date instances, BLF has the highest overall Brier Index among the compared external agents, and is the only compared method to significantly outperform a crowd+empirical-prior baseline in terms of overall performance. Our backtesting framework combines date-aware search tools, leakage auditing, and paired comparisons for variance control to provide a rigorous empirical framework for comparing agentic forecasting systems.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
forecasting, bayesian agents, LLM agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4sN6OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a new method for search-based agent for forecasting named "Bayesian Linguistic Forecaster" where the agent is able to update its belief state consisting of a probability and a rationale. They show how their proposed framework allow models to outperform external agents on ForecastBench.

Critical strengths
  • There is a lot of effort into preventing leaks from backtesting such as various auditing, date filtered search, etc.
  • Nice analysis with the paired comparisons and variance decomposition. Decoupling question difficulty and method from the outcome allows for better interpretability.
  • The research question is interesting, but the setup isn't ideal.
Critical weaknesses
  • Paper only studies performance across Gemini models, but uses GPT-5.4 for certain ablations. Not sure why not extend the coverage of models tested and why gemini.
  • ForecastBench isn't a very good dataset to use even if others are using it. If you look at the dataset, most questions can be answered without any knowledge of the topic mostly because it is binary and contain logical leakage.
  • While the paper state the work was done around 4.20, I think it would have been better to continue the work and test it on new benchmarks like FutureSim who looked at very similar things. https://arxiv.org/abs/2605.15188
  • Lack comparison to structured agent memory baselines
  • Results seems to be picked to look better. In the appendix, we see how BLF scores considerably lower on AIBQ2 or that the zero-shot baseline ties the best external agent and outperforms their own search agent.
Questions for authors

In Weaknesses section

Reviewer_3EyJOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper gives a heuristic method that introduces a text belief to a search-enabled forecasting agent problem. The belief state is a semi-structured JSON object containing a probability p, confidence, and related information. By including the belief, the author implements the ReAct-like agent framework and tests the agent's performance on ForecastBench. The ablation study of w/wo belief shows improvement on two LLM settings and a decrease on the GPT model.

Critical strengths
  • The paper provides a valuable empirical study and evaluation framework with a context-engineering method (structured belief updates)
  • In the experimental section, the author provides a very useful within-model ablation, where BLF and NoBel use the same main model and search setup.
  • The paper carefully sets up an evaluation environment. It uses state-of-the-art tools and leakage defences, etc. The agentic design and experimental setup is a valuable contribution.
Critical weaknesses
  • The biggest concern is from the result of Table 3, the ablation shows that the gain obtained by Belief is not consistent, and the score for GPT is worse. This result challenges the robustness of the proposed method. The authors acknowledge that they also don't know the reason. Given that the proposed method is a heuristic method, there is no guarantee under which conditions the method would work well.

  • The paper reads very much like AI-generated, including the tables and figures.

  • The idea of injecting structural belief is trivial.

Questions for authors
  • In Section 6, the author gives explicit Bayesian updating, and Table 3 finds no overall benefit from BLF’s linguistic belief state with GPT. Could the authors evaluate the explicit Bayesian updater in Section 6 using the same evidence and aggregation settings, and compare it with both heuristic BLF and NoBel? I think it would be valuable to understand how the belief improves performance and why it fails.

  • How much is the paper assisted by AI? I feel the idea is simple, and most of the work could be done with AI coding.

Reviewer_Yr8aOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors student the problem of agentic forecasting and introduce the Bayesian Linguistic Forecaster (BLF), which produces a belief state that includes an estimate of event probability as well as a structure evidence summary. They show that BLF outperforms baseline forecasting systems in terms of Brier Index. Additionally, they ablate the belief state and show that it improves performance for two out of three considered models. They additionally study the effect of Full-History memory, where the entire interaction history is retained, compared to Markov history, where the last belief, action, and observation are retained instead, showing a small benefit to Markov history. They finally compare their heuristic approach with explicit Bayesian inference, where the BLF outperforms.

Critical strengths

The main strength of this paper is related to experimentation. The authors provide ablations to separate the effect of belief tracking, compare different versions of memory (full-history versus Markov), and finally compare their heuristic with explicit Bayesian updates. The authors attention to detail regarding temporal leakage (the four-layer approach) is an additional strength.

Critical weaknesses

The main weakness of this paper is related to novelty. The method involves a ReAct loop combined with summarization, an output belief state, and an action. This approach is similar to existing work on scratchpads, iterative summarization, and compaction with agents.

I also have some concerns about the belief state ablation. It seems that the NoBel ablation prompt differs from the belief state prompt by more than just the omission of belief state related things. It also includes a section on the suggested strategy, which the belief state prompt does not include. There are also components of the belief state prompt which could influence things, such as the directions regarding authoritativeness of sources. A more thorough ablation is needed to determine the true impact of belief state.

Questions for authors

Can you report failure rates for Table 3 for the different models and methods?

For the explicit Bayesian experiments, did you use Pro for the linguistic BLF (versus Flash for the 0-10 scores)? If so, did you record results with matched models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.