acceptodds
Under review as a conference paper at ICLR 2027

BRIEF: Belief Revision for Iterative Evidence-based Forecasting

Abstract

Most forecasting agents that retrieve evidence use it in the same way. Once the search stops, everything collected is placed in a single context, and the model reads a forecast off it. Holding the evidence corpus fixed, we show that this practice makes poor use of it. Presenting the corpus all at once yields a Brier loss of 19.34, processing it in rounds lowers this to 18.13, and revising an explicit forecast belief as evidence arrives lowers it further to 16.52. Motivated by this finding, we introduce BRIEF (Belief Revision for Iterative Evidence-based Forecasting), a framework in which a belief state is initialized from multi-perspective research, revised under a Bayesian-inspired principle as new evidence arrives, and used to direct further research through the gaps and conflicts it leaves unresolved. A final refinement stage integrates candidate judgments and applies calibration and target-specific structural adjustments. Compared with twelve baselines spanning reasoning models, prompting and search strategies, and deep research systems, BRIEF achieves the best aggregate performance on FutureX, ForecastBench, and Prophet Arena, with the lowest variance across three runs. Relative to the strongest baseline, it improves the FutureX score by 7.86 points and reduces ForecastBench Brier loss by 5.3%, and it achieves the lowest Brier loss on Prophet Arena. These results indicate that how evidence is used is itself a design decision for forecasting agents, and that a maintained belief state offers an effective way to connect research decisions, evidence interpretation, and final forecast construction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, LLM agents, forecasting, belief revision, probabilistic forecasting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3E92OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how LLM forecasting systems can more effectively incorporate and use evidence, proposing BRIEF, a framework based on an explicit belief state. The framework first forms an initial forecast, then uses an evidence critic and a sufficiency controller to identify evidence gaps, guide further searches, and iteratively revise the belief as new evidence becomes available. Experiments on FutureX, ForecastBench, and Prophet Arena demonstrate some performance advantages over the evaluated baselines.

Critical strengths
  1. The paper addresses an important and practically relevant task. Real-world forecasting is a valuable application of LLM and requires reasoning under uncertainty with evolving information.

  2. The motivation is reasonable: forecasting quality depends not only on retrieving relevant evidence, but also on how that evidence is interpreted and incorporated into predictions. Studying evidence utilization is a meaningful research direction.

  3. The proposed framework demonstrates good empirical performance across FutureX, ForecastBench, and Prophet Arena. The component ablations and fixed-evidence comparisons provide useful additional analyses.

Critical weaknesses
  1. The methodological distinction from closely related work is insufficiently established. Although the paper cites and briefly compares against BLF (https://arxiv.org/pdf/2604.18576v1), the incremental contribution remains unclear. BLF already maintains a structured belief state containing a probability estimate, supporting and opposing evidence, unresolved questions, and explanations for belief updates. It uses this state during iterative tool use and incorporates forecast aggregation and calibration. The paper should more clearly explain what BRIEF adds beyond this framework and include a direct experimental comparison. Moreover, it does not discuss AuditForecast (https://arxiv.org/pdf/2609.05905), which also studies structured evidence integration through a model-derived base probability and evidence-grounded updates. This relationship warrants discussion.

  2. The Bayesian interpretation is insufficiently substantiated. Although the authors appropriately qualify the method as “Bayesian-inspired,” the operational update remains largely heuristic: an LLM interprets existing and newly retrieved evidence and produces a revised forecast, without explicit likelihood estimation or a numerical update rule. It is unclear how reliably the system detects overlap between old and new evidence or accounts for dependencies among evidences. Controlled tests involving duplicated, correlated, and contradictory evidence would help establish what the revision mechanism actually achieves.

  3. It is good to report search calls, page fetches, and research time, but the paper does not provide a comparison of total token consumption or inference cost across methods. Given the additional critic, controller, revision, and candidate-generation calls, it is difficult to determine whether the improvements reflect better evidence utilization or greater computational expenditure.

  4. The leakage controls require more rigorous validation, the retrieval pipeline just relies on an LLM to identify and remove information unavailable before the cutoff. Also, retrieving information and filtering it does not necessarily reproduce the information environment available at the future-driven forecast. Evaluation on real future event forecasting can futher verify the effectiveness of proposed method.

  5. Several methodological details need clearer explanation. For example, the distinction between “driver” and “disconfirmer” queries is unclear: a driver can itself provide disconfirming evidence, so the rationale for separating these roles should be explained. The paper also introduces fitted bias calibration parameters without specifying how they are estimated.

Questions for authors

N/A

6 comments

Thank you for the careful review and the constructive suggestions. We respond to each point below and would be glad to clarify anything that remains unclear.

Q1: What BRIEF adds beyond BLF and how it relates to AuditForecast.

A1: BRIEF differs from BLF [1] in three ways.

First, we show that, with the evidence held fixed, better use of evidence improves accuracy. BLF updates its belief together with search, so its gains combine retrieval and evidence use. Section 5.1 holds the corpus fixed and changes only how it is used, and Table 7 exchanges frozen evidence between DeerFlow and BRIEF. In both experiments, the way evidence is used changes accuracy on identical evidence.

Second, we stop research when the evidence is sufficient, not when the outcome is certain. BRIEF continues only when a gap is searchable and could change the forecast, which is when new information has positive expected value [2]. BLF, in contrast, stops when the agent submits or reaches its step limit. Removing this control lowers the FutureX score by 29.29 points (Table 3).

Third, we extend belief states from binary probabilities to structured answers, such as options, values, and probabilities for several non-exclusive markets. BLF’s belief is a single binary probability.

We also compared with BLF directly; the setup is given in our reply to Reviewer DkXc (Q1). BLF runs on DeepSeek-V4.1-Flash, since the V4-Flash API has been retired, and receives candidate answers for open-ended questions; both favor BLF.

Table 1: Comparison of BLF and BRIEF across three forecasting benchmarks.

MethodFutureX ↑ForecastBench Brier ↓Prophet Arena Brier ↓
BLF66.7012.7417.50
BRIEF76.9013.0116.73

BLF is slightly better on ForecastBench, the binary benchmark it was designed for, while BRIEF is clearly better on FutureX and Prophet Arena, where targets go beyond a single binary probability.

Compared with AuditForecast [3], concurrent work (arXiv, September 2026), BRIEF differs in two ways. AuditForecast updates a base probability with odds multipliers from factors checked for independence.

  1. We use the revised belief to direct research. BRIEF revises its belief round by round and judges each new report against the evidence already gathered. The belief then decides what to research next and when to stop, not only how the final probability is computed.
  2. We support structured targets. Odds updates apply to the probability of a single event, while BRIEF also handles choices, values, and multi-market events. AuditForecast also reports FutureX results, but it splits L1–L2 multiple-choice questions into independent binary questions and excludes L3–L4, which carry 70% of the FutureX score (Eq. 7); it forecasts only binary events, so multi-market events are not covered. BRIEF is evaluated on all four FutureX levels and on multi-market Prophet Arena events, and L3–L4 are where its margin over the best baseline is largest (7.88 and 9.18 points; Section 4.2).

We will discuss AuditForecast in the related work.

Q2. Substantiating the Bayesian interpretation.

A2: Eq. (4) describes the structure of each revision rather than a numerical rule, and each of its terms has a counterpart in BRIEF.

  1. Prior. P(Y∣Et)P(Y \mid E_t) is the current belief StS_t. Every revision starts from it, and the revised belief becomes the prior for the next round.
  2. Likelihood. P(ΔEt+1∣Y,Et)P(\Delta E_{t+1} \mid Y, E_t) is conditioned on what is already known. The model judges each new observation against the existing evidence, and a report already covered by EtE_t should leave the belief unchanged. Likewise, a report from an unreliable source is nearly as likely under every outcome and should move the belief little. The archive therefore keeps each fact's source, removes duplicates, and flags overlapping reports, and the critic marks repeated and dependent evidence (Appendix C.3).

We do not compute the update numerically. In our experiments, explicit Bayesian updating was less accurate than letting the model revise its belief directly, as BLF also found with LLM-estimated likelihoods, and it requires an enumerated hypothesis space that open-ended numeric and ranking targets lack.

On 44 two-outcome match events from Prophet Arena, we inject into each frozen corpus the same report that contradicts the current forecast (a player of the favored team will miss the match), varying only who reports it and whether follow-up reports are independent; this covers duplicated, correlated, and contradictory evidence. The table gives the shift of the forecast toward the outcome the report favors, relative to the uninjected corpus (mean over 44 events, standard error in parentheses).

ConditionReported byExpected under Eq. (4)Actual shift (pp)
lowAnonymous forum postAlmost no shift+0.2 (0.3)
oneOne outlet citing an anonymous insider, unconfirmedSmall, well below auth+0.7 (0.7)
relayone + three outlets citing that outlet's earlier reportSame as one; a relay is not new evidence+0.8 (0.4)
indepone + the same three outlets, same headline and wording, each citing its own sourcesClearly above relay, still below auth+2.5 (0.6)
authOfficial confirmation by the league or teamLargest+5.4 (1.0)
indep − relayIndependence gapClearly > 0+1.7 (0.7)
relay − oneRelays counted as new evidence?≈ 0+0.1 (0.7)
auth − lowReliability gapLargest gap in the table+5.2 (1.0)

The results match all three predictions.

  1. Reliability. The same claim moves the forecast by 5.4 pp when officially confirmed and by 0.2 pp when posted on an anonymous forum.
  2. Dependence. Three relays add only 0.1 pp beyond the original report, so they are not counted as new evidence.
  3. Independence. With the same headline and wording but their own sources, the same three outlets move the forecast 1.7 pp more than the relays (p < 0.05), so independent corroboration is counted.

To see where this behavior comes from, we also tested overlap detection directly. The archive's merge decisions show where this behavior comes from.

Follow-up reportsCountMerged into the originalRate
relay (citing the original)13212997.7%
indep (own sources)13200%
para (paraphrases)3636100%
dup (verbatim copies)132132100%
  1. BRIEF recognizes reports of the same message from the same source and merges them in the archive. All verbatim copies and paraphrases were merged, as were 129 of the 132 relays.
  2. BRIEF does not merge reports from independent sources; it uses them to strengthen the belief. None of the 132 independent reports was merged, although their headline and wording match the relays; they count as independent corroboration, which is the 1.7 pp independence gap above.

We will include this experiment and its findings in the revised paper.

Q3: How BRIEF’s total token consumption and inference cost compare with those of the baselines.

A3: We split cost into retrieval cost (Serper requests) and inference cost (input and output tokens), counting all model calls of each method, and report per-question costs on FutureX, which has the most varied and most demanding questions of the three benchmarks.

Table 2: Per-question cost, proportion of decisive pre-cutoff facts, and FutureX score. Scores are means of three runs (Table 2 of the paper).

MethodFutureX score ↑Serper requestsDecisive pre-cutoff factsInput tokensOutput tokensTotal tokens
BRIEF77.4533.55.1%250k51k301k
Open Deep Research63.0344.63.4%183k76k259k
DeerFlow69.5911.21.7%241k25k266k
GPT Researcher48.535.02.5%6k9k15k
ReAct + search57.6310.61.7%21k14k35k

More computation does not explain the gains. BRIEF uses 13–16% more tokens than DeerFlow and Open Deep Research but scores 7.86 and 14.42 points higher, respectively, and it issues fewer search requests and output tokens than Open Deep Research. Token use is similar across these three systems (259k–301k per question), yet their scores range from 63.03 to 77.45, so how the budget is spent matters more than its size. Part of BRIEF’s token use goes to leakage checks, which grow with the amount of retrieved content and cover both fetched pages and extracted facts (Appendix B.2). Its retrieval is also more productive: 5.1% of the facts it retrieves are decisive and available before the cutoff, against 3.4% for Open Deep Research and 1.7% for DeerFlow.

Q4: How the leakage controls are validated and whether BRIEF performs well on genuinely future events.

A4: We control leakage at two levels. First, the backbone’s knowledge cutoff (May 2025) precedes the forecast dates of FutureX and ForecastBench, so outcomes cannot come from the model’s training data. Second, retrieval does not rely on a single LLM judgment. The cutoff is taken from the benchmark metadata rather than inferred by the model, Serper restricts results to the cutoff, and result dates are screened against it. Because date filters alone still leak through updated pages and inaccurate timestamps [4, 5], BRIEF then checks each fetched page for later information and reports of the outcome, and checks each extracted fact again before it enters the evidence archive (Appendix B.2). LLM-based leakage checks are also the standard safeguard in recent forecasting agents [6, 1], and BRIEF applies them at both the page and the fact level. On Prophet Arena, the evidence is the snapshot collected at forecast time, which reproduces the information environment of the forecast and needs no filtering, and BRIEF also performs best there.

For evaluation on real future events, we have submitted BRIEF to the live FutureX evaluation. FutureX accepts forecasts until Wednesday of each week, the answers resolve over the following week, and scores are released the week after, about two weeks after submission. The scores will not be available within the rebuttal period, and we will report them in the revised paper.

Q5: Why driver and disconfirmer queries are separated and how the calibration parameters are estimated.

A5: The two roles differ in where the query starts, not in the direction of the evidence it returns. A driver query asks what could move the outcome away from the base rate. A disconfirmer query starts from the currently favored hypothesis, such as the status quo, and searches for evidence against it (Section 3.3). A driver query may return counterevidence, but only the disconfirmer role guarantees a deliberate search for it, which guards against confirmation bias [7]. When the two kinds of evidence conflict, the critic flags the conflict and the controller can probe which source to trust (Appendix C.3).

Platt scaling is applied only on ForecastBench, with α\alpha and β\beta fitted on 60 resolved ForecastBench questions disjoint from the 200 test questions. We will add these details to Section 3.5.

[1] Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs. 2026, ICML Workshop on Forecasting as a New Frontier of Intelligence.

[2] Information Value Theory. 1966, IEEE Transactions on Systems Science and Cybernetics.

[3] From Narrative to Auditable Forecasts: A Structured Scaffold for Agentic Forecasting. 2026, Findings of EMNLP.

[4] Pitfalls in Evaluating Language Model Forecasters. 2026, ICLR.

[5] Temporal Leakage in Search-Engine Date-Filtered Web Retrieval: A Retrospective Forecasting Case Study. 2026, ACL (Short Papers).

[6] AIA Forecaster: Technical Report. 2025, arXiv.

[7] Superforecasting: The Art and Science of Prediction. 2016, Random House.

Reviewer_DkXcOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an LLM-based forecasting framework that iteratively retrieves evidence and updates belief, with aggregation and post-hoc calibration. They backtested it on a subset of Future X, ForecastBench, and Prophet Arena benchmarks, and show it outperforms general purpose models, other search strategies, and deep research systems.

Critical strengths

The proposed framework outperforms across several major forecasting benchmarks. Although the improvements cannot be attributed clearly to belief revision, the overall results are promising.

Critical weaknesses
  1. The novelty relative to existing methods (especially BLF) is incremental. BRIEF combines several existing ideas such as Bayesian-inspired belief updating, aggregation, and calibration.

  2. Insufficient evidence supporting the main motivation. The motivation in the introduction is interesting: "How effectively do these systems turn the evidence they collect into a forecast?" However, the experiments do not sufficiently answer this question.

  • The main results in Table 2 primarily show that the overall BRIEF system outperforms the baselines. However, it does not control for evidence retrieval. BRIEF may simply retrieve more or better evidence than the baselines, rather than making better use of the same evidence. Also since BRIEF combines multiple techniques, these results can not conclude that the improvements come from better evidence processing. (While Section 5.3 partially addresses this concern, it only compares against one baseline on FutureX and does not control for retrieval budgets.)

  • Section 5.1 attempts to show that belief revision is better than presenting all evidence at once or showing it sequentially. However, the comparison is confounded because full BRIEF includes additional components other than belief revision. A cleaner ablation should hold other components fixed (e.g. also apply aggregation, platt scaling, and other refinement steps) and only vary how evidence is processed.

  1. FutureX contains only 59 questions. It seems too few to support improvements across four difficulty levels.

  2. Lack of several important implementation and evaluation details. Please see questions.

Questions for authors
  1. Given BLF is the closest Bayesian-inspired belief updating method, could you include that as a baseline as well?

  2. Which versions of the three benchmarks are used (what are the resolution date ranges)? Without these details, it is difficult to assess potential data leakage of whether the questions were resolved before the model's training cutoff.

  3. Platt scaling: which dataset is used to fit a and b? I did not find details about which validation sets are used.

  4. Bayesian-inspired revision: the authors mention that Equation (4) "guides the model’s reasoning", but the prompt in Appendix C.3 does not mention Bayesian updating or any mechanism derived from it. How is this implemented?

  5. How does iterative retrieval work with fixed evidence in Prophet Arena? The evidence appears to contain only 10 short summaries per event, which can easily fit into the model's context window. Why are multiple retrieval rounds in BRIEF necessary or beneficial in this setting?

  6. Tables 2 and 3 report different results for full BRIEF. Could authors clarify?

  7. How is per-level score SlS_l in Equation (7) calculated? Is it accuracy?

8 comments

Thank you for your valuable feedback to help us improve our paper. We have revised our paper based on your feedback. We detail our response below and please kindly let us know if our response addresses your concerns.

W1: Differences from BLF.

Response: We would like to explain three differences between BRIEF and BLF [1] and the contributions behind them.

We show that, with the evidence held fixed, better use of evidence improves accuracy. In BLF, the belief state is updated together with search, so its gains combine better retrieval with better use of evidence. We separate the two. On the same frozen corpus, changing only how the evidence is processed changes accuracy (Section 5.1). On the same frozen FutureX evidence, the BRIEF predictor improves on the benchmark prompt by 2.49 points with DeerFlow’s evidence and by 6.04 points with BRIEF’s (Table 7). Good research alone is not enough. With the same backbone, DeerFlow and Open Deep Research rank first and second when they gather their own evidence, but fifth and seventh of eight on Prophet Arena, where the evidence is fixed (Table 2).

We stop research when the evidence is sufficient, not when the outcome is certain. Part of a forecast’s uncertainty comes from the future and cannot be removed by search. BRIEF therefore continues only when a gap is searchable and could change the forecast, which is when acquiring the information has positive expected value [2]. Before this decision, a separate critic checks the belief for unsupported assumptions, conflicts, and repeated reports. The two cases in Appendix E show the distinction: research stops when the missing value is an unpublished statistic and continues when it is a retrievable price. Removing this control lowers the FutureX score by 29.29 points (Table 3). By contrast, BLF stops when the agent submits or reaches its step limit.

We extend belief states from binary probabilities to structured answers. BLF’s belief is a single probability for a binary question. BRIEF’s belief holds a provisional answer in the target’s format: an option, a set of options, a value, or probabilities for several non-exclusive markets. This structure guides the research roles, the sufficiency judgment, the aggregation, and the coherence constraints.

We further include a direct comparison against BLF, with the experimental setup and results presented in our response to Q1 below.

W2: Whether the experiments show that BRIEF uses the same evidence better.

Response: On Prophet Arena, every method reads the same frozen corpus, so the Prophet Arena results in Table 2 and the comparison in Section 5.1 already hold retrieval fixed. Section 5.1 compares three ways of using this corpus. One-shot integration reads the whole corpus at once and forecasts once. Sequential accumulation splits the corpus into three preset parts, reads one part per round, and forecasts anew after each round, carrying forward only the text of earlier rounds. In belief revision, every query reads the full corpus and extracts the facts that answer it, and each forecast revises the explicit belief. Only the use of evidence differs across the three settings; the forecast refinement stage (Stage 4 in Figure 1) is identical in all of them. Belief revision lowers Brier from 19.34 under one-shot integration and 18.13 under sequential accumulation to 16.52. Even without refinement it reaches 17.67, still below sequential accumulation with refinement (Table 3).

Section 5.3 controls retrieval by exchanging frozen evidence with DeerFlow, the strongest baseline on FutureX; a weaker baseline would make the comparison less informative. Within each evidence source, the two predictors see identical evidence. On DeerFlow’s evidence, the BRIEF predictor scores 71.42, against 68.93 for the benchmark-prompt predictor that DeerFlow uses. On BRIEF’s evidence, DeerFlow’s predictor reaches 70.86, below both full BRIEF (76.90) and the BRIEF predictor on DeerFlow’s evidence (Table 7). On FutureX, BRIEF uses only about 13% more inference tokens per question than DeerFlow (301k vs. 266k).

W3: Size of FutureX.

Response: We appreciate the reviewer’s concern and acknowledge that the limited sample size warrants caution when interpreting the results. Our evaluation covers one complete weekly round of 59 questions, consistent with FutureX’s weekly release format. Approximately four-fifths require multiple selections or open-ended predictions, and individual questions can involve establishing several quantities, as illustrated in Appendix E. We also report means and standard deviations over three runs: BRIEF achieves a higher mean score than the evaluated baselines with low run-to-run variability. This provides evidence that the observed advantage is reproducible on this evaluation set, although repeated runs do not replace evaluation on additional questions.

Q1: BLF as a baseline.

A1: We have added BLF with the same retrieval as the other methods. Because DeepSeek has retired the V4-Flash API, BLF runs on its successor, DeepSeek-V4.1-Flash, and binary questions use the original BLF prompts verbatim. Since BLF supports only binary questions, we changed only the form of its belief for other targets: a probability per option or outcome for choice questions and Prophet Arena events, and weighted candidate answers for open-ended questions, with a median and a 10–90% interval for numeric ones. For open-ended and numeric questions, we also gave BLF a list of candidate answers, which BRIEF does not receive.

Table 1: Comparison of BLF and BRIEF across three forecasting benchmarks.

MethodFutureX ↑ForecastBench Brier ↓Prophet Arena Brier ↓
BLF66.7012.7417.50
BRIEF76.9013.0116.73

BLF is slightly better on ForecastBench, the binary benchmark it was designed for and evaluated on, where BRIEF is within 0.27 Brier of it. On FutureX and Prophet Arena, where targets go beyond a single binary probability, BRIEF is clearly better, even though BLF received candidate answers for open-ended questions. Both the candidate answers and the newer backbone favor BLF.

Q2: Benchmark versions and leakage.

A2: BRIEF and all method baselines share the backbone DeepSeek-V4-Flash (released April 24, 2026; knowledge cutoff May 2025). The FutureX questions come from the week with an evidence cutoff of July 29, 2026. The ForecastBench questions have a forecast date of April 12, 2026 and resolve between April 13 and July 21, 2026. Both are well after the knowledge cutoff. Prophet Arena uses the frozen corpus of Prophet-Arena-Subset-100. Any residual knowledge in the backbone is available to every method alike and does not favor BRIEF.

Q3: Data used to fit Platt scaling.

A3: Platt scaling is applied only on ForecastBench. α\alpha and β\beta are fitted on 60 resolved ForecastBench questions that do not overlap with the 200 test questions.

Q4: How the Bayesian-inspired revision is implemented?

A4: Eq. (4) specifies the structure of each revision rather than a numerical computation. Although the prompt does not state Bayes' rule, each of its terms is implemented through the belief state, the revision prompt, and the evidence archive.

The prior P(Y∣Et)P(Y \mid E_t) is the current belief StS_t. Revision starts from StS_t rather than from scratch and records why the forecast changes or stays the same (Appendix C.3), and the revised belief becomes the prior for the next round.

The likelihood P(ΔEt+1∣Y,Et)P(\Delta E_{t+1} \mid Y, E_t) is conditioned on what is already known. The prompt asks the model to judge each new observation against the existing evidence, deciding whether it addresses the critic's concerns, contradicts earlier evidence, or adds new information. A report already covered by EtE_t has a likelihood ratio near one and should not move the belief. The archive therefore removes duplicates and flags overlapping reports, the critic lists repeated evidence, and the prompts forbid counting dependent reports as independent support.

We do not compute the update numerically. In our experiments, explicit Bayesian updating was less accurate than letting the model revise its belief directly, as BLF also found with LLM-estimated likelihoods, and it requires an enumerated hypothesis space that open-ended numeric and ranking targets lack.

Q5: Iterative retrieval on Prophet Arena.

A5: Prophet Arena has no search engine. Each research call, whether an initial subquestion or a targeted query, reads the event’s full frozen corpus (about 10 sources, at most 24k characters) and extracts the facts that answer it. The rounds are therefore not needed to fit the corpus into context; they direct the reading. A targeted query is written for a specific gap or conflict found by the critic, so the model rereads the same sources with a narrower question and is more likely to surface the details that bear on it.

Having the whole corpus in context does not ensure that it is used well: models underuse information depending on where it appears in the input [3]. One-shot integration, which sees every source at once, gives the worst forecasts in Table 5 (Brier 19.34). On Prophet Arena, removing the critic, role guidance, or sufficiency control raises Brier by 1.91, 0.57, and 0.78, respectively (Table 3). Directed rounds therefore help even when the corpus is small.

Q6: Different values for full BRIEF in Tables 2 and 3.

A6: Table 2 reports the mean of three runs, whereas Table 3 reports the single run against which the ablations are compared.

Q7: Per-level score SℓS_\ell in Eq. (7).

A7: SℓS_\ell is the mean question score at level ℓ\ell, reported in the L1–L4 columns of Table 2, and SFutureXS_{\text{FutureX}} in Eq. (7) is the Score column.

[1] Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs. 2026, ICML Workshop on Forecasting as a New Frontier of Intelligence.

[2] Information Value Theory. 1966, IEEE Transactions on Systems Science and Cybernetics.

[3] Lost in the Middle: How Language Models Use Long Contexts. 2024, TACL.

Reviewer_Vy7zOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The classic approach to building forecasting agents consists of a research-judgment split. In the research, data from the world is collected. In the judgment step an agent converts the research into a probability. In other words, with all the context in the window, the agent is asked to produce a final probability. In this paper, the authors propose a new approach called BRIEF where the agent iteratively processes the evidence and then use a Bayesian-like updating process to update probabilities as new evidence comes out.

Critical strengths

Strengths

  • First of all, I want to say that this paper is wonderfully written. Sentences are clear and non-empty (read as non AI generated)
  • The paper also does a rigorous exploration of different components of the forecasting stack, keeping research fixed. It shows clear improvements that can be derived from harness improvements alone
Critical weaknesses

Weaknesses

  • The main weakness was some difficult in communicating parts of the methodology. See questions below.
  • In Section 3.3 we can do a better job basically writing out that those queries differ between question type (as shown in Table 8).
Questions for authors

Questions:

  • My biggest struggle was understanding how E_0 was created. My understanding is that we have a variable number of prompts that guide the evidence in a few different preliminary directions. Then all of those beliefs are aggregated into the evidence state E_0 (if so, should that be the text for “Evidence state” in panel 2 of Figure 1?). The in the BRIEF setting, when the agent does a targeted query it searches over the full corpus event state E_t? Or does it do additional tool use? Is there a tool you’ve built out to search over the full corpus of information? If they are able to call additional external tools, then how do we conduct cost-constrained forecasts?
  • “identifies the relevant deciding organization, authoritative sources, and status quo” (pdf) -> Why does the forecast frame identify the organizations, authoritative sources, and status quo
  • Is there a way to see differences in the cost compared with the different baselines.
  • I’m a little bit confused about the 3rd mode. “When additional evidence arises” meanwhile before they say they keep the evidence fixed? How is new evidence returned?
4 comments

Thank you for your valuable feedback to help us improve our paper. We have revised our paper based on your feedback. We detail our response below and please kindly let us know if our response addresses your concerns.

Q1: How E0E_0 is formed and how a targeted query is executed.

A1: E0E_0 is the evidence collected by the initial research step, which is deliberately broad. The planner generates up to six subquestions guided by the research roles: base rate, drivers, disconfirmers, market anchors, and the route-specific roles in Table 8. Each subquestion is researched separately, and the returned facts are merged and deduplicated into E0E_0. What is combined at this stage is evidence, not beliefs. The initial belief S0S_0 is the preliminary understanding and judgment that the model forms about the question after reading E0E_0 (Eq. 1). In panel 2 of Figure 1, EtE_t is the evidence at t=0t = 0; labeling it E0E_0 is clearer, and we will do so.

A targeted query rtr_t is executed by the same research function and retriever as the initial subquestions, and no other tool is called. On FutureX and ForecastBench, the retriever is built on Serper. On Prophet Arena there is no search engine, and the model reads the event’s frozen corpus directly.

Within targeted retrieval, EtE_t is used at two points only:

  1. The controller decides what to query next based on the current belief and the evidence in EtE_t, and asks only for information that EtE_t does not already contain.
  2. The facts returned by the query are deduplicated and merged into EtE_t to form Et+1E_{t+1}.

Retrieval cost is capped per question. Initial subquestions, direct source access, and targeted queries share a budget of B=20B = 20 units (Table 10; Appendix C.1). Because one research call may be split into several sub-queries, a unit is not the same as a single search request. The research loop runs for at most five rounds and stops earlier when the controller judges the evidence sufficient, and the final refinement stage makes no retrieval calls.

Q2: Why the frame identifies the deciding organization, authoritative sources, and the status quo.

A2: These elements help the model judge which factors may affect the outcome, which sources to trust, and where the forecast should start.

Deciding organization. Many outcomes are set by a specific body, and identifying it helps the model analyze the factors that may affect the outcome. A rate decision, for instance, depends mainly on what the central bank’s committee weighs, such as inflation, employment, and its own forward guidance.

Authoritative sources. Retrieved material ranges from official releases to social media. When sources conflict, the model should favor the authoritative one, so the frame lists authoritative sources and every piece of evidence keeps its source (Section 3.4). In the CGN Power case (Appendix E.2), the critic rejected inconsistent price snippets and sources that confused billions with hundreds of millions of yuan. This follows the likelihood term in Eq. (4). In a simple model, a report from a source that is correct with probability ρ\rho multiplies the prior odds by ρ/(1−ρ)\rho/(1-\rho), so an official source (ρ≈1\rho \approx 1) outweighs a social-media post (ρ≈0.5\rho \approx 0.5). BRIEF does not estimate ρ\rho explicitly; the source labels let the critic make this judgment.

Status quo. In an election or referendum, such as the FutureX question “Will the 2025 Guinea referendum pass?”, the state keeps changing until resolution, so the state at the cutoff is a more precise starting point than any earlier one. If Xearlier→Xnow→YX_{\text{earlier}} \to X_{\text{now}} \to Y is a Markov chain, the data processing inequality gives I(Y;Xnow)≥I(Y;Xearlier)I(Y; X_{\text{now}}) \ge I(Y; X_{\text{earlier}}).

Q3: How BRIEF’s computational and retrieval costs compare with those of the baselines.

A3: We split cost into retrieval cost (Serper requests) and inference cost (input and output tokens). The table below reports the per-question cost of each method on FutureX, which we chose because it has the most varied question types and more demanding questions than the other two benchmarks.

Table 1: Per-question retrieval and inference costs and the proportion of decisive pre-cutoff facts on FutureX.

MethodSerper requestsDecisive facts available before the cutoffInput tokensOutput tokens
BRIEF33.55.10%250k51k
Open Deep Research44.63.40%183k76k
DeerFlow11.21.70%241k25k
GPT Researcher52.50%6k9k
ReAct + search10.61.70%21k14k

BRIEF’s cost is comparable to that of the deep research systems. It uses 13–16% more tokens in total than DeerFlow and Open Deep Research, and fewer search requests and output tokens than Open Deep Research. Its token count also includes the model calls that check retrieved content for leakage (Appendix B.2). The added reasoning yields better retrieval and better forecasts: 5.1% of the facts BRIEF retrieves are decisive for the question and available before the cutoff, compared with 3.4% for Open Deep Research and 1.7% for DeerFlow, and BRIEF’s FutureX score is 14.42 and 7.86 points higher than theirs, respectively (Table 2).

Q4: Where new evidence comes from in the third mode when the corpus is fixed.

A4: No new information enters from outside the corpus. In Section 5.1, all three modes read the same frozen Prophet Arena corpus for each event. The new evidence in the third mode is the set of facts that each BRIEF query extracts from this corpus. These facts are new relative to what BRIEF has already collected in EtE_t, not new to the corpus.

The three modes differ only in how they read it. One-shot integration reads the whole corpus at once and forecasts once. Sequential accumulation splits the corpus into three preset parts, reads one part per round, and forecasts anew after each round, carrying forward only the text of earlier rounds. In belief revision, every query reads the full corpus and extracts the facts that answer it. What is carried forward is the explicit belief StS_t, and each forecast is formed by revising StS_t with the new facts.

Reviewer_Vy7zOpenReview ↗

Response

Thank you for this feedback! I will keep my score.

Thank you for taking the time to read our response, and for your encouraging assessment of our paper. Your questions helped us see where our explanation fell short, and the paper is clearer because of them. If anything else comes to mind, we would be glad to continue the discussion at any time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.