acceptodds
Under review as a conference paper at ICLR 2027

How Does Test-Time Compute Affect Agentic Forecasting?

Abstract

Test-time compute scaling has been explored extensively in easily verifiable domains such as math and code. However, much less is known about how the test-time budget should be spent for forecasting future events, where no test-time verifier is available. We study the problem of compute allocation for *agentic forecasting* by comparing three multi-agent policies that spend inference-time compute in different ways: static depth (*Predictor-Critic*), static breadth (*Ensemble + Aggregator*), and adaptive routing (*Hierarchical Orchestrator*). We perform a contamination-controlled backtesting evaluation on label-balanced binary questions from ForecastBench partitions, all resolved strictly after every base model's knowledge cutoff, under a shared web-filtering layer that screens retrieved pages for forecast leakage and post-resolution contamination. In the gpt-5.4-mini sweep, adaptive routing occupies the entire cost-accuracy Pareto frontier: the top Orchestrator reaches accuracy at USD per question, while the best Ensemble and Predictor-Critic reach at USD per question and at USD per question, respectively. The same Pareto ordering replicates on the gpt-5.4-nano sweep, where the Orchestrator is about cheaper than the top Ensemble, with no statistically significant accuracy gap. The Orchestrator's cost-quality advantage further extends to other base models, including Claude Sonnet 4.5, DeepSeek v4 Flash, and Gemini 2.5 Flash, so the result is not a single-model artifact. A two-stage mechanism diagnostic explains this win as *selective delegation*: on three of five base models, the Orchestrator allocates more compute to questions where its cheap direct baseline is uncertain, and this uncertainty can predict where extra compute helps. Adaptive routing wins not by spending less on every question, but by concentrating compute on the ones where it pays off. We release the benchmark, architecture implementations, pipeline, and analysis code.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test-time compute, forecasting, LLM multi-agent systems, adaptive compute allocation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_ZBBSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper treats verifier-free forecasting as a test-time compute-allocation problem. On 228 label-balanced, post-cutoff ForecastBench questions, and under one shared contamination filter, it compares static depth (Predictor–Critic), static breadth (Ensemble + Aggregator), and adaptive delegation (Hierarchical Orchestrator). In the gpt-5.4-mini sweep, the best Orchestrator setting reaches 80.7% accuracy and a Brier score of 0.153 at 0.175 dollar per question, against 78.1% at 0.90 dollar for the best Ensemble and 76.8% at 1.67 dollar for the best Predictor–Critic. Paired tests do not resolve those accuracy gaps.

A two-stage diagnostic correlates uncertainty from a separate direct run with orchestrator spend and with paired Brier improvement. The validation is a full configuration sweep on gpt-5.4-mini and gpt-5.4-nano, an Orchestrator-only scan on three further models, cost measured in dollars, tokens, and calls, McNemar and bootstrap intervals, and a 200-page human audit of the filter.

Critical strengths

The contamination audit and the complete configuration grids are strong enough to carry a measurement paper. Filter precision, residual leakage, and the 83-95% filter share are reported with intervals, and Tables 11-12 let a reader recompute the frontier. This is the part I trust. Extra critic rounds and extra ensemble members mostly buy more page screening, and accuracy inside each family saturates. I would still use that measurement if the causal story were dropped.

The authors also leave a 2-4 point accuracy gap unresolved, which is the right call at N = 228. Table 17 McNemar tests, and reporting cost in several proxies, are what make the mini result interpretable.

Critical weaknesses

The comparison does not isolate allocation. On mini, Predictor–Critic at T=2, E=0 scores 64.5%, while Orchestrator direct mode at D=0, T=2 scores 76.3%. Both are a single agent with two tool calls and no critic or delegation. That 12-point gap is larger than every between-architecture gap in the abstract. Ensemble workers and the critic also use a fixed tool budget M=5, while orchestrator workers use T=1 or T=2. The Pareto result can therefore come from a stronger base scaffold together with a smaller retrieval schedule. Their claim is that the schedule caused the gap. This comparison does not show that.

The main claims are also false on the paper’s own tables. The winning setting, D=2, T=2, uses 9 of 9 theoretical calls, so the early-stopping story in Section 5 does not apply to the result it explains. On nano, the joint accuracy frontier includes Predictor–Critic and Ensemble points, and Ensemble remains better on raw and Platt Brier (Table 14). Stage 1 is a rank correlation of about 0.14 on three of five models, computed from a separate run, with no multiplicity correction. The mechanism and replication claims in the abstract would have to survive those tables. They do not.

Questions for authors
  • Are the prompt, parser, and tool loop for Predictor–Critic T=2, E=0 and Orchestrator D=0, T=2 the same? If they differ, where? I wish to know whether the 64.5% versus 76.3% gap is an allocation effect.
  • Is the 9/9 call figure in Table 2 an exact per-question maximum, or a rounded mean? What fraction of questions stop before D delegations at the winning setting?
  • Were the reported best settings chosen on these 228 questions? If you already have a held-out or cross-fit selection, the frontier should be the one from that split.
  • Which calls put Ensemble K=1, R=2 at 108% of the worst-case ceiling in Table 2?
  • Did you run any randomized or forced budget on these same questions? I am asking whether that analysis already exists.
2 comments

Thank you for your time and thoughtful feedback. We have expanded our experiments to address differences in forecasting instructions and to examine task selection and the value of additional delegation.

W1. Additional controls for the forecasting scaffold

We added two complementary controls. With a shared forecasting pipeline, the Orchestrator improves on Direct from 69.91% to 76.39% accuracy and from 0.2165 to 0.1719 Brier. We also transferred the Orchestrator’s forecasting instructions to PC and Ensemble workflows at two tool-budget levels. Across all attempted forecasts, both Orchestrator references retain higher accuracy and lower Brier scores.

Comparing architectures using a shared forecasting pipeline: The methods share the base model, core forecasting instructions, and retrieval tools, while differing in how they organize and combine forecasting calls.

MethodAccuracyRaw Brier
Direct T269.91%0.2165
Arithmetic-mean Ensemble K272.56%0.1875
Predictor–Critic E172.2%0.1915
Orchestrator D2 T276.39%0.1719
  • The Orchestrator improves on Direct by 6.48 percentage points in accuracy and lowers Brier from 0.2165 to 0.1719.
  • The Orchestrator has the highest accuracy and lowest Brier among the figures shown.

Orchestrator prompts at two budget levels: How competitive are PC and Ensemble when given the original Orchestrator’s forecasting instructions at two budget levels?

ConfigurationAccuracyBrier lossCost per question (USD)
PC E176.67%0.1895USD 0.0971
PC E271.67%0.2324USD 0.1993
Ensemble K2 R173.33%0.1965USD 0.0859
Ensemble K2 R273.33%0.2438USD 0.1626
Orchestrator D2 T281.67%0.1320USD 0.1687
Orchestrator D5 T283.33%0.1153USD 0.3013
  • Across all attempted forecasts, both Orchestrator settings have higher accuracy and lower Brier scores than the revised static workflows.
  • Additional static revision rounds do not improve these outcomes. D2 T2 also costs less than PC E2, while PC E1 and both Ensemble configurations cost less than D2 T2.

W2. Separating stopping, model quality and the mechanism

We will distinguish these claims. Historical D2 T2 uses both delegations, so its advantage does not come from early stopping. Nano shows a cost–quality trade-off, while mini D2 T2 remains on both agent-cost frontiers with corrected probability interpretation. Accounting for comparisons across models weakens the uncertainty–compute association, so we treat it as diagnostic evidence. The task selection and additional delegation comparison adds more direct, exploratory evidence on task content and continuation, rather than proof of optimal allocation.

Task selection and additional delegation: The Orchestrator creates tasks specific to the question by tailoring each sub-agent’s research instructions to the forecasting question and any earlier sub-agent findings. We compare these with fixed tasks and test whether further delegation after the first sub-agent improves forecasts.

MetricA: tasks specific to questionB: fixed tasksC: stop after A’s first sub-agent
Accuracy87.50%75.00%81.25%
Brier0.11920.14180.1786
Agent USD/questionUSD 0.01366USD 0.01124USD 0.00703
Recorded agent and filter cost (USD/question)USD 0.13789USD 0.09862USD 0.08791
  • Task selection: tasks specific to the question produce lower Brier than fixed tasks, 0.1192 versus 0.1418, at higher recorded cost.
  • Additional delegation: continuing from A’s first sub-agent state produces lower Brier than stopping there, 0.1192 versus 0.1786.
  • Quality gained for the extra cost: continuation lowers Brier by 0.0595 for approximately USD 0.050 more recorded agent and filter cost per question.

Q1. Shared components and differences in the original prompts

The original configurations share the parser, model-call wrapper and retrieval filter, but differ in their full prompts and context. The PC prompt requests a longer answer and more searches than the configured allowance permits, but the implementation enforces the tool budget. The direct Orchestrator states the limit explicitly in its prompt. The follow-up using a shared forecasting pipeline makes Direct, Ensemble K1, PC E0 and Orchestrator D0 use the same forecasting call. The Orchestrator-prompt comparison separately transfers the Orchestrator’s forecasting instructions to static workflows with revisions.

Q2. The rounded call count and actual stopping

Nine is a rounded mean: historical D2 T2 averages 8.991 agent calls and always uses two delegations. Its cost variation tracks completion tokens much more closely than call count. More broadly, the Orchestrator does not always exhaust its available delegation budget: at D5 T2, 80.7% of questions stop below the larger delegation cap. This shows that the controller can leave budget unused, but early stopping does not explain the winning D2 setting. Early stopping is one aspect of the Orchestrator’s compute management.

Q3. Configuration selection on held-out partitions

Yes, the original best settings were selected on the evaluation questions. We added a check that selects configurations on the other partitions and scores them on the held-out partition. Mini D2 T2 is selected in every fold and retains 80.7% accuracy, compared with 74.1% for Ensemble and 74.6% for PC. This strengthens the mini result against configuration-selection optimism within the tested grid, while remaining an internal check on the existing benchmark.

Q4. What the reported 108% represents

The 108% compares 25.87 recorded calls with a nominal 24-call schedule. This schedule was a reference for accounting, rather than the limit enforced by the implementation. Each of the three predictor passes and the aggregator could use up to ten loop generations followed by a final-answer generation, allowing up to 44 agent generations per question. The recorded count therefore does not exceed the implemented limit.

Q5. New task content and continuation comparisons

We added a comparison in which both versions use two sub-agents, with either tasks tailored by the Orchestrator to each question or fixed tasks. A separate comparison starts from A’s saved first sub-agent state and produces the forecast without further delegation. Tasks specific to the question have lower observed Brier than fixed tasks, and continuation improves Brier relative to stopping. The cost columns show how much additional computation accompanies that improvement. These are exploratory task content and continuation contrasts.

Reviewer_sXZLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper is a study of 3 different inference time scaling and compute allocation approaches for using agents to forecast: Actor-Critic, best-of-N with aggregation, and an adaptive MetaGPT/Autogen style with an orchestrator and sub agents. Their goal is to do a indepth analysis of the pareto frontier of these techniques with respect to cost on a curated set of 228 binary questions which involve future forecasting. They find that the agentic adaptive routing approach occupies the pareto frontier for each model in their sweep across different hyperparameters and they propose a selective-delegation-hypothesis which supposes that the orchestrator assigns compute to questions which correlate with its uncertainty and this correlate with improvement.

Critical strengths

One strength is the absolute depth of the details provided in the main paper. Many papers would have left many of the details regarding the filter or the actual hyperparameter sweep missing or sweeped over them, which can lead to the results being iffy.

I think that the depth of the exploration of the smaller dataset is to the benefit of the paper. By focusing on these 228 tasks, I think that the paper has very thoroughly explored all of the different phenomena in this dataset, and has highlighted a lot of different trends and behaviors with agentic test time scaling that are underexplored or tribal knowledge in the field.

Critical weaknesses

The main critical weakness is the lack of additional datasets or frontier models explored in the paper. On their extremely limited 228 tasks, they did not explore any larger GPT models or other models except in Sec 4.5 and in Appendix S, where the results are extremely limited and partial. Furthermore, despite the dataset being part of ForecastBench, authors never try to expand their analysis to larger subsets of forecast bench, or other forecast benchmarks. This is significant, as without these, this is essentially just an indepth analysis of a custom 228 subset of ForecastBench on mostly 2 tiny gpt models without any significant extensions to other models or datasets. Their hypothesis and results that are derived in the paper do not have sufficient grounding in empirical evidence without further experimentation with other models and it is unclear if these observations will scale to larger datasets.

Questions for authors

Unclear from the manuscript how important this contamination safeguard is to the work. While it is an important component to do this exact experiment in the paper, in the realworld, I would assume that these kinds of leakage would be impossible as the events just haven't occured yet in real time. If so, then the discussion might be better summarized in the main work and extended in the appendix, if the current detail is just to cover all bases.

As a followup, is the cost of the filtering itself considered in the analysis in Figure 2 and beyond? I would think that this should not be considered, as at test-time, this will not a concern.

This manuscript makes a case on a "verifier-free" regime, but isn't this a property of the entire field of agentic forecasting?

Furthermore, the statements in the abstract and main paper that the agentic orchestrator occupies the entire pareto frontier does not seem to be true. Clearly in Figure 2, the Ensambling method dominates at larger costs, and it is unclear to me as the reviewer for these datasets whether a 2% difference in accuracy constitutes a significant or not accuracy gap. The abstract states that this is not a "statistically significant gap"

3 comments

Thank you for your time and constructive review. We have expanded our experiments and clarified the distinction between agent costs and the cost of retrospective filtering.

W1. Dataset and model coverage

Full architecture sweeps cover mini and nano, with Orchestrator-only runs for the other three models. Within the existing benchmark, mini D2 T2 retains 80.7% accuracy under held-partition configuration selection, and the accuracy ordering survives the forecast-cutoff check. Given the limited rebuttal period and ongoing experiments, we are happy to extend the matched comparisons in a later version, or run and share a predefined 50-question evaluation on one base model if helpful.

Q1. The role of contamination filtering

Yes. Outcome-leakage filtering is a retrospective safeguard because backtests can retrieve information published after resolution. Excluding other forecasters’ predictions is a separate design choice. Our human audit found residual contamination, so we do not claim complete removal. Restricting to questions whose forecast cutoff precedes resolution preserves the mini accuracy ordering. We will distinguish model knowledge cutoffs, forecast cutoffs and resolution dates explicitly.

Q2. Separating agent cost from filtering cost

Yes, Figure 2 includes the model cost of filtering retrieved evidence. For the architecture comparison, we distinguish agent-only costs from combined agent and filter costs. With filter costs excluded and no-tool baselines included, mini D2 T2 remains on both the accuracy and Brier frontiers, as discussed in Q4. The new delegation comparison also reports both cost measures: Brier falls from 0.1786 to 0.1192 for approximately USD 0.00662 more agent cost, or USD 0.050 more recorded agent and filter cost, per question. Both cost measures exclude external search fees.

Task selection and additional delegation: The Orchestrator creates tasks specific to the question by tailoring each sub-agent’s research instructions to the forecasting question and any earlier sub-agent findings. We compare these with fixed tasks and test whether further delegation after the first sub-agent improves forecasts.

MetricA: tasks specific to questionB: fixed tasksC: stop after A’s first sub-agent
Accuracy87.50%75.00%81.25%
Brier0.11920.14180.1786
Agent USD/questionUSD 0.01366USD 0.01124USD 0.00703
Recorded agent and filter cost (USD/question)USD 0.13789USD 0.09862USD 0.08791
  • Task selection: tasks specific to the question produce lower Brier than fixed tasks, 0.1192 versus 0.1418, at higher recorded cost.
  • Additional delegation: continuing from A’s first sub-agent state produces lower Brier than stopping there, 0.1192 versus 0.1786.
  • Quality gained for the extra cost: continuation lowers Brier by 0.0595 for approximately USD 0.050 more recorded agent and filter cost per question.

Q3. What “no test-time verifier” means

The term describes a forecasting constraint, not a new task category. Outcomes are unavailable when the system decides whether to retrieve, delegate or stop. Our contribution is to compare forecast quality and expenditure across compute policies under that constraint. The shared forecasting pipeline and continuation comparisons provide concrete evidence in this setting, without claiming that the absence of a verifier causes the architecture differences.

Comparing architectures using a shared forecasting pipeline: The methods share the base model, core forecasting instructions, and retrieval tools, while differing in how they organize and combine forecasting calls.

MethodAccuracyRaw Brier
Direct T269.91%0.2165
Arithmetic-mean Ensemble K272.56%0.1875
Predictor–Critic E172.2%0.1915
Orchestrator D2 T276.39%0.1719
  • The Orchestrator improves on Direct by 6.48 percentage points in accuracy and lowers Brier from 0.2165 to 0.1719.
  • The Orchestrator has the highest accuracy and lowest Brier among the figures shown.

Q4. Frontier claims by model and metric

We will state the frontier result by model and metric. After adding no-tool baselines, mini D2 T2 remains on the agent-cost accuracy and Brier frontiers under the corrected and text-supported probability readings. On nano, D3 T2 remains on the Brier frontier, but no routed Orchestrator remains on the accuracy frontier. These results support efficient operating points, not an Orchestrator-only frontier. We will describe nano as a cost–quality trade-off rather than claim equal quality.

I thank the authors for their response. While I find the results of the study to be thorough and in-depth, the paper, I believe, is missing some additional experiments on larger datasets and/or with stronger models to see if these insights generalize, or if they are specific to this scale of models and/or these datasets in particular. Due to this, I believe that the results in the paper are not significant enough for this venue, so I will keep my overall rating, but I will raise my soundness score.

Thank you again for taking the time to review our paper, consider our responses, and provide further feedback.

To address your concern about model coverage, we ran additional experiments with four newer models on 50 recently resolved ForecastBench questions. Given the limited time available during the discussion period, we focused on this smaller extension. We report the results for the configurations below.

GPT-6.2 Astra

ConfigurationAccuracyRaw BrierLLM cost (USD/question)
PC E264.29%0.15830.49429
Ensemble K2 R275.00%0.15520.44627
Orchestrator D2 T273.68%0.14270.25404

Claude Opus 5.5

ConfigurationAccuracyRaw BrierLLM cost (USD/question)
PC E283.33%0.10070.57396
Ensemble K2 R280.00%0.14300.53669
Orchestrator D2 T275.00%0.13130.27927

Gemini 4 Carbon

ConfigurationAccuracyRaw BrierLLM cost (USD/question)
PC E268.75%0.15020.60794
Ensemble K2 R280.00%0.13260.68953
Orchestrator D2 T275.00%0.17740.36735

DeepSeek V4 Pro

ConfigurationAccuracyRaw BrierLLM cost (USD/question)
PC E276.47%0.17050.43270
Ensemble K2 R275.00%0.16300.41084
Orchestrator D2 T270.59%0.21000.21541

Across all four models, Orchestrator D2 T2 had a lower mean LLM cost per question than PC E2 and Ensemble K2 R2. Forecast quality rankings varied by model, and the small sample limits what we can conclude about these differences, and whether they are statistically significant. This extension provides an initial check on newer models and more recent questions, but we agree that it does not fully address your concern about dataset size. We plan to evaluate all configurations from the paper on a larger set of questions to assess how broadly the findings generalize. Thanks.

Reviewer_JKhmOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how test-time compute should be allocated in LLM-based forecasting, where predictions cannot be verified at inference time. It compares three agent architectures representing static depth, static breadth, and adaptive routing. Experiments show that adaptive routing achieves a more favorable cost–accuracy trade-off, and the authors further analyze this advantage through prediction uncertainty and compute allocation.

Critical strengths

The paper addresses a practically relevant problem and provides a systematic comparison of different agent architectures.

Critical weaknesses
  1. The methodological novelty is relatively incremental, as adaptive compute allocation and the three agent architectures are largely based on existing approaches. The main contribution is an empirical comparison in a forecasting setting, but the paper does not clearly establish what fundamentally new insights this setting provides beyond prior work.

  2. The theoretical analysis mainly relies on standard arguments and descriptive empirical observations, rather than providing new theoretical insights or predictions. In particular, the proposed explanation for the benefits of adaptive routing is primarily supported by correlations, making it difficult to distinguish an underlying mechanism from a post-hoc interpretation of the experimental results.

Questions for authors
  1. What new insights about test-time compute allocation emerge specifically from the verifier-free forecasting setting compared with existing adaptive inference research?
  2. Could the authors compare the Orchestrator against a simple uncertainty-based routing strategy that allocates additional compute when the initial forecast is uncertain? This would help determine whether hierarchical agent coordination is necessary for the observed gains.
  3. Can the authors provide more direct evidence that selective delegation is responsible for the observed efficiency gains, beyond the reported correlations? For example, could a controlled comparison with randomized compute allocation help validate this explanation?
4 comments

Thank you for your time and careful review. We have expanded our experiments and the new results provide concrete comparisons of task selection and continuation.

W1. An empirical contribution on how to use compute

Our contribution is to compare ways of spending inference compute when forecast correctness cannot be checked immediately. The follow-ups examine simple averaging, repeated critique, peer revision and adaptive delegation. The internal comparisons then measure the quality associated with task choice and the additional cost of continuing after the first sub-agent.

Comparing architectures using a shared forecasting pipeline: The methods share the base model, core forecasting instructions, and retrieval tools, while differing in how they organize and combine forecasting calls.

MethodAccuracyRaw Brier
Direct T269.91%0.2165
Arithmetic-mean Ensemble K272.56%0.1875
Predictor–Critic E172.2%0.1915
Orchestrator D2 T276.39%0.1719
  • The Orchestrator improves on Direct by 6.48 percentage points in accuracy and lowers Brier from 0.2165 to 0.1719.
  • The Orchestrator has the highest accuracy and lowest Brier among the figures shown.

Orchestrator prompts at two budget levels: How competitive are PC and Ensemble when given the original Orchestrator’s forecasting instructions at two budget levels?

ConfigurationAccuracyBrier lossCost per question (USD)
PC E176.67%0.1895USD 0.0971
PC E271.67%0.2324USD 0.1993
Ensemble K2 R173.33%0.1965USD 0.0859
Ensemble K2 R273.33%0.2438USD 0.1626
Orchestrator D2 T281.67%0.1320USD 0.1687
Orchestrator D5 T283.33%0.1153USD 0.3013
  • Across all attempted forecasts, both Orchestrator settings have higher accuracy and lower Brier scores than the revised static workflows.
  • Additional static revision rounds do not improve these outcomes. D2 T2 also costs less than PC E2, while PC E1 and both Ensemble configurations cost less than D2 T2.

Task selection and additional delegation: The Orchestrator creates tasks specific to the question by tailoring each sub-agent’s research instructions to the forecasting question and any earlier sub-agent findings. We compare these with fixed tasks and test whether further delegation after the first sub-agent improves forecasts.

MetricA: tasks specific to questionB: fixed tasksC: stop after A’s first sub-agent
Accuracy87.50%75.00%81.25%
Brier0.11920.14180.1786
Agent USD/questionUSD 0.01366USD 0.01124USD 0.00703
Recorded agent and filter cost (USD/question)USD 0.13789USD 0.09862USD 0.08791
  • Task selection: tasks specific to the question produce lower Brier than fixed tasks, 0.1192 versus 0.1418, at higher recorded cost.
  • Additional delegation: continuing from A’s first sub-agent state produces lower Brier than stopping there, 0.1192 versus 0.1786.
  • Quality gained for the extra cost: continuation lowers Brier by 0.0595 for approximately USD 0.050 more recorded agent and filter cost per question.

W2. Theory as framing, followed by empirical tests

We use the Bayes-risk decomposition to frame the empirical analysis. It separates limits imposed by available evidence from errors in using that evidence, but does not predict which architecture wins. Our contribution is empirical. The new comparisons find lower observed Brier with tasks tailored by the Orchestrator to each forecasting question and with continuation after the same first sub-agent. These exploratory results examine task content and continuation, rather than establish better allocation across questions.

Q1. Compute decisions before outcomes are known

Forecasting requires decisions about retrieval, delegation and stopping before the outcome is known. We measure the quality and cost of alternative workflows under this constraint. The shared forecasting pipeline comparison shows a gain over Direct, and the internal comparison quantifies the extra cost of useful continuation. We do not claim that these findings are unique to forecasting without a matched comparison against tasks with immediate correctness checks.

Q2. A simple uncertainty-routing baseline

We tested an uncertainty rule that chooses between Direct and Orchestrator forecasts, selecting the rule on development partitions before evaluation. It did not improve on random routing: Brier was 0.21456 versus 0.20301, with closely matched expenditure. Matching combined agent and filter spending gives the same conclusion. This external rule differs from the controller’s internal task choices. Uncertainty can be associated with improvement without producing a better routing policy.

Q3. An intervention at the continuation decision

We added a continuation comparison that starts from the same saved first sub-agent state. Using a second sub-agent before the final forecast reaches 0.1192 Brier, compared with 0.1786 when producing the final forecast immediately. The additional recorded cost is approximately USD 0.050 per question. Separately, tasks tailored by the Orchestrator to each forecasting question outperform fixed tasks on Brier. These comparisons examine task content and continuation within the controller’s workflow.

Thank you for the detailed response and additional experiments. The new comparisons are useful, particularly the uncertainty-based routing baseline and the ablations on task selection and continuation, which address several of my concerns. However, the main contribution remains an empirical comparison of existing agentic strategies, without clearly establishing new insights beyond prior adaptive compute allocation research. The theoretical analysis also remains largely descriptive, and the additional experiments do not fully explain the underlying mechanism behind the observed efficiency gains. Therefore, I will keep my score unchanged.

Thank you again for taking the time to review our paper, consider our response, and provide further feedback. Appreciate it.

Reviewer_Z6TEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to allocate test-time compute in event forecasting, where no correctness signal is available at inference time. Using a shared retrieval-and-filtering pipeline, the authors compare static depth (Predictor–Critic), static breadth (Ensemble), and dynamic delegation (Orchestrator) on 228 label-balanced ForecastBench questions. The experiments include architecture sweeps with gpt-5.4-mini (mini) and gpt-5.4-nano (nano), along with selected Orchestrator configurations on three additional models. The Orchestrator achieves a favorable cost–quality trade-off on mini and provides a low-cost option on nano. Increasing the budget does not necessarily improve forecasts, while retrieval and filtering dominate expenditure. The authors further use correlation analyses to examine selective delegation as an explanation for these gains.

Critical strengths
  1. The problem is important, and the efficiency gains have practical value.

The paper extends the study of compute allocation to forecasting tasks without an immediate verification signal. On mini, the Orchestrator achieves good forecast quality at a lower cost, providing useful empirical evidence for forecasting systems with limited budgets.

  1. The evaluation is reasonably comprehensive, with attention to probabilistic quality and contamination control.

The authors assess forecast quality using accuracy, probabilistic scores, cross-fitted calibration, and paired tests. They also examine filter effectiveness through a metadata-based audit and a blinded human audit of 200 web pages. Reporting residual contamination further improves the transparency of the evaluation.

  1. The cost analysis identifies a concrete direction for system optimization.

The paper separates agent-side expenditure from retrieval-and-filtering costs, finding that the latter account for 83–95% of total expenditure in the mini sweep. This identifies the main bottleneck in the backtesting system and points to reducing retrieval-side costs as an optimization direction.

Critical weaknesses
  1. Implementation differences and baseline coverage limit attribution to the architectures themselves.

On mini, Orchestrator D=0, T=2 achieves 76.3% accuracy, whereas Predictor–Critic T=2, E=0 achieves 64.5% (Table 11). Neither configuration involves delegation or critique, and both have the same tool budget, yet they differ by 11.8 percentage points. The effects of prompts, context, stopping rules, and run-to-run variability need to be clarified before the gains attributable to compute allocation itself can be assessed.

The main experiments also lack a lightweight ensemble with no revision rounds, simple aggregation, and a low retrieval budget. Costs are not reported for the no-tool baselines in Table 21, where nano with high reasoning effort already reaches 71.9% accuracy. These comparisons would help establish how competitive static approaches are in the low-cost regime.

  1. Claims of cross-model superiority extend beyond the comparisons provided.

On nano, the Ensemble achieves 72.8% accuracy and a raw Brier score of 0.179, compared with the Orchestrator’s 71.1% and 0.209. These results reflect a cost–quality trade-off; a statistically nonsignificant accuracy difference does not establish equivalence in forecast quality. Only the Orchestrator is evaluated on the other three models, supporting its applicability across models but not its superiority over static architectures. The claims should be limited to the comparisons actually performed.

  1. Evidence for the selective-delegation mechanism remains preliminary.

Section 5 correlates uncertainty from a separate direct-baseline run with delegation expenditure and Brier improvement. This does not establish that the router actually uses this signal. Uncertain questions may benefit more even from a fixed additional budget because they have greater room for improvement. Moreover, the best mini configuration’s mean number of agent-side LLM calls per question and its reported theoretical maximum are both nine. The respective contributions of dynamic subtask selection and early stopping therefore remain to be distinguished.

  1. The findings and conclusions are intuitive and offer no further insights.

The main findings—that adaptive allocation can save compute, retrieval and filtering dominate expenditure, and additional reasoning over fixed evidence has limited benefits—are broadly intuitive. The experiments provide quantitative validation of these observations, but do not sufficiently clarify the test-time scaling behavior specific to verifier-free forecasting. The additional insight beyond these intuitions remains limited.

Questions for authors
  • Why do the two configurations that reduce to a single agent in Table 11 differ by 11.8 percentage points? Do differences in prompts, context, stopping rules, or retry policies also carry over to the main experiments?
  • Could the authors add a representative ensemble baseline with no revision rounds, simple aggregation, and a low retrieval budget, and report costs for the no-tool baselines, to assess the efficiency advantage in the low-cost regime?
  • How are delegation counts distributed, and how often does the best configuration stop early? Are per-question cost differences driven mainly by the number of calls or by the number of tokens per call? This would help distinguish the effects of task selection from those of allocating different amounts of compute.
2 comments

Thank you for your time and helpful feedback. We have expanded our experiments to address differences in prompting, evaluate simple static baselines, and examine the value of additional delegation.

W1a. Forecasting instructions and the architecture comparison

We added PC and Ensemble workflows using the original Orchestrator’s forecasting instructions at two tool-budget levels. Across all attempted forecasts, both Orchestrator references retain higher accuracy and lower Brier scores. This extends the comparison to static baselines built around the Orchestrator’s forecasting instructions.

Orchestrator prompts at two budget levels: How competitive are PC and Ensemble when given the original Orchestrator’s forecasting instructions at two budget levels?

ConfigurationAccuracyBrier lossCost per question (USD)
PC E176.67%0.1895USD 0.0971
PC E271.67%0.2324USD 0.1993
Ensemble K2 R173.33%0.1965USD 0.0859
Ensemble K2 R273.33%0.2438USD 0.1626
Orchestrator D2 T281.67%0.1320USD 0.1687
Orchestrator D5 T283.33%0.1153USD 0.3013
  • Across all attempted forecasts, both Orchestrator settings have higher accuracy and lower Brier scores than the revised static workflows.
  • Additional static revision rounds do not improve these outcomes. D2 T2 also costs less than PC E2, while PC E1 and both Ensemble configurations cost less than D2 T2.

W1b. An Ensemble with simple averaging

We added an Ensemble that averages two independent forecasts without revision rounds or an LLM aggregator. It improves on Direct, while the Orchestrator improves further, reaching 76.39% accuracy and 0.1719 Brier. For comparison, PC E1 is at 72.2% accuracy and 0.1915 Brier.

Comparing architectures using a shared forecasting pipeline: The methods share the base model, core forecasting instructions, and retrieval tools, while differing in how they organize and combine forecasting calls.

MethodAccuracyRaw Brier
Direct T269.91%0.2165
Arithmetic-mean Ensemble K272.56%0.1875
Predictor–Critic E172.2%0.1915
Orchestrator D2 T276.39%0.1719
  • The Orchestrator improves on Direct by 6.48 percentage points in accuracy and lowers Brier from 0.2165 to 0.1719.
  • The Orchestrator has the highest accuracy and lowest Brier among the figures shown.

W2. What the cross-model comparisons establish

Full architecture sweeps cover mini and nano. The other three models have Orchestrator runs only, so they show its behaviour across models rather than superiority over static methods. For the nano configurations compared in the paper, Ensemble achieves better forecast quality than the Orchestrator, but costs substantially more. The Orchestrator’s advantage in this comparison is its lower cost.

W3. Task choice and the value of continuation

We added two complementary comparisons inside the Orchestrator workflow. Tasks tailored by the Orchestrator to each forecasting question achieve lower Brier than fixed tasks, 0.1192 versus 0.1418. Continuing after the same first sub-agent also achieves lower Brier than stopping, 0.1192 versus 0.1786. Stopping is cheaper, which lets us quantify the quality gained for the additional delegation.

Task selection and additional delegation: The Orchestrator creates tasks specific to the question by tailoring each sub-agent’s research instructions to the forecasting question and any earlier sub-agent findings. We compare these with fixed tasks and test whether further delegation after the first sub-agent improves forecasts.

MetricA: tasks specific to questionB: fixed tasksC: stop after A’s first sub-agent
Accuracy87.50%75.00%81.25%
Brier0.11920.14180.1786
Agent USD/questionUSD 0.01366USD 0.01124USD 0.00703
Recorded agent and filter cost (USD/question)USD 0.13789USD 0.09862USD 0.08791
  • Task selection: tasks specific to the question produce lower Brier than fixed tasks, 0.1192 versus 0.1418, at higher recorded cost.
  • Additional delegation: continuing from A’s first sub-agent state produces lower Brier than stopping there, 0.1192 versus 0.1786.
  • Quality gained for the extra cost: continuation lowers Brier by 0.0595 for approximately USD 0.050 more recorded agent and filter cost per question.

W4. What the new results add

The additional comparisons in W1 to W3 characterize the observed quality gains. The shared forecasting pipeline comparison shows a gain over Direct, the prompt-transfer comparison includes static methods with revisions, and the internal comparisons connect better forecasts with tasks tailored to each forecasting question and further delegation. This gives concrete evidence on task content and continuation, alongside their recorded costs.

Q1. Why the original direct configurations differ

The original direct modes differ in instructions, context placement and answer structure. These differences also carry into the multi-agent workflows. We therefore added a shared forecasting pipeline and a separate Orchestrator-prompt comparison. The Orchestrator improves on Direct, while the Orchestrator references retain better outcomes in the Orchestrator-prompt comparison. The follow-ups reduce specific implementation differences without making historical prompts or retrieved evidence identical.

Q2. Ensemble averaging and no-tool costs

Yes. The shared forecasting pipeline comparison includes the arithmetic-mean Ensemble. We also recovered the no-tool costs: mini with medium reasoning reaches 74.6% accuracy at USD 0.003863 per question, and nano with high reasoning reaches 71.9% at USD 0.000837. With corrected probability interpretation, mini D2 T2 remains on both agent-cost accuracy and Brier frontiers, while no-tool settings occupy the cheapest region. The Ensemble costs in the Orchestrator-prompt comparison include aggregation.

Q3. Delegation counts, stopping and token use

Historical D2 T2 uses both delegations on every question, so early stopping does not explain that result. Its cost variation mainly tracks completion tokens. At D5 T2, 80.7% of questions stop below the delegation cap, illustrating early stopping as one aspect of the Orchestrator’s compute management. The separate continuation comparison shows lower observed Brier for approximately USD 0.050 more recorded cost per question. This measures the value of continuation in that comparison, rather than proving that the controller stops optimally.

Comments on the paper

1 comment

For all revierers: as the discussion period is approaching its end, please join the discussion and put acknowledge or feedback on authors' rebuttal, thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.