- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper treats verifier-free forecasting as a test-time compute-allocation problem. On 228 label-balanced, post-cutoff ForecastBench questions, and under one shared contamination filter, it compares static depth (Predictor–Critic), static breadth (Ensemble + Aggregator), and adaptive delegation (Hierarchical Orchestrator). In the gpt-5.4-mini sweep, the best Orchestrator setting reaches 80.7% accuracy and a Brier score of 0.153 at 0.175 dollar per question, against 78.1% at 0.90 dollar for the best Ensemble and 76.8% at 1.67 dollar for the best Predictor–Critic. Paired tests do not resolve those accuracy gaps.
A two-stage diagnostic correlates uncertainty from a separate direct run with orchestrator spend and with paired Brier improvement. The validation is a full configuration sweep on gpt-5.4-mini and gpt-5.4-nano, an Orchestrator-only scan on three further models, cost measured in dollars, tokens, and calls, McNemar and bootstrap intervals, and a 200-page human audit of the filter.
Critical strengths
The contamination audit and the complete configuration grids are strong enough to carry a measurement paper. Filter precision, residual leakage, and the 83-95% filter share are reported with intervals, and Tables 11-12 let a reader recompute the frontier. This is the part I trust. Extra critic rounds and extra ensemble members mostly buy more page screening, and accuracy inside each family saturates. I would still use that measurement if the causal story were dropped.
The authors also leave a 2-4 point accuracy gap unresolved, which is the right call at N = 228. Table 17 McNemar tests, and reporting cost in several proxies, are what make the mini result interpretable.
Critical weaknesses
The comparison does not isolate allocation. On mini, Predictor–Critic at T=2, E=0 scores 64.5%, while Orchestrator direct mode at D=0, T=2 scores 76.3%. Both are a single agent with two tool calls and no critic or delegation. That 12-point gap is larger than every between-architecture gap in the abstract. Ensemble workers and the critic also use a fixed tool budget M=5, while orchestrator workers use T=1 or T=2. The Pareto result can therefore come from a stronger base scaffold together with a smaller retrieval schedule. Their claim is that the schedule caused the gap. This comparison does not show that.
The main claims are also false on the paper’s own tables. The winning setting, D=2, T=2, uses 9 of 9 theoretical calls, so the early-stopping story in Section 5 does not apply to the result it explains. On nano, the joint accuracy frontier includes Predictor–Critic and Ensemble points, and Ensemble remains better on raw and Platt Brier (Table 14). Stage 1 is a rank correlation of about 0.14 on three of five models, computed from a separate run, with no multiplicity correction. The mechanism and replication claims in the abstract would have to survive those tables. They do not.
Questions for authors
- Are the prompt, parser, and tool loop for Predictor–Critic T=2, E=0 and Orchestrator D=0, T=2 the same? If they differ, where? I wish to know whether the 64.5% versus 76.3% gap is an allocation effect.
- Is the 9/9 call figure in Table 2 an exact per-question maximum, or a rounded mean? What fraction of questions stop before D delegations at the winning setting?
- Were the reported best settings chosen on these 228 questions? If you already have a held-out or cross-fit selection, the frontier should be the one from that split.
- Which calls put Ensemble K=1, R=2 at 108% of the worst-case ceiling in Table 2?
- Did you run any randomized or forced budget on these same questions? I am asking whether that analysis already exists.
2 comments
Q1. Shared components and differences in the original prompts
The original configurations share the parser, model-call wrapper and retrieval filter, but differ in their full prompts and context. The PC prompt requests a longer answer and more searches than the configured allowance permits, but the implementation enforces the tool budget. The direct Orchestrator states the limit explicitly in its prompt. The follow-up using a shared forecasting pipeline makes Direct, Ensemble K1, PC E0 and Orchestrator D0 use the same forecasting call. The Orchestrator-prompt comparison separately transfers the Orchestrator’s forecasting instructions to static workflows with revisions.
Q2. The rounded call count and actual stopping
Nine is a rounded mean: historical D2 T2 averages 8.991 agent calls and always uses two delegations. Its cost variation tracks completion tokens much more closely than call count. More broadly, the Orchestrator does not always exhaust its available delegation budget: at D5 T2, 80.7% of questions stop below the larger delegation cap. This shows that the controller can leave budget unused, but early stopping does not explain the winning D2 setting. Early stopping is one aspect of the Orchestrator’s compute management.
Q3. Configuration selection on held-out partitions
Yes, the original best settings were selected on the evaluation questions. We added a check that selects configurations on the other partitions and scores them on the held-out partition. Mini D2 T2 is selected in every fold and retains 80.7% accuracy, compared with 74.1% for Ensemble and 74.6% for PC. This strengthens the mini result against configuration-selection optimism within the tested grid, while remaining an internal check on the existing benchmark.
Q4. What the reported 108% represents
The 108% compares 25.87 recorded calls with a nominal 24-call schedule. This schedule was a reference for accounting, rather than the limit enforced by the implementation. Each of the three predictor passes and the aggregator could use up to ten loop generations followed by a final-answer generation, allowing up to 44 agent generations per question. The recorded count therefore does not exceed the implemented limit.
Q5. New task content and continuation comparisons
We added a comparison in which both versions use two sub-agents, with either tasks tailored by the Orchestrator to each question or fixed tasks. A separate comparison starts from A’s saved first sub-agent state and produces the forecast without further delegation. Tasks specific to the question have lower observed Brier than fixed tasks, and continuation improves Brier relative to stopping. The cost columns show how much additional computation accompanies that improvement. These are exploratory task content and continuation contrasts.
Thank you for your time and thoughtful feedback. We have expanded our experiments to address differences in forecasting instructions and to examine task selection and the value of additional delegation.
W1. Additional controls for the forecasting scaffold
We added two complementary controls. With a shared forecasting pipeline, the Orchestrator improves on Direct from 69.91% to 76.39% accuracy and from 0.2165 to 0.1719 Brier. We also transferred the Orchestrator’s forecasting instructions to PC and Ensemble workflows at two tool-budget levels. Across all attempted forecasts, both Orchestrator references retain higher accuracy and lower Brier scores.
Comparing architectures using a shared forecasting pipeline: The methods share the base model, core forecasting instructions, and retrieval tools, while differing in how they organize and combine forecasting calls.
Orchestrator prompts at two budget levels: How competitive are PC and Ensemble when given the original Orchestrator’s forecasting instructions at two budget levels?
W2. Separating stopping, model quality and the mechanism
We will distinguish these claims. Historical D2 T2 uses both delegations, so its advantage does not come from early stopping. Nano shows a cost–quality trade-off, while mini D2 T2 remains on both agent-cost frontiers with corrected probability interpretation. Accounting for comparisons across models weakens the uncertainty–compute association, so we treat it as diagnostic evidence. The task selection and additional delegation comparison adds more direct, exploratory evidence on task content and continuation, rather than proof of optimal allocation.
Task selection and additional delegation: The Orchestrator creates tasks specific to the question by tailoring each sub-agent’s research instructions to the forecasting question and any earlier sub-agent findings. We compare these with fixed tasks and test whether further delegation after the first sub-agent improves forecasts.