acceptodds
Under review as a conference paper at ICLR 2027

Hedge-Bench: Can Agents Reason Their Way to Better Predictions?

Abstract

Modern AI agents are increasingly capable of mechanical financial-analysis tasks such as retrieving documents and updating spreadsheets. A harder challenge is forecasting an unknowable future from incomplete information. Public markets provide a natural testbed: analysts routinely predict discrete events from point-in- time information. We present Hedge-Bench, a benchmark of 92 real-world tasks derived from hedge fund analysts’ reasoning traces and paired with subsequently resolved prediction questions. Agents reason from frozen information snapshots, and their forecasts are evaluated against realized outcomes. Across 11 frontier models and three trials per task, joint success rates range from 2.54 to 31.52%, requiring both a correct forecast and over 60% research coverage. Using GRPO with turn-level credit assignment, we train Muse-Glimmer on 165 environments and increase coverage on 22 held-out tasks from 17.17% to 33.36% at low reasoning effort and from 30.19% to 47.02% at extra-high effort. The trained model at low effort also exceeds the base model’s extra-high-effort coverage while using 47.9% fewer accepted trajectory tokens. We also observe evidence of generalization: held-out forecasting accuracy at high effort improves by 10.06 percentage points, while external evaluations improve by 3.94 points on GDPval, 2.29 on GDP.pdf, 7.07 on Harvey LAB, and 4.96 on Frontier Finance under their respective metrics.

Primary area:
datasets and benchmarks
Keywords:
Reinforcement Learning, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pfMwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Hedge-Bench, built from recorded conversations between pairs of buy-side analysts. Each conversation is turned into a hidden rubric of “analytical moves” plus a separate binary forecasting question that has since resolved. Agents work in an offline, point-in-time container of filings, transcripts and prices, write a cited report and give a Yes/No forecast. A three-model LLM jury scores rubric coverage after a citation-grounding check, and joint success requires a correct forecast plus more than 60% coverage. Eleven frontier models reach 2.5% to 31.5% joint success on the 92 public tasks. The authors then train Muse-Glimmer-30B with TurnCredit, a GRPO variant where a judge assigns signed per-turn credit after the rollout. This improves held-out coverage by about 16 points over the base model and by 12 to 15 points over a magnitude-matched control, and it gives modest gains on GDPval, GDP.pdf, Harvey LAB and Frontier Finance.

Critical strengths

Strength 1. The source material is the real asset here. Rubrics come from unscripted analyst debates rather than LLM-written checklists, the environments are frozen at a cutoff with no internet, and the grounding step (a move is discounted if a majority flags its citation) is a good guard against confident fluff. I find this a much better target than the usual retrieval-style finance QA.

Strength 2. The TurnCredit ablations are better than usual. The magnitude-matched control (Table 2) directly addresses the obvious objection that turn weights just shrink the advantage, and the trajectory length, repetition and entropy curves in Figure 4 tell a consistent story.

Critical weaknesses

Weakness 1. The forecasting half does not support the question in the title. Within model and environment, coverage and correctness are basically uncorrelated (-0.12 to 0.12, nothing survives Holm, Section 4). Question-only accuracy is already 56 to 67% against a 53% majority baseline (Table 16), so a good part of the “forecasting skill” comes from priors before any document is read. The recency argument (later resolutions score higher, Table 17) does not rule out contamination. It is a comparison across different tasks. On Hedge-Bench-Private every model is below the constant-Yes baseline, including GPT-5.6 Sol at 35.85% (Table 28), and the trained model’s “+10 points” moves it from 38% to 48%, still under 52.8%. A benchmark where the strongest model does worse than a coin with a Yes bias on fresh questions makes me want to look first at how the questions were built or selected. I don’t buy the “genuinely more difficult” explanation. Statistically, the outcome is binary with no probabilities (no Brier, no calibration), the 92 questions cluster in 23 companies, and the permutation tests in H.1/H.2 only shuffle trials within fixed environments, so they ignore task-level sampling. With 22 held-out tasks that matters.

Weakness 2. The coverage gains are close to training on the test metric. The attribution judge sees the hidden rubric and the grader’s verdict on which moves were accepted and missed (Section 5.1), and the synthesis reward requires “substantial accepted coverage”. Held-out tasks share companies (and many source files, 730 distinct contents across 2,967 copies) with the 70 public training tasks, and the reward judge (Gemini 3.7 Flash) is a sibling of one of the jurors. So a 16-point coverage jump on Heldout mostly shows that the policy learned what this jury rewards. The external benchmarks help, but gains of 2 to 5 points on judge-scored rubrics are a weaker form of evidence than the abstract suggests.

Two minor things. How tasks were “handpicked” for difficulty out of 5,112 candidates is not specified. If model performance was involved, that biases the frontier numbers. The human validation uses 8 reports and 81 shared decisions with 70% inter-expert agreement. That is thin for anchoring a 1,150-move rubric.

Questions for authors

Q1. Why does every model fall below the majority class on Private? Is there something systematic about how forecast questions are phrased, for example drawn from issues the analysts expected to resolve against consensus?

Q2. Can you report probabilistic forecasts with Brier or log score, and confidence intervals that resample tasks (clustered by company) rather than trials only?

Q3. How was candidate difficulty assessed during selection, and were any of the evaluated models used for it?

Q4. What happens to Heldout coverage if the attribution judge does not see the rubric verdict, or if Gemini is dropped from the evaluation jury?

5 comments

Response to Reviewer pfMw (1/2)

Thank you very much for the detailed review, and for recognizing the analyst-derived rubrics, the grounding step and the magnitude-matched control. We newly added the clustered intervals, evaluation without Gemini and the Brier scores you asked for (Appendices N.1, N.2 and N.6), and we answer Q1 and Q3 directly.

W1. "The forecasting half does not support the question in the title." Thank you for the careful reading. We read the title's question as having two answers, and the paper measures both. [AP] We agree that coverage and forecast correctness are not correlated; it is the reason we evaluate them separately (Conclusion). The paper reports this in Section 4 ("Research coverage and forecast accuracy can differ"): within the same model and task, reports with higher coverage are not more likely to forecast correctly (-0.123 to 0.119 as pointed out by you). We also show this qualitatively in Appendix L.2 (Table 37): on Fiserv, one Muse Spark 1.3 output has 21.43% coverage and a correct forecast, and another has 71.43% and an incorrect one. The Flywire pair also shows the same. We then train a model on research and forecast together: the reward uses the synthesis judge, which expects both a good forecast and thorough research coverage. For example, the synthesis judge must score a wrong forecast below 0.5 (AP, Appendix F.1). On Private, whose companies are absent from training, forecast accuracy at high effort rises from 38.36% to 48.43% and coverage rises by 16.44 points (AP, Table 26); the newly added company-clustered 95% CI of the +10.06 forecast gain is [2.38, 17.95] (N.1). On Heldout, the base model already scores 19/22 at pass@8 (NeA, N.6), so the gain shows in joint success (a correct forecast with coverage above 60%): from at most 4 of 176 reports to between 14 and 34 across four efforts (AP, Table 30).

Priors (AP). As you note, the question alone beats the majority baseline, but the documents add to it: ten of eleven models forecast better with the full environment, by 2.2 to 16.3 points (Figure 6).

Contamination (AP). We agree Table 17 compares different tasks. Table 18 keeps only outcomes made public after each model's documented knowledge cutoff and compares the same questions with and without the documents. They still help five of the six models with such a cutoff, by 2.9 to 12.7 points (GPT-5.6 Sol: 63.73% against 50.98% on 34 tasks).

Finally, the analysts' conversations are gathered on a day-to-day basis, so we can avoid contamination.

Q1. Why is every model below the majority class on Private? Is something systematic in the questions? Thank you for the question. We answered why every model is below the majority class on Private in our overall comment.

  • Selection. The Private questions were built with the same procedure as the other splits and then set aside as a split for unseen companies (AP, Section 2.3). We didn’t intend to make them more difficult than other sets. Each question and label was only verified for correctness, and Section 2.1 now states that the set was fixed before any model was evaluated (NeA). The outcomes nevertheless exhibit a pattern similar to the one you describe. We provide ten qualitative examples in Appendix O (NeA), which lists eight forecasts in which GPT-5.6 Sol answers No and the outcome is Yes.

  • Some misses are numerically close: For example, FIS reported adjusted free-cash-flow conversion of 88.02%, above the 85% top of the range management had reiterated. Sol forecasts approximately 84% and answers No. In another example, Sol forecasts 62 basis points, while the reported 65 exceed the 63-basis-point threshold.

  • Scope of the effect. On public questions disclosed after its knowledge cutoff, Sol still scores 63.73% (AP, Table 18), so recency alone does not explain 35.85% on Private.

Following your comment, H.2.1 now cites this evidence for its "more difficult" statement (NeA). Sol's research coverage is also lower on Private (45.33%, against 59.17% on Public; AP, Table 28).

Response to Reviewer pfMw (2/2)

Q2. Probabilistic forecasts and company-clustered intervals. Thank you for both suggestions; both are newly added.

  • Clustered intervals (NeA, N.1). As you suggested, we computed company-clustered numbers. At high effort the forecast gain is +10.06 [2.38, 17.95] on Private and +5.68 [1.56, 10.23] on Heldout. Coverage gains under the jury without Gemini and with company-clustered intervals are in the table below (NeA, N.2, Table 45). Private coverage improves for all 11 companies (AP, H.2.1).

  • Brier loss (NeA, N.6). Now models report P(Yes) on all 75 Heldout and Private environments. On Private, training reduces the loss by 0.047 [0.023, 0.068], from 0.389 to 0.342, level with GPT-5.6 Sol at 0.341. Sol numbers are close to Muse-Glimmer-Hedge even though its accuracy is about 13 points lower, because Sol has many near misses. On easier Heldout all three models are near 0.19.

Q3. How was difficulty assessed, and were evaluated models used? The conversations and transcripts are manually checked by experts on a day-to-day basis, and no model was used in the selection of the environments. All the conversations go through a difficulty check that makes sure they are of good quality. We chose the 92 public environments by hand from the filtered set. Section 2.1 now reads: "We assess candidate difficulty manually, without using any model, and two independent human reviewers verify task quality [...] Both the public and private task sets were fixed before any model was evaluated on them."

W2 / Q4. "The coverage gains are close to training on the test metric." We understand the concern and appreciate the chance to address it. We check in four ways whether the policy only learned what this jury rewards.

Test 1: jury without Gemini (NeA, N.2). A move is credited only if both GPT-5.6 Sol and Claude Opus 5 mark it present:

Split, effortCoverage gain [company-clustered 95% CI]
Heldout, low+16.57 [13.36, 21.37]
Heldout, high+17.15 [11.54, 20.81]
Private, low+12.30 [9.55, 15.26]
Private, high+14.71 [11.99, 17.41]

The Heldout gains are within 0.5 points of the three-juror gains.

Test 2: new companies (AP, Table 26). Private shares no company with training; coverage rises by 13.26 and 16.44 points.

Test 3: external benchmarks (AP, Appendix H.5, Tables 32 and 33). After only 21 updates, one pass over the 165 Hedge-Bench training environments (AP, Section 5), the external gains go beyond 2 to 5 rubric points:

  • GDPval, blind pairwise comparison. The trained model beats the base model in 149 to 172 of 220 comparisons at each effort and loses 46 to 70. Its Elo rises by 108.7 to 167.1 (Table 33; [NeA] Table 52 lists wins, losses and ties per effort).
  • Harvey LAB (legal tasks, two judges). +7.13 under Claude Sonnet 4.6 and +7.02 under GPT-5.5. [NeA] N.7 adds the task-level count: 73 of 100 tasks improved.
  • Rubric scores. All seven external rubric gains have 95% intervals above zero (Table 32).

Test 4: what improves (NeA, N.3 and N.4). On Heldout the gains concentrate in comparisons, counterarguments and causal inference, and 295 of 522 move-credit gains (against 121 losses) are in themes where the base report earned nothing. N.4 shows the qualitative changes in the reports themselves.

Attribution judge without the rubric verdict. The magnitude-matched ablation addresses this question. The magnitude-matched control optimizes the same synthesis score with the same advantage magnitude and gains 2.6 to 4.5 points over the base model, compared to 16.2 to 17.4 for TurnCredit (Table 2), so the rubric-aware reward alone yields little. The verdict the attribution judge sees always concerns a training task, and the four tests above indicate that what the policy learned from it is not specific to this jury.

Minor: human validation (AP, Appendix D.3 and Table 9).

The jury agrees with each expert on 63 of 81 shared decisions (77.8%), above the 57 of 81 (70.4%) between the experts, and on the 57 requirements where the experts agree it matches them on 51 (89.5%, kappa 0.76). In total the two experts made 197 judgments on 110 requirements. We spent about $2,000 (for 20 hours) on this expert review, and a larger one would cost much more. Since the result is consistent across both experts, we hope it is reasonable to stop here.

Thanks again for the constructive feedback and for helping us improve the paper! We hope this resolves your concerns, and we would be glad to discuss any remaining point. If it does, we would be grateful if you would consider updating your rating.

Dear Reviewer pfMw, Thank you again for your detailed review. Since the discussion period is about to end (November 18th), we wanted to follow up on our response. Since then, we have also posted an independent check of the training judge by two human readers and a model jury (in our reply to Reviewer rqFN). Please let us know if our response adequately addresses your concerns. If so, we would kindly ask you to consider updating your score. If not, we are willing to engage further to strengthen the paper. An early reply would give us enough time to work on any remaining points.

Reviewer_9sG6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Hedge-Bench, which separately evaluates an agent’s reasoning process and the accuracy of its predictions based on actual discussions among investment analysts. It also proposes TurnCredit, which trains agents based on the contribution of each turn in their reasoning process. Using test sets, the paper verifies and reports on the improvement of the agent's reasoning process and its generalization to other specialized tasks.

Critical strengths
  • Using a unique dataset created from actual expert discussions, the paper evaluates whether forecasting agents performed the necessary analyses and whether their predictions were accurate, while also proposing a learning method. This represents a significant contribution to the field.
  • Agents trained using TurnCredit to address tasks in the financial sector have also been reported to improve performance in other specialized fields, such as law, demonstrating the method’s potential as a more general-purpose learning approach.
Critical weaknesses
  • The accuracy rate of predictions made by agents trained using TurnCredit still falls below the score achieved by always answering “yes”, making the claim that predictive capability has been sufficiently enhanced less convincing. Furthermore, there is a lack of analysis regarding the specific mechanisms by which the agents’ imitation of human analytical processes actually aids in prediction. Has such an analysis been conducted?
  • In the proposed method, there is a constraint that requires the penalty for a missed prediction to be assigned to at least one turn. However, even the best prediction may miss the actual correct answer. The proposed learning method carries the risk of penalizing even appropriate reasoning processes. While the current results demonstrate the effectiveness of TurnCredit, questions remain as to whether this is the correct design. What is the rationale for imposing this constraint?
Questions for authors

See critical weaknesses

3 comments

Response to Reviewer 9sG6

Thank you for the supportive review, for recognizing the expert-derived data and the transfer to other professional domains, and for appreciating the presentation.

1. Accuracy "still falls below the score achieved by always answering 'yes'". Why models forecast worse on Private is answered in Appendix N.5 [NeA] and qualitative examples are provided in Appendix O. Please check our comment to all reviewers. We provide additional probabilistic forecasts as well to strengthen our claim that the training helped in forecasting. Brier loss on Private falls by 0.047 (95% CI [0.023, 0.068]; Table 51).

2. "Has such an analysis been conducted?" We have added a new qualitative analysis in Appendix N.4 and quantitative ones in N.3 and N.5.

N.4 compares base and trained reports on the same Heldout environments:

  • Playboy, margin question. Credited moves rise from 4 of 16 to 11 of 16. The trained report separates long-run margin improvement from next-quarter seasonality and investment spending, and its forecast changes from Yes to the correct No.
  • Playboy, licensing share. The trained report checks the comparison base, finds the licensing step-up already in the year-ago quarter, and changes Yes to the correct No (6 of 12 to 8 of 12 moves).
  • On Private. The move type that gains most is counterarguments and risks (+20.33 points [17.48, 22.22], Table 48), and on the eleven single-report questions that resolved Yes, training corrects six of the base model's 33 wrong No answers (N.5). We also provide qualitative examples of how Sol fails on Private.

3. "What is the rationale for imposing this constraint?" Thank you so much for raising this question. This is one of the moments which highlighted the importance of Metaprompting. The rule was introduced by the MetaPrompt agent (Codex here) that produced the judge prompts, during prompt development, after it observed that the judge could then assign no penalty to an incorrect forecast when the supporting research was strong. As you note, a sound forecast can still miss. [NeA] We ran a replay credit assignment with and without the blame rule to measure the MetaPrompt agent's judgement.

  • The penalty is there either way; the rule redistributes it appropriately (N.8 and Table 53). We replayed the judge on 100 of the 660 wrong-forecast rollouts with and without the rule, two calls each. Either way, every rollout receives blame and keeps positive credit. Without the rule, 102 of 200 judgments blame only the report-writing turn and with it only 3 do, and blamed turns per judgment rise only from 1.55 to 2.06. Therefore the rule helped in correcting credit distribution. Of the 84 turns blamed only under the rule, two independent LLM readers classified 76 as the turn where the model chooses or justifies its forecast, and 3 as evidence gathering or planning.

We are glad you raised this question!

Thank you again for the thoughtful questions. If you believe our responses appropriately addressed your concerns, please consider raising the score!

Thank you for the detailed response and the new analyses. I now understand the empirical benefits of TurnCredit, and most of my concerns have been addressed. I also think that the analysis comparing settings with and without the blame rule has clarified the role of this rule.

Thank you for the response and thoughtful questions, which helped strengthen the paper. We are glad that most of the concerns are addressed!

Reviewer_rqFNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes Hedge-Bench. Each task gives an agent the documents of one company up to a cutoff date. The agent writes a research report and answers a yes or no question that resolves after the cutoff. The report is scored against a hidden list of analytical steps. The list comes from discussions between working analysts. The paper tests 11 frontier models, and the best joint success rate is about 32 percent. The paper also proposes TurnCredit. It uses an LLM judge to give each turn of a trajectory a positive or negative score for RL. After training, research coverage rises clearly.

Critical strengths
  1. The benchmark design is new and carefully built. It scores research quality and forecast accuracy apart within the same task. The reference comes from real analyst discussions, not from rules set by the authors. The jury used for evaluation is checked against human experts, and its agreement with each expert is no lower than the agreement between the experts. The finding that the two scores are almost unrelated is also useful in itself.
  2. The training part shows some clear gains. On held out tasks, research coverage and joint success rise clearly, and the trained model reaches higher coverage with less reasoning. It also gives small gains on several other professional tasks.
Critical weaknesses
  1. The data is fairly narrow and still far from a real research setting.
  • On the documents: the environment only has documents issued by the company. There is no third party news, no regulatory documents and no industry data.
  • On the rubric and the forecast: the rubric keeps only steps that these documents can support. But the answer to the forecast question often depends on the missing information. The paper finds that coverage and forecast correctness are almost unrelated. This fits the same view. I suggest the paper explain which part of real research this setting represents.
  1. The method mostly follows existing practice, and several key parts are not evaluated enough.
  • On the judges: the synthesis judge and the turn-level attribution judge used in training are not checked directly. The paper only gives one example.
  • On the training gains: coverage rises, but the paper does not show which kinds of analysis improve. So it is not clear whether the model gains real skill or adapts to the style of the rubric.
  • On the forecast: the forecast gain is only reported on the Private split. On this split, all models, including frontier ones, score below always answering Yes. So the gain is hard to read. I suggest a human check of the judge scores and a breakdown of the gains by type of analysis.
Questions for authors
  1. With only the company’s own documents, how far can the forecast outcomes be inferred from them?
  2. Could you run a human check on the turn-level scores of the attribution judge? Could you break down the coverage gain by theme or by type of step?
  3. What is the forecast accuracy on the held-out split before and after training? Why do all models lean strongly toward No on the Private split?
6 comments

Response to Reviewer rqFN (1/2)

Thank you for the constructive review! We appreciate that the reviewer recognized the separate research and forecast scoring and highlighted the human expert check of the jury, and we are also glad the reviewer raised the ethics question. Several of the newly added analyses come directly from your questions (Appendices N.3, N.5, N.8 and N.9); we believe the new analyses strengthen the paper.

W1a. Documents in the environment. We would like to respectfully clarify what the environments contain. Apart from the company's own transcripts and press releases, they also include regulatory filings, peer-company financials, third-party releases and industry statistics from external sources. We apologize if it was not clear before; now we have expanded Appendix C to make it clear. Appendix C details the source composition ([NeA]; [AP] Table 4).

Of the 2,967 files:

  • 494 are SEC regulatory filings (157 Form 10-K, 314 Form 10-Q, 23 Form S-1);
  • 540 are peer-company financial files (270 income statements, 270 cash-flow statements);
  • 129 are releases from law firms and shareholder-service organizations, covering litigation, regulatory disputes and company disclosures;
  • 232 file copies (63 distinct), mostly earnings-call transcripts and SEC filings, carry industry statistics from sources including the World Bank, Nielsen, the National Association of Realtors and the Centers for Medicare & Medicaid Services.

As the reviewer noted, news articles are not included ([AP], Table 4).

W1b / Q1. Which part of real research this is, and how far the documents inform the forecast. [NeA] Section 2.2 states the scope: the setting represents the document-based part of fundamental research, forming a view from the source files.

Two results that demonstrate that the environments help the forecast:

  • Table 56 (NeA, N.9) qualitatively shows four examples where the environment documents help in forecasting.
  • The full environment is ahead of the question-only scores for five of the six models with a documented cutoff. GPT-5.6 Sol moves from 50.98% to 63.73% on its 34 such tasks (AP, Table 18). Across all 92 Public tasks, ten of eleven models are more accurate in the full environment than from the question alone, by 2.2 to 16.3 points (AP, Figure 6).

W2. "The method mostly follows existing practice." We appreciate the chance to make the difference clearer.

[AP] We would like to highlight the importance of TurnCredit here. With only 21 updates, we show a better test-time scaling curve than the base model both in distribution (Hedge-Bench-Heldout, Fig 5A) and out of distribution (GDPval, Fig 5B). Post-training models on long-horizon tasks with limited budgets is an important problem, as noted in [1]. We provide a new method that uses meta-prompting (Appendix F.1; its effect is shown in the newly added Appendix N.8) and LLM judges as critics. We demonstrate that TurnCredit avoids entropy collapse and performs much better than outcome-only GRPO (Figure 4, Table 2). Appendix A compares this against the closest work. DRACO, for example, preserves the trajectory's sign. The magnitude-matched control isolates the gain: with the same reward and advantage magnitude but without turn-level direction, coverage is 11.67 to 14.82 points lower (Table 2).

W2a / Q2. Are the training judges checked directly? [AP] Yes, Appendix G.2 contains three examples (Centene, e.l.f. and VeriSign), and we newly added more analyses in N.8:

  • Judge consistency. Trajectories are scored twice. The two synthesis judge scores differ by 0.022 on average (0 to 1 scale), and 93% of 1,327 pairs are within 0.05. For reasoning blame, the attribution judge's two calls agree on the turns with an F1 of 0.89 (Table 54).
  • Manual checks (Table 55). For three trajectories (GRAIL, Luckin, Apollo), we provide a manual analysis of TurnCredit's effectiveness. The diagnosis matched in all three, and the turn weights were well supported in two. The synthesis score also correlates well with forecasting (correlation 0.91; AP, Appendix F.1).
  • Judge prompt replay. The newly added replay (N.8) with and without the blame rule also highlights the effectiveness of Metaprompting and the judge prompt.

[1] To Critic or Not to Critic: GRPO vs PPO. Jason Wei, George Pu, Jingxuan Fan, Yuan (Emily) Xue, Yuan (Christy) Li. https://labs.scale.com/blog/to-critic-or-not-to-critic-grpo-vs-ppo

Response to Reviewer rqFN (2/2)

W2b / Q2. Which kinds of analysis improve: real skill or rubric style? Thank you for this question. It led to a new analysis which we believe strengthens the paper.

[NeA] Appendix N.3 classifies the 974 rubric moves of the 75 environments into five categories (Table 47). Coverage gains (high effort) under the two-juror jury of GPT-5.6 Sol and Claude Opus 5, with company-clustered 95% intervals (Table 48):

Move typeHeldoutPrivate (unseen companies)
Comparisons+21.88 [17.20, 25.87]+13.91 [8.24, 20.63]
Causal inference+16.36 [14.22, 18.16]+13.06 [7.08, 18.10]
Counterarguments / risks+19.90 [8.33, 32.61]+20.33 [17.48, 22.22]
Evidence identification+4.46 [0.00, 13.46]+12.82 [3.17, 20.83]
Numerical calculation+7.69 [-12.50, 15.38]+8.33 [0.85, 16.24]

On Heldout the gains are largest for comparisons, counterarguments and causal inference; on Private all five types improve. High-effort Heldout report pairs record 522 move-credit gains compared to 121 losses, and 295 of the gains are in themes where the base report earned no credit.

Evidence of skill improvement that points away from rubric style:

  • [NeA] The gains hold without the Gemini juror (Table 45).
  • [AP] The gains transfer to external benchmarks with different rubrics: Harvey LAB with both Claude Sonnet 4.6 and GPT-5.5 judges, and blind pairwise comparisons on GDPval (Appendix H.5; NeA in N.7).

Appendix N.4 qualitatively shows this difference between Muse-Glimmer-Hedge and Muse-Glimmer on individual reports.

W2c / Q3. Forecast accuracy on Heldout, and why models lean No on Private. [NeA] We have added the Heldout accuracies of Muse-Glimmer (73.86%) and Muse-Glimmer-Hedge (79.55%), an improvement of +5.68 points (company-clustered 95% CI [1.56, 10.23]). Initially we didn’t report the Heldout accuracy numbers because they share companies with the training set. Additionally, Muse-Glimmer already scores 19/22 pass@8 in the Heldout set and performs as well as GPT-5.6 Sol.

Why models lean No is answered in Appendix N.5 [NeA] and summarized in our comment to all reviewers: for 17 of 53 Private questions (1/22 on Heldout), one future report meeting the condition is enough for Yes, and a recurring error is to expect the recent trend to continue. On the eleven such questions that resolved Yes, training corrects six of the base model's 33 wrong No answers.

Ethics. Please see the "Ethics and data release" part of our comment to all reviewers.

We hope this answers your questions, and we would be glad to clarify any other questions further. If your concerns are resolved, we would be grateful if you would consider updating your rating. Thank you again for your time!

Thank you for the detailed response and new analyses. Most of my concerns are addressed, and I will raise my score. One small point remains: the training judges are only checked for consistency and on three cases. A larger independent check would help.

Comments on the paper

2 comments
AuthorsOpenReview ↗

To all reviewers and the AC: summary of the revision

We sincerely thank all three reviewers for their careful and constructive reviews and we appreciate that they recognized the analyst-derived rubrics (all three), separate evaluation of research and forecast (rqFN, 9sG6) and magnitude-matched credit ablations (pfMw). The revised PDF contains new content in blue. Here in our replies, we marked items as newly added (NeA) in the revision or already present (AP) in the submission.

Newly added analyses (App N and O)

  1. Company-clustered uncertainty (N.1; pfMw). Following pfMw's suggestion, we resample companies, keeping all tasks and repeats of a company together. The Private forecast gain of +10.06 points at high effort (AP, Table 26) has a company-clustered 95% interval of [2.38, 17.95]. [NeA] Held-out forecast accuracy: at high effort, it rises from 73.86% to 79.55% (+5.68 points [1.56, 10.23]), against a 54.55% majority baseline.
  2. Jury without Gemini (N.2, Table 45; pfMw). The reward judge is a sibling Gemini model, so we rescored coverage with only GPT-5.6 Sol and Claude Opus 5. The gains hold similar to the three-model jury. Scores: +16.57 to +17.32 on Heldout and +12.30 and +14.71 on Private, and every company-clustered interval excludes zero.
  3. Which kinds of analysis improve (N.3, Table 48; rqFN). We classified the 974 rubric moves of the 75 (53 Hedge-Bench-Private + 22 Hedge-Bench-Heldout) environments into five types. On Heldout the largest gains are in comparisons (+21.88), counterarguments (+19.90), causal inference (+16.36), and on Private all five types improve.
  4. What changes inside a report (N.4; 9sG6, rqFN). We provide paired qualitative examples of how Muse-Glimmer-Hedge performs better than Muse-Glimmer.
  5. What explains the Private forecast results (N.5, Table 49, Appendix O; all reviewers). All models scored below the majority baseline on Hedge-Bench-Private. The main reason behind this is that models mostly give the answer if nothing changes (trend continues) and on Private that answer is usually wrong. It is correct on 25/56 Public but only on 9/41 Private. The form of the questions also adds to this. 17 of 53 Private questions turn Yes if any one of several future reports meets the condition (1 of 22 on Heldout); eleven of them ended Yes, and Sol answered No on ten. Additionally we provide 10 qualitative examples from Private to show this in detail (Appendix O).
  6. Probabilistic forecasts (N.6, Tables 50 and 51; pfMw). We report P(Yes) on all 75 environments. Training reduces Brier loss on Private by 0.047 [0.023, 0.068].
  7. External benchmarks (N.7, Table 52; pfMw). [AP] Muse-Glimmer-Hedge shows a step-change TTS curve on GDPval compared to Muse-Glimmer and we have added task-level paired comparisons on Harvey LAB and other benchmarks in N.7.
  8. Training judges (N.8, Tables 54 and 55; rqFN). Three qualitative examples were already present (Appendix G.2). We newly (N.8) added consistency checks of the synthesis and attribution judges and also report three more examples manually analyzed.
  9. Why a wrong forecast must receive blame (N.8, Table 53; 9sG6). This is a design choice introduced by Metaprompting. Without the rule the judge credit assignment was skewed towards the final turn. Replaying the judge without the rule shows the penalty is there either way. The rule redistributes the blame from the report-writing (final turn) to the forecast reasoning (other turns). We explain it in detail in Appendix N.8.
  10. Forecast-relevant evidence in the documents (N.9, Table 56; rqFN). We qualitatively show, on four Public environments, that the forecasting questions can be answered with the documents provided in the environment, which indicates that the lower correlation between research and forecasting is not due to missing information.

Newly added text in existing sections

  • Source composition (Appendix C; rqFN). We have added the composition of regulatory documents, industry data etc. in the 2,967 source files.
  • Task selection (Section 2.1; pfMw). Clarified that the environments were chosen manually without any use of frontier models.
  • Ethics and data release (Appendix J; rqFN).

We thank Reviewer rqFN for raising this, and we are glad to provide the details

[AP] We already provided details of the source files (Appendix C), explained our agreement with contributors involving a perpetual, royalty-free license, and described our work with these firms' compliance teams in Appendix J

[NeA] The submission said that all 2,967 file copies would be released. When we confirmed the redistribution terms with the vendor during this discussion period, the vendor informed us that earnings-call transcripts cannot be redistributed. The release therefore includes 1,929 non-earnings-call files and, for the 1,038 earnings-call transcripts, we will make them easy to use with their own keys

We have also expanded Appendix J to include these details & glad to provide any further details.

AuthorsOpenReview ↗

General Comment - 2

We thank all the reviewers for constructive feedback and for helping us strengthen the paper.

During the discussion period, we added the following additional analyses to address reviewers' concerns:

  1. Following Reviewer rqFN's suggestion, we ran an independent human check of the judge's credit assignment over 240 turns. In each category, at least one independent reader agrees with the judge's assignment about 70% of the time or more.
  2. We added a new external forecasting benchmark, FutureSim, which shows the forecasting gain transfers beyond Hedge-Bench. [pfMw]
  3. We added a new ablation where the attribution judge doesn't see either the rubric or the move-by-move verdict, to disassociate the benefit of each component. The newly trained model still beats the magnitude-matched control by 6.05 points [3.62, 8.47] (Medium) and 6.69 points [3.78, 9.64] (High). [pfMw]

We highlight that after the rebuttal, Reviewer pfMw noted that our additional results resolved Q3, the Gemini-free jury, the human validation, and concerns about the external gains. [2] and [3] address the remaining two concerns.

Thank you for engaging during the discussion period!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.