- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Hedge-Bench, built from recorded conversations between pairs of buy-side analysts. Each conversation is turned into a hidden rubric of “analytical moves” plus a separate binary forecasting question that has since resolved. Agents work in an offline, point-in-time container of filings, transcripts and prices, write a cited report and give a Yes/No forecast. A three-model LLM jury scores rubric coverage after a citation-grounding check, and joint success requires a correct forecast plus more than 60% coverage. Eleven frontier models reach 2.5% to 31.5% joint success on the 92 public tasks. The authors then train Muse-Glimmer-30B with TurnCredit, a GRPO variant where a judge assigns signed per-turn credit after the rollout. This improves held-out coverage by about 16 points over the base model and by 12 to 15 points over a magnitude-matched control, and it gives modest gains on GDPval, GDP.pdf, Harvey LAB and Frontier Finance.
Critical strengths
Strength 1. The source material is the real asset here. Rubrics come from unscripted analyst debates rather than LLM-written checklists, the environments are frozen at a cutoff with no internet, and the grounding step (a move is discounted if a majority flags its citation) is a good guard against confident fluff. I find this a much better target than the usual retrieval-style finance QA.
Strength 2. The TurnCredit ablations are better than usual. The magnitude-matched control (Table 2) directly addresses the obvious objection that turn weights just shrink the advantage, and the trajectory length, repetition and entropy curves in Figure 4 tell a consistent story.
Critical weaknesses
Weakness 1. The forecasting half does not support the question in the title. Within model and environment, coverage and correctness are basically uncorrelated (-0.12 to 0.12, nothing survives Holm, Section 4). Question-only accuracy is already 56 to 67% against a 53% majority baseline (Table 16), so a good part of the “forecasting skill” comes from priors before any document is read. The recency argument (later resolutions score higher, Table 17) does not rule out contamination. It is a comparison across different tasks. On Hedge-Bench-Private every model is below the constant-Yes baseline, including GPT-5.6 Sol at 35.85% (Table 28), and the trained model’s “+10 points” moves it from 38% to 48%, still under 52.8%. A benchmark where the strongest model does worse than a coin with a Yes bias on fresh questions makes me want to look first at how the questions were built or selected. I don’t buy the “genuinely more difficult” explanation. Statistically, the outcome is binary with no probabilities (no Brier, no calibration), the 92 questions cluster in 23 companies, and the permutation tests in H.1/H.2 only shuffle trials within fixed environments, so they ignore task-level sampling. With 22 held-out tasks that matters.
Weakness 2. The coverage gains are close to training on the test metric. The attribution judge sees the hidden rubric and the grader’s verdict on which moves were accepted and missed (Section 5.1), and the synthesis reward requires “substantial accepted coverage”. Held-out tasks share companies (and many source files, 730 distinct contents across 2,967 copies) with the 70 public training tasks, and the reward judge (Gemini 3.7 Flash) is a sibling of one of the jurors. So a 16-point coverage jump on Heldout mostly shows that the policy learned what this jury rewards. The external benchmarks help, but gains of 2 to 5 points on judge-scored rubrics are a weaker form of evidence than the abstract suggests.
Two minor things. How tasks were “handpicked” for difficulty out of 5,112 candidates is not specified. If model performance was involved, that biases the frontier numbers. The human validation uses 8 reports and 81 shared decisions with 70% inter-expert agreement. That is thin for anchoring a 1,150-move rubric.
Questions for authors
Q1. Why does every model fall below the majority class on Private? Is there something systematic about how forecast questions are phrased, for example drawn from issues the analysts expected to resolve against consensus?
Q2. Can you report probabilistic forecasts with Brier or log score, and confidence intervals that resample tasks (clustered by company) rather than trials only?
Q3. How was candidate difficulty assessed during selection, and were any of the evaluated models used for it?
Q4. What happens to Heldout coverage if the attribution judge does not see the rubric verdict, or if Gemini is dropped from the evaluation jury?
5 comments
Response to Reviewer pfMw (2/2)
Q2. Probabilistic forecasts and company-clustered intervals. Thank you for both suggestions; both are newly added.
-
Clustered intervals (NeA, N.1). As you suggested, we computed company-clustered numbers. At high effort the forecast gain is +10.06 [2.38, 17.95] on Private and +5.68 [1.56, 10.23] on Heldout. Coverage gains under the jury without Gemini and with company-clustered intervals are in the table below (NeA, N.2, Table 45). Private coverage improves for all 11 companies (AP, H.2.1).
-
Brier loss (NeA, N.6). Now models report P(Yes) on all 75 Heldout and Private environments. On Private, training reduces the loss by 0.047 [0.023, 0.068], from 0.389 to 0.342, level with GPT-5.6 Sol at 0.341. Sol numbers are close to Muse-Glimmer-Hedge even though its accuracy is about 13 points lower, because Sol has many near misses. On easier Heldout all three models are near 0.19.
Q3. How was difficulty assessed, and were evaluated models used? The conversations and transcripts are manually checked by experts on a day-to-day basis, and no model was used in the selection of the environments. All the conversations go through a difficulty check that makes sure they are of good quality. We chose the 92 public environments by hand from the filtered set. Section 2.1 now reads: "We assess candidate difficulty manually, without using any model, and two independent human reviewers verify task quality [...] Both the public and private task sets were fixed before any model was evaluated on them."
W2 / Q4. "The coverage gains are close to training on the test metric." We understand the concern and appreciate the chance to address it. We check in four ways whether the policy only learned what this jury rewards.
Test 1: jury without Gemini (NeA, N.2). A move is credited only if both GPT-5.6 Sol and Claude Opus 5 mark it present:
| Split, effort | Coverage gain [company-clustered 95% CI] |
|---|---|
| Heldout, low | +16.57 [13.36, 21.37] |
| Heldout, high | +17.15 [11.54, 20.81] |
| Private, low | +12.30 [9.55, 15.26] |
| Private, high | +14.71 [11.99, 17.41] |
The Heldout gains are within 0.5 points of the three-juror gains.
Test 2: new companies (AP, Table 26). Private shares no company with training; coverage rises by 13.26 and 16.44 points.
Test 3: external benchmarks (AP, Appendix H.5, Tables 32 and 33). After only 21 updates, one pass over the 165 Hedge-Bench training environments (AP, Section 5), the external gains go beyond 2 to 5 rubric points:
- GDPval, blind pairwise comparison. The trained model beats the base model in 149 to 172 of 220 comparisons at each effort and loses 46 to 70. Its Elo rises by 108.7 to 167.1 (Table 33; [NeA] Table 52 lists wins, losses and ties per effort).
- Harvey LAB (legal tasks, two judges). +7.13 under Claude Sonnet 4.6 and +7.02 under GPT-5.5. [NeA] N.7 adds the task-level count: 73 of 100 tasks improved.
- Rubric scores. All seven external rubric gains have 95% intervals above zero (Table 32).
Test 4: what improves (NeA, N.3 and N.4). On Heldout the gains concentrate in comparisons, counterarguments and causal inference, and 295 of 522 move-credit gains (against 121 losses) are in themes where the base report earned nothing. N.4 shows the qualitative changes in the reports themselves.
Attribution judge without the rubric verdict. The magnitude-matched ablation addresses this question. The magnitude-matched control optimizes the same synthesis score with the same advantage magnitude and gains 2.6 to 4.5 points over the base model, compared to 16.2 to 17.4 for TurnCredit (Table 2), so the rubric-aware reward alone yields little. The verdict the attribution judge sees always concerns a training task, and the four tests above indicate that what the policy learned from it is not specific to this jury.
Minor: human validation (AP, Appendix D.3 and Table 9).
The jury agrees with each expert on 63 of 81 shared decisions (77.8%), above the 57 of 81 (70.4%) between the experts, and on the 57 requirements where the experts agree it matches them on 51 (89.5%, kappa 0.76). In total the two experts made 197 judgments on 110 requirements. We spent about $2,000 (for 20 hours) on this expert review, and a larger one would cost much more. Since the result is consistent across both experts, we hope it is reasonable to stop here.
Thanks again for the constructive feedback and for helping us improve the paper! We hope this resolves your concerns, and we would be glad to discuss any remaining point. If it does, we would be grateful if you would consider updating your rating.
Dear Reviewer pfMw, Thank you again for your detailed review. Since the discussion period is about to end (November 18th), we wanted to follow up on our response. Since then, we have also posted an independent check of the training judge by two human readers and a model jury (in our reply to Reviewer rqFN). Please let us know if our response adequately addresses your concerns. If so, we would kindly ask you to consider updating your score. If not, we are willing to engage further to strengthen the paper. An early reply would give us enough time to work on any remaining points.
Response to Reviewer pfMw (1/2)
Thank you very much for the detailed review, and for recognizing the analyst-derived rubrics, the grounding step and the magnitude-matched control. We newly added the clustered intervals, evaluation without Gemini and the Brier scores you asked for (Appendices N.1, N.2 and N.6), and we answer Q1 and Q3 directly.
W1. "The forecasting half does not support the question in the title." Thank you for the careful reading. We read the title's question as having two answers, and the paper measures both. [AP] We agree that coverage and forecast correctness are not correlated; it is the reason we evaluate them separately (Conclusion). The paper reports this in Section 4 ("Research coverage and forecast accuracy can differ"): within the same model and task, reports with higher coverage are not more likely to forecast correctly (-0.123 to 0.119 as pointed out by you). We also show this qualitatively in Appendix L.2 (Table 37): on Fiserv, one Muse Spark 1.3 output has 21.43% coverage and a correct forecast, and another has 71.43% and an incorrect one. The Flywire pair also shows the same. We then train a model on research and forecast together: the reward uses the synthesis judge, which expects both a good forecast and thorough research coverage. For example, the synthesis judge must score a wrong forecast below 0.5 (AP, Appendix F.1). On Private, whose companies are absent from training, forecast accuracy at high effort rises from 38.36% to 48.43% and coverage rises by 16.44 points (AP, Table 26); the newly added company-clustered 95% CI of the +10.06 forecast gain is [2.38, 17.95] (N.1). On Heldout, the base model already scores 19/22 at pass@8 (NeA, N.6), so the gain shows in joint success (a correct forecast with coverage above 60%): from at most 4 of 176 reports to between 14 and 34 across four efforts (AP, Table 30).
Priors (AP). As you note, the question alone beats the majority baseline, but the documents add to it: ten of eleven models forecast better with the full environment, by 2.2 to 16.3 points (Figure 6).
Contamination (AP). We agree Table 17 compares different tasks. Table 18 keeps only outcomes made public after each model's documented knowledge cutoff and compares the same questions with and without the documents. They still help five of the six models with such a cutoff, by 2.9 to 12.7 points (GPT-5.6 Sol: 63.73% against 50.98% on 34 tasks).
Finally, the analysts' conversations are gathered on a day-to-day basis, so we can avoid contamination.
Q1. Why is every model below the majority class on Private? Is something systematic in the questions? Thank you for the question. We answered why every model is below the majority class on Private in our overall comment.
Selection. The Private questions were built with the same procedure as the other splits and then set aside as a split for unseen companies (AP, Section 2.3). We didn’t intend to make them more difficult than other sets. Each question and label was only verified for correctness, and Section 2.1 now states that the set was fixed before any model was evaluated (NeA). The outcomes nevertheless exhibit a pattern similar to the one you describe. We provide ten qualitative examples in Appendix O (NeA), which lists eight forecasts in which GPT-5.6 Sol answers No and the outcome is Yes.
Some misses are numerically close: For example, FIS reported adjusted free-cash-flow conversion of 88.02%, above the 85% top of the range management had reiterated. Sol forecasts approximately 84% and answers No. In another example, Sol forecasts 62 basis points, while the reported 65 exceed the 63-basis-point threshold.
Scope of the effect. On public questions disclosed after its knowledge cutoff, Sol still scores 63.73% (AP, Table 18), so recency alone does not explain 35.85% on Private.
Following your comment, H.2.1 now cites this evidence for its "more difficult" statement (NeA). Sol's research coverage is also lower on Private (45.33%, against 59.17% on Public; AP, Table 28).