- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces BoleBench, a benchmark of "research taste": the ability to predict, before outcomes are known, which of two research attempts will score better. The ground truth comes from execution rather than publication or preference. Different agents attempt to improve the same frozen baseline under matched compute budgets, and a hidden evaluator scores each attempt. A judge LLM reads two blinded, outcome-free descriptions of attempts on the same task and picks the one it expects to score higher.
The release contains 4,059 pairs from four corpora:
- AI4AI: 1,842 pairs from 271 attempts on 10 ML research tasks.
- arXiv tasks: 1,897 pairs from 398 attempts by 15 models on 35 tasks derived from 2026 papers.
- RSI-Exam: 221 pairs.
- Paper ablations: 99 pairs.
On AI4AI, items are organized into five decision tracks: select-best, reject-below-baseline, select-ambitious, weigh-evidence and override-prior. Select-best is further tested at four information levels: a 350-character gist, the full description, plus exploration evidence, and plus system identity. Each judge receives a nine-axis report card.
Critical strengths
-
Ground truth from executed, matched alternatives, with failures retained. Most prior "research taste" or outcome-prediction benchmarks grade against publication records, leaderboards, citations or expert preference (Wen et al. 2025; Mule et al. 2026; TASTE; ForeSci). These sources filter out abandoned and failed attempts and cannot provide a counterfactual. BoleBench pairs attempts made from the same frozen start under the same budget and grades them with a hidden evaluator, so every comparison has an observed alternative, including failures. The design also scales: new item banks can be built from fresh corpora as models are retrained. This is the right kind of ground truth for the question practitioners care about, namely which experiment an automated research system should spend compute on. The benchmark is valuable as infrastructure even beyond the specific findings.
-
Diagnostics that undercut the aggregate leaderboard. The most consequential finding is the boldness heuristic. A one-line rule beats every LLM judge on clear-gap pairs, and on tied pairs the best judges retain only a modest margin above chance (l. 257–262). This is an important caution for the growing use of LLM judges and research preference models to allocate compute. The paper also adds several analyses that make the benchmark more than a leaderboard:
- one-of-k selection replay, which shows that pairwise accuracy does not determine selection value and that some judges fall below random as candidate pools grow (Fig. 5, Table 4);
- order-swap, idea-only and house-style rewrite controls, which show that stable pooled accuracy hides 21% pick flips (Table 3);
- an explicit treatment of what single executions can and cannot identify (Appendix F).
Critical weaknesses
-
The construct measured is narrower than "research taste", and several headline findings are partly products of how items are selected.
- Task scope. The paper motivates taste as problem choice under slow feedback (Hamming; Nanda; l. 82–84). The items, however, compare code changes to a fixed baseline on a fixed metric within a single compute budget. The paper itself notes that more than half of the AI4AI attempts change only schedules, hyperparameters or other run settings (l. 47–48, Fig. 8). This is closer to predicting which ML-engineering variant wins than to choosing research directions. The title question, "Do LLMs have research taste?", is therefore broader than what the evidence can answer.
- Descriptions are written after the fact. They are reconstructed from completed programs by LLM reviewer and rewriter sessions (l. 628–635). They include implementation constants and "risk signals" that would not exist before execution. The authors concede in Appendix F (l. 999–1002) that removing outcomes does not make a description prospective. Information that only exists after execution may be present, for example a bug visible in the final code. In that case the benchmark partly measures code-reading or debugging rather than foresight. The "+exploration evidence" level supplies interim evidence from the run (l. 203–204). Its +12.4-point gain (Table 2) may therefore reflect reading partial outcomes rather than taste.
- Override-prior and trap results follow from item selection. The abstract says "no judge exceeds chance" when the bolder attempt loses, and the main text presents this as a failure (l. 25–26, 265–266, 302–303). Appendix F shows that a calibrated, accuracy-optimal policy that follows the boldness prior would score 0% on exactly this subset (l. 981–986). Judges score 36.9–50%, which is above what the optimal prior-follower would get. These subsets are selected on the outcome, so they cannot by themselves show poor judgment. The headline should be revised or paired with a matched comparison, for example accuracy on the subset where the bolder attempt wins.
- The independence baseline for shared errors is too weak. "135× the independent-error expectation" (l. 90–91, 273–275) compares against a baseline that ignores item difficulty. Eq. (8) shows that varying item difficulty alone produces excess joint failure. The paper cites item-response models (l. 157–158) but does not use one. A comparison against an IRT- or difficulty-adjusted expectation is needed before "shared priors across families" can be read from this number.
- The provenance of the boldness heuristic is unclear. "Pick the bolder" uses evaluator-side committee labels (l. 184, 929–930). If the committee is made of LLMs or annotators who read the full descriptions, then the "one-line heuristic" is itself the output of a judgment process. Comparing it with judges is then not a comparison between a trivial cue and full judgment. Its 77.9% is also measured on a subset defined by having a clear label gap.
-
The effective sample is small, ground-truth noise is not quantified, and many judge-level claims are not statistically supported.
- AI4AI concentration. All track, information-level, boldness and human-baseline analyses use AI4AI. Its 1,842 pairs come from only 271 attempts on 10 tasks. Task-clustered bootstrap intervals with 10 clusters are unreliable, and pairs that share attempts are strongly dependent. In Fig. 3, the top ~10 judges lie within a few points of each other on 326–333 pairs.
- Leaderboard rankings are not robust. The paper concedes that no judge clearly separates from the majority (l. 239–240). Yet the abstract states that "the newest frontier model leads at 70.8%". On the balanced 99-per-corpus subset, gpt-5.6-sol (71.7) and claude-opus-5 (71.0) rank above claude-fable-5-1 (68.6) (Table 6). Track rankings (Fig. 4b) and best-of-ten results (Table 5, overlapping CIs of roughly ±10 points) are similarly fragile.
- Single-run labels. Each label comes from one execution, and arXiv-task pairs need only a normalized-score difference greater than 0.005 (l. 638). The paper acknowledges that it cannot identify a run-noise ceiling (l. 996–999). Without repeated executions of at least a subset of attempts, we cannot tell whether ~63% on tied pairs is close to the achievable ceiling or far from it. This matters for the conclusion that "current judges have much room to improve".
- Thin human baseline. It uses 60 pairs in 20-pair slices with 10 shared anchors. The "identical choices including the same three errors" claim in the abstract rests on 10 items. The paper itself says the sample cannot rank individuals (l. 432).
- Contamination claims are not quantified. Renewal against contamination is a stated contribution (l. 99–100), but the memorization quiz results are not reported. The paper-ablation corpus, whose accuracies reach 77–95% with a 74.5% length baseline, is presumably built from published papers. It may be both memorized and publication-filtered, which contradicts the paper's motivation.
These issues determine whether the nine-axis report card and the leaderboard can be used, as the paper recommends (Appendix G), to choose a judge for deployment.
Questions for authors
- Boldness committee. Who assigns the boldness and change-family labels: humans (with what expertise), LLMs, or both? What inputs do they see, and what is their inter-annotator agreement? If the labels come from reading full descriptions, how should "a one-line heuristic beats every judge" be interpreted?
- Override-prior and trap framing. Given Appendix F's result that an optimal prior-follower scores 0% on these subsets, would you revise the abstract's "no judge exceeds chance"? Could you instead report a symmetric statistic, such as the judge's departure rate from the prior and the accuracy of those departures on all clear-gap pairs (Eq. 4)?
- Information available only after execution. How much of the description content could not have been known before execution, such as bugs visible in the final code, risk signals or crash traces? Could you run a judge on descriptions limited to the planned change, for example the agent's initial plan before execution, on a subset? Does the exploration-evidence level contain interim metric values?
- Run-to-run noise. For a subset of attempts, can you re-execute with different seeds and report how often the pairwise label flips? This would give a noise ceiling for pairwise accuracy, especially for pairs with small gaps.
- Shared-error baseline. Can you report joint failure against a difficulty-adjusted expectation, e.g., a fitted IRT or Rasch model, rather than the independence product?
- Corpus details. How were RSI-Exam and the paper-ablation pairs constructed, and are they post-cutoff for all judges? What did the memorization quiz find for each judge and corpus? Since the arXiv-task construction is deferred to a companion paper, what are the selection criteria for the 35 papers?
- Description authorship. Which models powered the reviewer, rewriter and checker sessions? Are any of them among the judges? This matters for the self-favoritism and executor-bias analyses.
- Validity checks. Please report the convergent, discriminant and known-groups checks mentioned in l. 86–87 and l. 220, and clarify what "agreement 0.85–0.95, κ 0.71–0.89" across three 200-item replicates measures (judge test–retest? label agreement?).
- Statistical comparisons. Can you provide paired significance tests for the leaderboard ordering and for the track-ranking reversals (e.g., Opus 4.8 on reject-below-baseline), given the 10-task clustering in AI4AI?