acceptodds
Under review as a conference paper at ICLR 2027

ResearchTree: Measuring the Research Horizon of Language Models

Abstract

How capable are language models at research? We measure one part of this question: how much of existing human research a model can reconstruct on its own, from a single step up to a whole paper. We introduce ResearchTree, which turns a research paper into a tree of self-contained problems, from its main result down to atomic steps with solutions taken from the paper. The model under test answers every problem independently, and a judge scores each answer against the paper's derivation beneath it. Two measurements follow. The research horizon is the problem size, in atomic steps, at which a model's expected score falls to one half. The decomposition curve shows how much of a paper a model recovers as the paper is given to it in smaller and smaller parts, and its deficit is the share the model still misses on average. On 20 recent theoretical papers and ten models, frontier models solve almost every atomic step but much less of each whole paper. Research horizons range from under one step to about 40, differ several-fold between fields, and across four leading releases have doubled roughly every eight months, a trend that rests heavily on the earliest release, while the decomposition deficit has halved about once a year. Posed a whole paper, models mostly leave parts of the derivation out rather than get them wrong. Because the trees are built automatically from new papers, the evaluation can be renewed after every model release and could extend to other fields. We release ResearchTree as a public dataset: 20 paper trees with 1,079 problems (744 atomic steps with reference solutions from the paper), 10,790 model answers and 32,880 graded judgments, together with the code to build new trees.

Primary area:
datasets and benchmarks
Keywords:
evaluation, benchmark, research horizon, scientific reasoning, language models, problem decomposition, LLM-as-a-judge, long-horizon reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_tFw1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the "research" capabilities of large language models. The authors select 20 very recent theoretical papers and decompose the problems addressed in these papers into a hierarchical tree structure, ranging from high-level research problems to smaller, more specific subproblems. The models are then asked to solve problems at different levels of this hierarchy. Afterward, the authors use large language models as judges to evaluate the quality of the generated solutions. The results show that some of the latest models achieve relatively high scores on atomic problems, but generally perform poorly when tackling the complete, high-level research problems addressed by the original papers.

Critical strengths
  1. Evaluating the “research” capabilities of models is indeed important, and using research papers as a starting point is a good choice. The authors progressively decompose the problems addressed in these papers into smaller subproblems, providing a natural and systematic way to evaluate the capabilities of different models..

  2. The paper is well-written, with a clear structure and logical flow. The dataset construction and model evaluation methods are generally reasonable and well-designed. The presentation and analysis of the evaluation results are also clear and easy to follow.

Critical weaknesses
  1. Although the results are largely in line with expectations, this also means that the paper lacks more surprising or exciting findings. In my view, the findings are reasonable but not particularly insightful. The paper would be stronger if the authors could explore the results more deeply and uncover more interesting conclusions.

  2. This paper aims to evaluate the models' “research” capabilities. However, the selected papers are all theoretical, and in my view, the proposed method would be difficult to extend to experimental papers. Moreover, the authors mention that models are not allowed to use web search or other external tools when answering questions (which I understand is intended to ensure fairness). However, the ability to search for and gather information is an important part of research. Therefore, I feel that the paper focuses more on evaluating the models' “reasoning” capabilities rather than their “research” capabilities. I hope the authors can further clarify this distinction.

  3. The paper uses Claude-family models for both question generation and the final judging stage. Although the authors used GPT models to double-check four papers, this setup may still introduce some degree of bias.

  4. The markers in Figure 1 are somewhat confusing. Claude models are represented by squares in the legend, but by circles in Figure 1(b).

Questions for authors

I would appreciate it if the authors could address the concerns raised in the Weaknesses section, particularly regarding the broader significance of this work and whether it evaluates research capabilities or reasoning capabilities. Overall, I think this is a fairly good paper, but I still have some concerns. If the authors can adequately address these concerns, I would consider raising my score.

1 comment

Thank you for the review, and for saying what would change your assessment. We take your points in order.

W1. Broader significance.

The revision adds two findings that go beyond "steps are easier than wholes".

  1. The bottleneck is knowing which intermediate results to establish. In a controlled comparison at matched answer length, telling a model which sub-problems to solve recovers most of the step-to-whole gap (58% for Opus 5.5 and 86% for GPT-5.6 Sol at depth 1). A longer answer or solving the pieces in separate sessions adds little (new App. P, Fig. 17). Models can carry out the steps of a research derivation; what they mostly lack is knowing which results to establish along the way.
  2. The trees can train models, and the gain transfers. In a preliminary pilot (one seed; new App. R), reinforcement learning on the trees of 20 papers improves a 9B open-weight model on held-out papers at every problem size: +0.05 to +0.10 under an independent judge (Fig. 19). Under the local judge used in training, all 22 held-out papers improve, including one from a field absent from training. Training on atomic steps alone already lifts larger problems of unseen papers. The gains carry over to outside benchmarks: PRL-Bench +8.1 points and HARDMath2 +2.9, with no loss on ArXivMath (Fig. 20).

The original findings also hold on twice as many papers (new App. Q): Opus 5.5 scores 0.97 on steps and 0.60 on whole papers, and the horizon doubles about every nine months.

W2. Research or reasoning?

We agree that the distinction matters, and a new paragraph in the introduction, "What ResearchTree measures", now states it. ResearchTree measures one part of research: reconstructing a long derivation from a precise statement of its goal. It does not measure choosing problems, searching the literature or running experiments.

  • Why search is off. The source papers are public, so with search the task becomes finding the paper. The models were not otherwise tool-free: each ran in its lab's agent harness with files and a Python sandbox.
  • Why this is more than short reasoning items. The problems come from papers posted after nine of the ten models were released, ask for several results, and span 15 to 130 steps. The strongest model scores 0.98 on single steps but 0.62 on whole papers, and the tree shows where such long chains break.
  • Experimental papers. We chose theory because its steps can be evaluated without running experiments. Trees for empirical papers are possible (FIRE-Bench, now discussed in App. A, builds them for ML papers), but their leaves need an execution environment. We list this as untested.

W3. Claude builds and judges.

  • Judge (App. C.1, Fig. 6). The GPT-5.6 Sol judge now covers eight papers, two per field, under a protocol fixed in advance. The family contrast is −0.001 (95% interval [−0.020, +0.017]), so there is no family-level self-preference. The six models judged on all eight papers have the same order under both judges. Against GPT-6 Astra alone there is a small relative effect (+0.033), which the design cannot attribute to either judge; we report it.
  • Builder (App. C.2, Table 3, new since the version you read). GPT-6 Astra rebuilt four trees. Opus 5.5, GPT-5.6 Sol, Sonnet 5 and GPT-5 keep their order. The Opus–Astra comparison depends on the builder, and we say so.
  • A second scoring rule (App. C.5). Under a correctness judge that uses no reference solution, with GPT-6 Astra writing the checklist and judging, Opus 5.5 is still first.

W4. Figure 1 markers. Fixed: Claude models are squares in every panel.

We hope this addresses your concerns, and we are glad to clarify anything further.

Reviewer_vQ75OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RESEARCHTREE evaluates how much of a theoretical paper’s derivation a language model can reconstruct independently. To move beyond fixed-granularity evaluations, it automatically constructs trees of self-contained problems and introduces a research horizon and decomposition deficit. Across 20 papers, four fields, and ten models, eight models score above 0.9 on atomic steps, while the best mean whole-paper score is 0.62. Fitted horizons range from approximately 0.7 to 38 steps and differ several-fold across fields. Across four leading releases, the authors report roughly eight-month horizon doubling and annual deficit halving, with the horizon trend sensitive to the earliest release. Judge-based analysis attributes most lost coverage to omissions rather than explicit errors.

Critical strengths

[Informative Multiscale Evaluation]

  • Independently evaluating every node enables comparisons across problem sizes within the same derivation. This distinguishes strong performance on individual steps from limited reconstruction of larger arguments, providing more diagnostic information than a single aggregate score. Automatic construction from recent papers also offers a practical route to renewing the benchmark.

[Substantial Paired Evaluation and Useful Robustness Checks]

  • All ten models answer the same 1,079 problems, supporting paired comparisons. The substantial step-to-whole-paper coverage gap persists after disputed-paper exclusions. Cross-lab rejudging, paper-level uncertainty estimates, and weighting analyses strengthen the descriptive findings, while transparent reporting identifies model comparisons that are sensitive to construction errors.
Critical weaknesses

[Reference-Based Scoring Limits Fairness of Capability Comparisons]

  • Sections 2.1–2.2 explicitly permit valid alternative derivations to receive little credit, so scores reflect adherence to the selected reference route alongside problem-solving ability. Generated task errors can additionally penalize justified objections. The authors investigate these issues in Appendices C and K and show that the main coverage gap survives disputed-task exclusions. However, these checks do not independently calibrate how rankings correspond to mathematical solution quality. Expert assessment of representative answers against their original questions would strengthen the capability comparisons in Table 1.

[Limited Additional Diagnostic Insight]

  • The qualitative finding that models solve individual steps more successfully than combined tasks. Measuring this across complete research derivations is valuable, but Sections 4.3–4.4 provide limited evidence about what causes the remaining deficit. Decomposition supplies premises and additional opportunities to reason, while the omission analysis identifies missing content without establishing why it was omitted. The authors acknowledge these limitations, but controlled interventions are still needed to distinguish the relevant bottlenecks. Consequently, the findings characterize performance more clearly than they identify promising directions for improving research capability.
Questions for authors
  1. How closely do reference-coverage rankings agree with expert assessments that credit valid alternative derivations and justified refutations?
  2. Under comparable resource budgets, how much improvement comes from supplied intermediate premises, explicit planning support, or subdivision itself? Distinguishing these effects would make the benchmark more informative for model development.
3 comments

Thank you for the careful review. Both of your questions led to new experiments.

W1 / Q1. Does reference coverage track solution quality when alternative routes and justified objections get credit?

Expert ratings are still coming in, so we answer in two parts.

A judge that credits any valid route (new App. C.5, Table 4, Fig. 9). This judge sees the problem and the answer but no reference solution. It checks each result the problem asks for and gives full credit to any valid route and to any justified objection. Two judges from different labs (Claude Sonnet 5 and GPT-6 Astra) graded the whole-paper answers of all ten models on four papers.

  • The two judges agree (Pearson 0.95 per answer, Spearman 0.99 on the model ranking), and the judge is stable when repeated (weighted κ = 0.97).
  • Correctness and reference coverage rank the models alike: Spearman 0.92–0.97 for the nine models other than GPT-6 Astra, and Opus 5.5 is first under every score.
  • The exception is the effect you describe. GPT-6 Astra rises from 4th–7th to 2nd, mostly through one paper where both judges give its alternative proof full credit and coverage gives 0.38.
  • Such cases are rare below the root. Of 410 answers to intermediate problems, 6 are judged correct while covering at most 0.4 of the reference.
  • Among these answers there is one justified objection, and the two judges disagree on it (0.59 against 0.06). LLM judges do not settle that case.

Table 1 is therefore now labelled as reference-route coverage, with the Opus–Astra order flagged as its least stable comparison.

Expert assessment (new App. S). In response to your review we invited the authors of all 40 source papers to rate anonymous answers to their own paper. For each answer they mark which requested results it establishes by any valid route and whether it has a substantive error, and they score it with the judge's rubric. Ratings are coming in. The analysis was fixed in advance (agreement with the judge as weighted κ and rank correlation), and the final version will report the complete study, whatever it shows.

W2 / Q2. What causes the deficit: premises, planning support, or subdivision itself?

We ran the controlled intervention you suggest (new App. P, Table 15, Figs. 17–18). On four papers, Opus 5.5 and GPT-5.6 Sol answered every root and depth-1 problem under four conditions:

  • A: the original answer;
  • B: the same problem, at an answer length matched to the combined answers to its sub-problems;
  • S: as B, with the sub-problem statements added to the prompt (one prompt, one answer);
  • C: the sub-problems solved separately, as in the tree.

Results at depth 1 (15 problems per model; 95% intervals in brackets):

StepWhat it isolatesOpus 5.5GPT-5.6 Sol
B − Amore room to answer+0.039−0.017
S − Bbeing told the sub-problems+0.060 [+0.011, +0.109]+0.140 [+0.039, +0.240]
C − Ssubdivision itself (separate sessions)+0.004+0.039

Most of the gap (58% for Opus, 86% for Sol) closes when the model is told which intermediate results to establish, at the same answer length and in one context. More room and subdivision add little. Output tokens were comparable: the single S answer used 0.92× (Opus) and 1.21× (Sol) the tokens of the separately solved sub-problems combined.

Two limits, both stated in the paper. Premises and planning support cannot be separated in these trees, because one sub-problem's goal is the next one's premise. And with four roots per model, the root-level intervals are too wide to separate the three steps.

Why content is omitted. The same comparison bears on the omission analysis. Once the sub-problems are named, the same model at the same length supplies most of the missing content. This suggests that the omissions come from not knowing which intermediate results the argument needs, rather than from being unable to derive them or lacking room.

Directions for improvement. The bottleneck this locates is finding the decomposition. A preliminary pilot (new App. R) also shows that the trees can be used for training: reinforcement learning on the trees of 20 papers improves a 9B open-weight model on held-out papers at every problem size (Fig. 19) and on outside benchmarks (Fig. 20).

Thanks for the additional experiments. The 40-paper extension makes the gap between step-level and whole-paper coverage more convincing, and the reference-free evaluation broadly preserves the model rankings. I’m still cautious about interpreting those rankings as evidence of mathematical solution quality. On the four-paper subset, Astra moves from 4th–7th to 2nd under the new rubric, and the Opus–Astra comparison is also sensitive to the choice of tree builder. The expert study directly addresses this concern, and its results are still pending.

The matched-length experiment addresses part of my second concern. Providing the subproblems recovers most of the coverage gap at depth 1 for both models. As Appendix P notes, the prompts supply both intermediate goals and premises, and the rubric can award credit for restating supplied results. This makes it hard to tell how much of the gain comes from planning support. With only four root problems per model, it’s also unclear whether the same explanation holds for whole papers.

I still see value in evaluating the same derivation at several levels. The revision strengthens the descriptive finding, but the remaining questions about scoring validity and what drives the whole-paper gap are central to my assessment. I’m keeping my rating at 2 (weak rejection).

Thank you for reading the revision closely. We agree with your account of what the new experiments show and what they leave open. One piece of new information bears on your first point.

1. The paper behind Astra's change of rank. Astra's move from 4th–7th to 2nd comes from one paper, 2609.19623. An author of that paper, replying to our invitation, pointed out that its proof was first found by an AI system: the acknowledgments of the version we used state that the proof was initially found by GPT-5.6 Sol and then checked and revised by the authors. On such a paper it is not surprising that the strongest models reach the theorems by routes other than the published one, so that coverage underrates them. On the other three papers of the subset, Astra is 7th under coverage and 7th under the Sonnet correctness judge (3rd under the Astra judge, which alone credits its objection on 2609.04992). Coverage and the Sonnet correctness judge then rank the ten models at Spearman 0.93, against 0.79 with that paper (0.82 against 0.76 for the Astra judge).

We then checked the text of all 40 source papers. Nine state that AI tools contributed to the mathematics (seven of the ten cs.CC papers and two math.CO papers; in each case an OpenAI system, most often GPT-5.6 Sol). Three report AI use for checking, literature search or editing only, two state that none was used for the mathematics, and 26 make no statement. Four of the nine are among the 20 papers of the submission. The submission did not report this, and it should have. On those 20 papers, leaving the four out does not change the order of the ten models by whole-paper score, step scores change by less than 0.01, and whole-paper scores rise (Opus 5.5: 0.62 to 0.67), so the gap between steps and whole papers remains (0.98 against 0.67). The final version will list the AI-use statement of every source paper and give every result with and without these papers.

2. Scoring validity. We agree that the authors' ratings are the direct test and that LLM judges do not settle it. The ratings are coming in, and the final version will report the complete study.

3. What drives the whole-paper gap. We agree with both limits. Our claim is limited to depth 1: there, a longer answer and separate sessions explain little of the gain, and being told the sub-problems explains most of it, without separating goals from premises. For the final version we will add a condition that lists the intermediate results as goals without the sub-problems' premises, score it without credit for results stated in the prompt, and run the roots of all 40 papers instead of four.

Thank you again for the careful reading.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Comment to All Reviewers and AC

We thank the reviewers for careful and constructive reviews. Your questions led to new experiments and a human study, all in the revised PDF. New text is in blue. Appendix letters A–O are unchanged, and Appendices P–S are new.

Three major additions

1. A controlled experiment on what causes the step-to-whole gap (App. P; vQ75 Q2, Qa11 Q2). At matched answer length, most of the gap closes once a model is told which sub-problems to solve. At depth 1 this accounts for 0.060 of 0.103 for Opus 5.5 and 0.140 of 0.162 for GPT-5.6 Sol. A longer answer (+0.039 / −0.017) or solving the pieces in separate sessions (+0.004 / +0.039) adds little.

2. A human evaluation by the papers' own authors (App. S; vQ75 Q1, Qa11 Q1). In response to your reviews we invited the 91 authors of all 40 source papers to rate their own paper's tree, its reference solutions and anonymous model answers, using the judge's rubric. Ratings are coming in. The analysis was fixed before any rating was seen, and we commit to reporting the complete study in the final version, whatever it shows.

3. A training pilot: the trees can train models (App. R; tFw1 W1, vQ75 W2). Reinforcement learning on the trees of 20 papers improves a 9B open-weight model on held-out papers at every problem size (+0.05 to +0.10 under an independent judge) and on outside benchmarks (PRL-Bench +8.1 points, HARDMath2 +2.9). It is preliminary (one seed).

Further checks

4. A judge that credits any valid route (App. C.5; vQ75 Q1, Qa11 W1). It sees no reference solution, and it ranks the models almost as reference coverage does (Spearman 0.92–0.97). The one exception is GPT-6 Astra, which moves up through a valid alternative proof on one paper. Table 1 is now labelled as reference-route coverage.

5. A judge and a builder from another lab (App. C.1, C.2; tFw1 W3, Qa11 Q3). The second-lab judge now covers eight papers under a protocol fixed in advance. There is no family-level self-preference (−0.001, 95% interval [−0.020, +0.017]), and the six models judged on all eight papers have the same order under both judges. A second-lab builder rebuilt four trees: four of the five models tested keep their order, and the comparison of the two leaders depends on the builder.

6. Twice as many papers (App. Q). On 40 papers every headline finding holds. Opus 5.5 scores 0.97 on steps and 0.60 on whole papers, and the horizon doubles about every nine months. One calibration test now fails, and we report it.

7. Writing. A new paragraph states what ResearchTree does and does not measure (tFw1 W2). Related work covers the five requested papers (Qa11 W5). The omission finding is now attributed to the judge (Qa11 W4). The markers of Figure 1 are fixed (tFw1 W4).

Thank you for the detailed review and its concrete suggestions. We acted on all three of your questions.

W1 / Q1. Expert validation of the questions, references and scoring.

We are doing this with the source papers' own authors (new App. S). In response to your review we invited the 91 authors of all 40 papers (ten per field). Each participating author checks, for their own paper:

  • the root statement (main result, hypotheses, whether it gives the method away);
  • every decomposition in the tree;
  • the reference solutions of eight sampled atomic steps;
  • anonymous answers at three depths (whole paper, intermediate problems, atomic steps), scored with the judge's rubric.

Ratings are coming in. The analysis was fixed before any rating was seen: expert–judge agreement as weighted Cohen's κ over the judge's five rubric bands and as rank correlation, plus the share of statements and references the authors find faithful and correct. The final version will report the complete study, whatever it shows.

On shared reference errors: the new correctness judge (App. C.5) sees no reference solution, and it ranks the models almost as reference coverage does (Spearman 0.92–0.97, apart from one model that moves up through a valid alternative proof). Errors in the references therefore cannot explain the ranking on these papers. The misstated roots remain audited in App. C.3.

W2 / Q2. Matched budget, and the separate benefit of intermediate results.

Done in new App. P (Table 15, Figs. 17–18): four papers, Opus 5.5 and GPT-5.6 Sol, all depth-1 problems (15 per model).

  • At matched answer length, solving the sub-problems still beats the whole-problem answer by 0.064 (Opus) and 0.179 (Sol).
  • Supplying the sub-problem statements in the same prompt, with one answer of the same length, accounts for 0.060 and 0.140 of that.
  • Solving them in separate sessions adds only 0.004 and 0.039.
  • Total output was comparable: the single answer used 0.92× (Opus) and 1.21× (Sol) the output tokens of the separately solved sub-problems combined.

The gap is therefore mainly added information (which intermediate results to establish), not granularity or compute. We agree that it does not isolate composition ability, and Sec. 4 now reports what the control shows.

W3 / Q3. Horizon robustness; a builder from another family.

Builder (App. C.2, Table 3; new since the version you read). GPT-6 Astra rebuilt four trees, and five models solved them. Astra cuts the papers into finer steps, so horizons in steps are 1.0–1.8 times longer and the ratios between horizons are not stable. The rankings mostly are: Opus 5.5, GPT-5.6 Sol, Sonnet 5 and GPT-5 keep their order on whole-paper score, horizon and deficit. GPT-6 Astra's place changes (by whole-paper score, last on the Opus-built trees and second on its own), so we treat the Opus–Astra comparison as builder-dependent and say so.

Trend (App. Q). On 40 papers the horizon doubles every 9.2 months [8.0, 10.6], against 7.9 on 20 papers. It still rests on the earliest release (3.9 months without GPT-5), so we present it as a trend over four releases, not a law.

Calibration. You are right, and more data confirms it. On 40 papers the pooled fit fails a calibration test for Opus 5.5: it underestimates the scores on the largest problems, and we report this (App. Q). The whole-paper score needs no fit, and its model ranking agrees with the horizon ranking (Kendall τ = 0.96).

Extrapolated field horizons. These arise in one field, statistical mechanics, where the strongest models stay above one half on every problem. They stay marked as extrapolations; on 40 papers they move closer to the observed range (39–77 steps, from 46–120; App. Q).

W4. Independent verification of the failure-mode finding.

We agree, and made three changes.

  • The finding is now attributed to the judge's explanations wherever it appears (abstract, Sec. 4, conclusion).
  • We report its limits. On 40 papers no error is identified for 89% of the lost score, but a blind second labelling agrees only moderately (κ = 0.52; App. Q).
  • The authors' ratings ask of every answer, "Is there a substantive error?" (App. S). The share of errors among low-scoring answers is one of the analyses fixed in advance.

The controlled comparison points the same way: naming the sub-problems recovers most of the lost score at the same length (App. P).

W5. Related work. App. A now discusses all five papers, with a pointer in the introduction. ACADREASON and FrontierScience Bench pose one question per paper and grade partial progress within one answer. InquiTree guides an agent through one inquiry. PhysLogic scores coverage of reference reasoning steps, the closest to our judge. SciCode-Verified shows how reference defects depress scores. ResearchTree instead poses every level of one derivation as its own problem, which lets it measure how the score falls with problem size.

We have also added a RL pilot for your reference, see general comment above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.