- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the "research" capabilities of large language models. The authors select 20 very recent theoretical papers and decompose the problems addressed in these papers into a hierarchical tree structure, ranging from high-level research problems to smaller, more specific subproblems. The models are then asked to solve problems at different levels of this hierarchy. Afterward, the authors use large language models as judges to evaluate the quality of the generated solutions. The results show that some of the latest models achieve relatively high scores on atomic problems, but generally perform poorly when tackling the complete, high-level research problems addressed by the original papers.
Critical strengths
-
Evaluating the “research” capabilities of models is indeed important, and using research papers as a starting point is a good choice. The authors progressively decompose the problems addressed in these papers into smaller subproblems, providing a natural and systematic way to evaluate the capabilities of different models..
-
The paper is well-written, with a clear structure and logical flow. The dataset construction and model evaluation methods are generally reasonable and well-designed. The presentation and analysis of the evaluation results are also clear and easy to follow.
Critical weaknesses
-
Although the results are largely in line with expectations, this also means that the paper lacks more surprising or exciting findings. In my view, the findings are reasonable but not particularly insightful. The paper would be stronger if the authors could explore the results more deeply and uncover more interesting conclusions.
-
This paper aims to evaluate the models' “research” capabilities. However, the selected papers are all theoretical, and in my view, the proposed method would be difficult to extend to experimental papers. Moreover, the authors mention that models are not allowed to use web search or other external tools when answering questions (which I understand is intended to ensure fairness). However, the ability to search for and gather information is an important part of research. Therefore, I feel that the paper focuses more on evaluating the models' “reasoning” capabilities rather than their “research” capabilities. I hope the authors can further clarify this distinction.
-
The paper uses Claude-family models for both question generation and the final judging stage. Although the authors used GPT models to double-check four papers, this setup may still introduce some degree of bias.
-
The markers in Figure 1 are somewhat confusing. Claude models are represented by squares in the legend, but by circles in Figure 1(b).
Questions for authors
I would appreciate it if the authors could address the concerns raised in the Weaknesses section, particularly regarding the broader significance of this work and whether it evaluates research capabilities or reasoning capabilities. Overall, I think this is a fairly good paper, but I still have some concerns. If the authors can adequately address these concerns, I would consider raising my score.
Thank you for the review, and for saying what would change your assessment. We take your points in order.
W1. Broader significance.
The revision adds two findings that go beyond "steps are easier than wholes".
The original findings also hold on twice as many papers (new App. Q): Opus 5.5 scores 0.97 on steps and 0.60 on whole papers, and the horizon doubles about every nine months.
W2. Research or reasoning?
We agree that the distinction matters, and a new paragraph in the introduction, "What ResearchTree measures", now states it. ResearchTree measures one part of research: reconstructing a long derivation from a precise statement of its goal. It does not measure choosing problems, searching the literature or running experiments.
W3. Claude builds and judges.
W4. Figure 1 markers. Fixed: Claude models are squares in every panel.
We hope this addresses your concerns, and we are glad to clarify anything further.