- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for autonomous optimization, where the goal is to improve the artifact through iterative experimentation, when the initial artifact, objective, and evaluator are given.
Critical strengths
None
Critical weaknesses
-
Limited Novelty: I think the proposed Arbor, based on Hypothesis Tree Refinement (HTR), is actually an existing concept. For example, MARS [1] proposes Comparative Reflective Memory which is highly similar to Hypothesis Tree Refinement strategy.
-
Reward Hacking Issue: The most critical weakness of this framework is that in Arbor Cycle, Decide process is done using the held-out merge gate. Since this process utilizes test set, I think it is a kind of reward hacking. I recommend authors to alleviate my concern, and also provide some ablation studies without the Decide process in Arbor Cycle.
-
Lack of benchmarks: The evaluation is only done on 6 tasks, and only the subset (i.e., Lite) version of MLE-Bench was utilized.
-
Lack of related works: Some related works are missing, e.g., MLE-STAR [2].
-
I looked through the generated ideas by Arbor. However, Arbor's research ability seems to remain far from that of human researchers, i.e., fail to identify a genuinely new mechanism.
[1] Chen et al., MARS: Modular Agent with Reflective Search for Automated AI Research, 2026
[2] Nam et al., MLE-STAR: Machine Learning Engineering Agent via Search and Target Refinement
Questions for authors
-
Can authors provide full results on MLE-Bench, including MLE-Bench-Medium, and MLE-Bench-Hard?
-
Highly appreciate for the result in Figure 3. However, can authors compare the Arbor with various backbones to some state-of-the-art baselines using same backbone?
-
Can Arbor be generalized beyond AI? E.g., domains such as biology, mathematics, and physics require benchmarks where valuable hypotheses are harder to specify and validate.
-
What is the key difference compared to MARS [1]? What is difference between HTR and Reflective Search, in terms of core idea?
[1] Chen et al., MARS: Modular Agent with Reflective Search for Automated AI Research, 2026\
9 comments
W2 — Reward Hacking
This is an important question, and we would like to clarify our reasoning. Early in writing, we used “held-out” to describe a subset whose underlying data, such as answers and detailed reasoning traces, were inaccessible, distinguishing it from training data. We subsequently recognized the ambiguity and changed the term to “admission” in the submitted version. Arbor trains and explores using training data, checks candidate improvements on an admission set serving a validation-like role, and evaluates generalization on separate held-out tasks, as in Figure 3(b). This protocol addresses the concerns about reward hacking and test-set use. Two further considerations motivate this design.
- In practical research, a fully held-out set is often unavailable. For our model-training and data-synthesis tasks, training, development, and admission conditions differ primarily in random seeds. In some outcome-driven settings, such as recommendation-model training or algorithm-efficiency optimization, a separate held-out set may also be impractical or unnecessary.
- Arbor fully supports final held-out testing. When such a test is available, the final artifact can be frozen and independently evaluated after optimization, as in Figure 3(b), with the existing framework.
The paper also already covers the requested evaluation without the DECIDE process through the following settings.
- In model training and data synthesis, development and admission differ primarily in random seeds, with similar underlying data distributions. DECIDE therefore checks whether candidate improvements hold under another set of random conditions.
- Figure 3(b) evaluates improvements independently of the optimization data. We freeze the BrowseComp-optimized harness and directly evaluate it on HLE and DeepSearchQA, which were excluded from optimization. Both improve, demonstrating gains beyond the admission set used for candidate selection.
W3 & Q1 — Benchmark Coverage
Our evaluation extends beyond six tasks. The six AO tasks in the main table span model training, harness engineering, and data synthesis, involving different artifacts and feedback mechanisms. We additionally conduct transfer experiments on two tasks excluded from optimization. Characterizing the evaluation as “only six tasks” understates its breadth and leaves out a substantial part of the experimental evidence.
As for MLE-Bench Lite, even recent large-scale foundation models use MLE-Bench Lite as an evaluation standard. For example:
- MiniMax-M2.7, released on March 18, 2026, reports results on all 22 Lite competitions in its official evaluation.
- Atria-Dawn-Preview, a 744B-parameter foundation model, reports Lite results in its technical report published on September 14, 2026.
The use of Lite to evaluate such capable foundation models supports its adequacy for evaluating our method. It also illustrates that exhaustive evaluation on the full benchmark is not a prerequisite for a meaningful assessment. Cost and the marginal value of additional evaluation are relevant considerations even for foundation-model developers, although their specific reasons for choosing Lite are not documented.
Our existing experiments have already consumed API usage equivalent to approximately US$100,000. We cannot currently afford the additional API and compute costs of evaluating the full benchmark. This is a resource constraint, rather than an observed inability of Arbor to solve harder tasks. HTR has no design restriction specific to the Lite split or its difficulty.
We acknowledge that Medium and Hard would extend difficulty coverage, and performance on these subsets has not yet been measured. However, the absence of these results does not, by itself, invalidate the conclusions supported by the existing experiments.
W4 — Related Work
Thank you for pointing out MLE-STAR. We have added citations and discussion in the main text and appendix. MLE-STAR describes itself as “a novel approach to build MLE agents”, with a scope focused on machine learning engineering. Arbor addresses a broader autonomous optimization setting spanning model training, harness engineering, and data synthesis.
The methodological distinction is also concrete. MLE-STAR constructs an initial solution through web search and uses code-block ablations to guide targeted refinement. Arbor organizes research hypotheses and experiments through a hypothesis tree, synthesizing local experimental findings into higher-level insights that guide further exploration. Both use experimental feedback, but they differ in their scope and in how they organize and accumulate research knowledge.
W5 & Q3 — Idea Novelty and Generality
The review states:
“I looked through the generated ideas by Arbor. However, Arbor's research ability seems to remain far from that of human researchers, i.e., fail to identify a genuinely new mechanism.”
We question both the criterion used here and the basis for this broad conclusion.
- The ability to propose valuable hypotheses depends fundamentally on the underlying model. Our contribution is an external research framework, rather than a new foundation model. However carefully we design the harness, external organization alone cannot give the model scientific judgment or research taste beyond its intrinsic capabilities. It is entirely unsurprising that current agents remain far behind professional researchers. We do not expect today’s agents to surpass domain experts. That gap does not negate Arbor’s contribution: improving how a given model organizes exploration, conducts experiments, and accumulates research progress. Requiring human-level originality to recognize this contribution would conflate the capabilities of the backbone with the value of the framework.
- Judging overall research ability from selected generated ideas is insufficient. Individual cases may reveal limitations, but they cannot substitute for an assessment of the full experimental evidence. Our results across tasks, transfer experiments, and component ablations provide a broader basis for evaluating Arbor. In particular, Table 3 shows that the hierarchy and upward insight feedback improve performance under the same backbone and wall-clock budget. These findings should be addressed when assessing whether Arbor supports more effective autonomous research.
Our central idea is to provide structured external guidance for autonomous research. HTR organizes three connected decisions: what to investigate next, what to learn from experiments, and how to accumulate progress. Hypotheses structure exploration, experimental findings update ancestor insights, and these insights guide subsequent research. This contribution does not require every generated idea to introduce a previously unknown scientific mechanism.
Regarding generality, the review states:
“E.g., domains such as biology, mathematics, and physics require benchmarks where valuable hypotheses are harder to specify and validate.”
We disagree that this constitutes a reason to question the applicability of our approach. Research in these fields progresses precisely by formulating hypotheses and testing them through experiments, proofs, simulations, or observations. Greater difficulty in specifying or validating hypotheses does not mean that these processes cannot be structured or supported by an external framework. The relevant question is whether a particular task provides an artifact or research state that can be modified and feedback that can guide further investigation.
Arbor can accommodate scalar or vector-valued feedback. Its applicability should therefore be considered at the level of concrete tasks, rather than ruled out by broad disciplinary labels. We have not yet validated Arbor in biology, mathematics, or physics, and mathematical theorem proving remains beyond its current demonstrated capabilities. These are boundaries of the present evidence; they do not establish that HTR is inherently unsuitable for those domains.
Thank you for reviewing our work. We appreciate your questions regarding novelty, evaluation, and generality. However, we find it difficult to understand how a paper receiving a score of 2 could be described as having no strengths whatsoever. In our academic experience, even for papers assigned scores of 1 or 0, we have rarely, if ever, encountered a review that identifies no strengths at all. This is particularly difficult to reconcile with your positive assessment of the paper’s clarity and your appreciation of Figure 3. It raises questions about the professional care given to the review, whether the substance of our contribution was fully understood, and whether personal sentiment influenced the assessment. We address the specific comments below and ask that the methodology and experimental evidence receive substantive consideration.
W1 & Q4 — Novelty
Trees are an established structure for organizing research and search, and both MARS and HTR use them. However, building on tree search and reflective memory does not make HTR equivalent to MARS. Our contribution is a research-state design that provides structured external guidance for autonomous research. The differences concern what the tree represents, how experiments relate to it, and how evidence changes subsequent research decisions.
We acknowledge MARS’s contributions and already cite it in §2 and include it as a baseline in Table 2. Arbor’s strongest configuration achieves a gold rate of 77.27%, compared with 53.03% for MARS. These configurations use different backbones, so this comparison alone does not isolate the contribution of the framework.
Table 3 provides more direct component-level evidence. Under the same backbone and wall-clock budget, the any-medal rate decreases from 81.82% for full Arbor to 63.64% without the hierarchy and 54.55% without upward insight feedback. Both variants retain access to leaf-level experimental evidence. These results support the value of how HTR organizes and synthesizes evidence beyond retaining experiment history. We ask that the novelty assessment address these specific mechanisms and results.