acceptodds
Under review as a conference paper at ICLR 2027

Toward Generalist Autonomous Research via Hypothesis-Tree Refinement

Abstract

Autonomous research requires agents to make sustained progress over long horizons by continuously refining hypotheses through experimental feedback. Although recent large language model agents can edit code, invoke tools, and execute experiments for extended periods, longer execution alone does not make research cumulative. Without an explicit research state, experiments remain isolated attempts, preventing evidence from being accumulated and reused to guide future exploration. We formulate this setting as Autonomous Optimization (AO), in which an agent iteratively improves an artifact under a fixed objective without step-level human supervision, and introduce Arbor, a general framework for AO. At the core of Arbor is Hypothesis Tree Refinement (HTR), an iterative algorithm that maintains a persistent hypothesis tree as the research state and continuously refines it as isolated experiments return evidence. By grounding each hypothesis in its artifact and evidence, and propagating findings to guide subsequent experiments, HTR connects local outcomes to cumulative hypothesis refinement. The tree serves as the search frontier, long-term memory, and audit trail. Across six real research tasks spanning model training, harness engineering, and data synthesis, Arbor improves the admission metric on every task under the same time budget, yielding roughly twice the gain of Codex and Claude Code. On MLE-Bench Lite it reaches a 77.3% gold rate, 24 points above the strongest prior system in our comparison. After optimization on BrowseComp, the frozen search harness transfers to unseen HLE and DeepSearchQA tasks, improving both without further optimization. Ab- lations and research traces support the value of connecting hypothesis management with evidence propagation for cumulative autonomous research.

Primary area:
optimization
Keywords:
agent, autonomous research, hypothesis tree

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qUNZOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for autonomous optimization, where the goal is to improve the artifact through iterative experimentation, when the initial artifact, objective, and evaluator are given.

Critical strengths

None

Critical weaknesses
  1. Limited Novelty: I think the proposed Arbor, based on Hypothesis Tree Refinement (HTR), is actually an existing concept. For example, MARS [1] proposes Comparative Reflective Memory which is highly similar to Hypothesis Tree Refinement strategy.

  2. Reward Hacking Issue: The most critical weakness of this framework is that in Arbor Cycle, Decide process is done using the held-out merge gate. Since this process utilizes test set, I think it is a kind of reward hacking. I recommend authors to alleviate my concern, and also provide some ablation studies without the Decide process in Arbor Cycle.

  3. Lack of benchmarks: The evaluation is only done on 6 tasks, and only the subset (i.e., Lite) version of MLE-Bench was utilized.

  4. Lack of related works: Some related works are missing, e.g., MLE-STAR [2].

  5. I looked through the generated ideas by Arbor. However, Arbor's research ability seems to remain far from that of human researchers, i.e., fail to identify a genuinely new mechanism.

[1] Chen et al., MARS: Modular Agent with Reflective Search for Automated AI Research, 2026
[2] Nam et al., MLE-STAR: Machine Learning Engineering Agent via Search and Target Refinement

Questions for authors
  1. Can authors provide full results on MLE-Bench, including MLE-Bench-Medium, and MLE-Bench-Hard?

  2. Highly appreciate for the result in Figure 3. However, can authors compare the Arbor with various backbones to some state-of-the-art baselines using same backbone?

  3. Can Arbor be generalized beyond AI? E.g., domains such as biology, mathematics, and physics require benchmarks where valuable hypotheses are harder to specify and validate.

  4. What is the key difference compared to MARS [1]? What is difference between HTR and Reflective Search, in terms of core idea?

[1] Chen et al., MARS: Modular Agent with Reflective Search for Automated AI Research, 2026\

9 comments

Thank you for reviewing our work. We appreciate your questions regarding novelty, evaluation, and generality. However, we find it difficult to understand how a paper receiving a score of 2 could be described as having no strengths whatsoever. In our academic experience, even for papers assigned scores of 1 or 0, we have rarely, if ever, encountered a review that identifies no strengths at all. This is particularly difficult to reconcile with your positive assessment of the paper’s clarity and your appreciation of Figure 3. It raises questions about the professional care given to the review, whether the substance of our contribution was fully understood, and whether personal sentiment influenced the assessment. We address the specific comments below and ask that the methodology and experimental evidence receive substantive consideration.

W1 & Q4 — Novelty

Trees are an established structure for organizing research and search, and both MARS and HTR use them. However, building on tree search and reflective memory does not make HTR equivalent to MARS. Our contribution is a research-state design that provides structured external guidance for autonomous research. The differences concern what the tree represents, how experiments relate to it, and how evidence changes subsequent research decisions.

  1. Nodes and parent–child relationships have different meanings. MARS nodes represent candidate solutions generated or modified through drafting, improvement, and debugging. HTR nodes represent research hypotheses at different abstraction levels. Parent–child relationships encode progressive refinement from broad directions to concrete interventions. Code ancestry is maintained separately.
  2. Hypothesis refinement and code evolution are decoupled. HTR executors implement assigned hypotheses starting from the current best artifact, independently of the code associated with the conceptual parent. Subsequent experiments in one direction can therefore build on code improvements obtained elsewhere.
  3. Evidence is organized and updated differently. MARS distills reusable lessons through comparative reflection. HTR progressively synthesizes local findings into ancestor insights, which guide subsequent hypothesis generation, selection, and pruning. Experimental results thus evaluate the current implementation and update the understanding of its broader research direction.
  4. Upward insight propagation explicitly connects experimental outcomes to future exploration. HTR propagates explanations, applicability conditions, and failure lessons from executed leaves through their ancestors. These updated insights are supplied to subsequent experiments and inform further research decisions. Knowledge can therefore accumulate even when the corresponding implementation is rejected. This connects individual experiments, broader research hypotheses, and subsequent exploration through an explicit update mechanism.

We acknowledge MARS’s contributions and already cite it in §2 and include it as a baseline in Table 2. Arbor’s strongest configuration achieves a gold rate of 77.27%, compared with 53.03% for MARS. These configurations use different backbones, so this comparison alone does not isolate the contribution of the framework.

Table 3 provides more direct component-level evidence. Under the same backbone and wall-clock budget, the any-medal rate decreases from 81.82% for full Arbor to 63.64% without the hierarchy and 54.55% without upward insight feedback. Both variants retain access to leaf-level experimental evidence. These results support the value of how HTR organizes and synthesizes evidence beyond retaining experiment history. We ask that the novelty assessment address these specific mechanisms and results.

W2 — Reward Hacking

This is an important question, and we would like to clarify our reasoning. Early in writing, we used “held-out” to describe a subset whose underlying data, such as answers and detailed reasoning traces, were inaccessible, distinguishing it from training data. We subsequently recognized the ambiguity and changed the term to “admission” in the submitted version. Arbor trains and explores using training data, checks candidate improvements on an admission set serving a validation-like role, and evaluates generalization on separate held-out tasks, as in Figure 3(b). This protocol addresses the concerns about reward hacking and test-set use. Two further considerations motivate this design.

  1. In practical research, a fully held-out set is often unavailable. For our model-training and data-synthesis tasks, training, development, and admission conditions differ primarily in random seeds. In some outcome-driven settings, such as recommendation-model training or algorithm-efficiency optimization, a separate held-out set may also be impractical or unnecessary.
  2. Arbor fully supports final held-out testing. When such a test is available, the final artifact can be frozen and independently evaluated after optimization, as in Figure 3(b), with the existing framework.

The paper also already covers the requested evaluation without the DECIDE process through the following settings.

  1. In model training and data synthesis, development and admission differ primarily in random seeds, with similar underlying data distributions. DECIDE therefore checks whether candidate improvements hold under another set of random conditions.
  2. Figure 3(b) evaluates improvements independently of the optimization data. We freeze the BrowseComp-optimized harness and directly evaluate it on HLE and DeepSearchQA, which were excluded from optimization. Both improve, demonstrating gains beyond the admission set used for candidate selection.

W3 & Q1 — Benchmark Coverage

Our evaluation extends beyond six tasks. The six AO tasks in the main table span model training, harness engineering, and data synthesis, involving different artifacts and feedback mechanisms. We additionally conduct transfer experiments on two tasks excluded from optimization. Characterizing the evaluation as “only six tasks” understates its breadth and leaves out a substantial part of the experimental evidence.

As for MLE-Bench Lite, even recent large-scale foundation models use MLE-Bench Lite as an evaluation standard. For example:

The use of Lite to evaluate such capable foundation models supports its adequacy for evaluating our method. It also illustrates that exhaustive evaluation on the full benchmark is not a prerequisite for a meaningful assessment. Cost and the marginal value of additional evaluation are relevant considerations even for foundation-model developers, although their specific reasons for choosing Lite are not documented.

Our existing experiments have already consumed API usage equivalent to approximately US$100,000. We cannot currently afford the additional API and compute costs of evaluating the full benchmark. This is a resource constraint, rather than an observed inability of Arbor to solve harder tasks. HTR has no design restriction specific to the Lite split or its difficulty.

We acknowledge that Medium and Hard would extend difficulty coverage, and performance on these subsets has not yet been measured. However, the absence of these results does not, by itself, invalidate the conclusions supported by the existing experiments.

W4 — Related Work

Thank you for pointing out MLE-STAR. We have added citations and discussion in the main text and appendix. MLE-STAR describes itself as “a novel approach to build MLE agents”, with a scope focused on machine learning engineering. Arbor addresses a broader autonomous optimization setting spanning model training, harness engineering, and data synthesis.

The methodological distinction is also concrete. MLE-STAR constructs an initial solution through web search and uses code-block ablations to guide targeted refinement. Arbor organizes research hypotheses and experiments through a hypothesis tree, synthesizing local experimental findings into higher-level insights that guide further exploration. Both use experimental feedback, but they differ in their scope and in how they organize and accumulate research knowledge.

W5 & Q3 — Idea Novelty and Generality

The review states:

“I looked through the generated ideas by Arbor. However, Arbor's research ability seems to remain far from that of human researchers, i.e., fail to identify a genuinely new mechanism.”

We question both the criterion used here and the basis for this broad conclusion.

  1. The ability to propose valuable hypotheses depends fundamentally on the underlying model. Our contribution is an external research framework, rather than a new foundation model. However carefully we design the harness, external organization alone cannot give the model scientific judgment or research taste beyond its intrinsic capabilities. It is entirely unsurprising that current agents remain far behind professional researchers. We do not expect today’s agents to surpass domain experts. That gap does not negate Arbor’s contribution: improving how a given model organizes exploration, conducts experiments, and accumulates research progress. Requiring human-level originality to recognize this contribution would conflate the capabilities of the backbone with the value of the framework.
  2. Judging overall research ability from selected generated ideas is insufficient. Individual cases may reveal limitations, but they cannot substitute for an assessment of the full experimental evidence. Our results across tasks, transfer experiments, and component ablations provide a broader basis for evaluating Arbor. In particular, Table 3 shows that the hierarchy and upward insight feedback improve performance under the same backbone and wall-clock budget. These findings should be addressed when assessing whether Arbor supports more effective autonomous research.

Our central idea is to provide structured external guidance for autonomous research. HTR organizes three connected decisions: what to investigate next, what to learn from experiments, and how to accumulate progress. Hypotheses structure exploration, experimental findings update ancestor insights, and these insights guide subsequent research. This contribution does not require every generated idea to introduce a previously unknown scientific mechanism.

Regarding generality, the review states:

“E.g., domains such as biology, mathematics, and physics require benchmarks where valuable hypotheses are harder to specify and validate.”

We disagree that this constitutes a reason to question the applicability of our approach. Research in these fields progresses precisely by formulating hypotheses and testing them through experiments, proofs, simulations, or observations. Greater difficulty in specifying or validating hypotheses does not mean that these processes cannot be structured or supported by an external framework. The relevant question is whether a particular task provides an artifact or research state that can be modified and feedback that can guide further investigation.

Arbor can accommodate scalar or vector-valued feedback. Its applicability should therefore be considered at the level of concrete tasks, rather than ruled out by broad disciplinary labels. We have not yet validated Arbor in biology, mathematics, or physics, and mathematical theorem proving remains beyond its current demonstrated capabilities. These are boundaries of the present evidence; they do not establish that HTR is inherently unsuitable for those domains.

Reviewer_KANxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formalized auto-research as Autonomous Optimization and came up with Arbor. In Arbor, a coordinator keeps a hypothesis tree, and multiple temporal executor test one hypothesis in each of their isolated worktrees. The insight gained from experiments propagates upward, and only when an independent admittance evaluator see gains on the plans to be selected, the plans will be merged. In a 48-hour budget, Arbor gains the best admittance scores on all 6 AO tasks; Compared with the better ones between codex and claude code, the median of rise is 2.16x.

Critical strengths

This design decouples hypothesis hierachy from code lineage, and force role boundries in tool level, so every change to an artifact can be traced back to a specific hypothesis, which is reusable.

Critical weaknesses
  1. The gains cannot yet be attributed to HTR itself. The final harness of BrowseComp requires 5 rollouts and 1 sharing, but the paper didn’t report the initial and final harness’ cost of each task. Moreover, the paper although reported the token usage of agents, but provided too few info on eval costs and throughputs, so the total cost of every method isn’t comparable. And no run using tree-search agents is conducted on AO task set with the same backbone and budget. Without these ablations, the result cannot tell how much gain on AO is from HTR.
  2. The evidence is thin compared with the paper’s claim. The paper didn’t report independent repeated search results under the same conditions. And although the admittance eval is used in selection, the 6 AO results didn’t have any independent tests after seatch. The ablation is run only once for every variation on MLE-Bench Lite, so it can support the comprehensive function of all mechenism, but cannot seperate the function of hierachy itself to the context and tool call’s function it introduced. Since many difference is thin, such as the gain in the 53 tasks from Terminal-Bench is 4 vs 2, the robustness of the reported gain is unclear.
Questions for authors
  1. What is the per-task inference costs of BrowseComp’s initial and final harness? How an easy multi-rollout baseline performs under the same budget?
  2. Except for reported agent’s token usage and Arbor’s admittance calls, how many experiments and development evaluations every method completes? How many admittance evaluations every baseline completes? How many executor runs Arbor starts, and how many of them completes? What hardware, parallel processing capabilities, and access to admissions evaluators were used by each method?
  3. Except for re-evaluating the final artifact, could you report the independent reproduction results of full search processes and ablation studies for at least part of the AO tasks?
7 comments

Thank you for recognizing the decoupling of hypothesis hierarchy and code lineage, as well as the tool-level role boundaries! We appreciate your questions about computational cost, attribution, and robustness, and provide additional measurements and experimental details below.

W1 & Q1 — BrowseComp Inference Cost and Efficiency

Thank you for raising this question. We measured the average per-question token usage of the initial and evolved harnesses and compared them with a simple multi-rollout baseline using majority voting. We set N=7 for this baseline to obtain an inference cost comparable to that of Arbor’s final harness.

MethodAverage tokens per questionAccuracy
Initial harness0.29M45.33%
Simple N=7 multi-rollout baseline2.08M62.33%
Arbor’s final harness1.81M67.67%

Arbor’s evolved harness achieves 5.34 percentage points higher accuracy while using fewer tokens per question than the simple multi-rollout baseline. This comparison supports the effectiveness of the evolved harness design at a comparable inference cost.

Q2 — Experimental Throughput, Evaluation Costs, and Resources

Thank you for identifying this reporting gap. We provide detailed statistics for BrowseComp as a representative example. This task uses 40 development questions and 300 admission questions, with a 48-hour search-time cap for each method.

StatisticArborClaude Code
Candidate configurations with completed full development evaluations103
Completed full development evaluations103
Additional small-sample development evaluations9, with 2 questions eachComplete count unavailable in the current archive
Candidate admission evaluation attempts43
Completed candidate admission evaluations41
Incomplete candidate admission evaluations02

Experiment execution. Arbor launched 14 executor runs. Of these, 10 completed and returned valid development scores, 3 timed out, and 1 terminated because of an LLM-call failure. An executor may perform a small-sample check before a full development evaluation, so executor runs and evaluator calls are counted separately.

Admission access. Both methods could invoke the admission evaluator during search, without an additional call quota. Arbor’s coordinator initiated candidate evaluations through the merge gate, and 2 of its 4 candidate admission evaluations resulted in merges. Claude Code determined when to invoke the evaluator, completing 1 of its 3 attempts and stopping the other 2 before completion. The table counts evaluations of optimization candidates. Arbor also initiated an admission re-evaluation of the initial harness near the end of the run, which was subsequently stopped. We report this separately as additional evaluation overhead.

Hardware and parallelism. All experiments ran on servers equipped with 4 NVIDIA A100 GPUs. Arbor ran up to 3 executors concurrently, each implementing and evaluating a candidate in an isolated worktree. Full development and admission evaluations used 8 question-level workers. Executor parallelism specifies the number of candidates explored concurrently, while question-level parallelism specifies the number of questions processed concurrently within an evaluation. Arbor’s search lasted approximately 42.2 hours.

W2 & Q3 — Independent Search Repeats

Thank you for suggesting additional experiments to assess robustness. We conducted three independent full-search runs on each of Optimizer Design, Architecture Design, Terminal-Bench, and BrowseComp. The admission scores of the final artifacts are reported below.

TaskArbor #1Arbor #2Arbor #3Claude CodeCodex
Optimizer Design, steps ↓3237.53250.03225.03287.53325
Architecture Design, loss ↓1.0281.0301.0291.0331.083
Terminal-Bench, accuracy ↑77.36%77.36%79.25%71.70%73.59%
BrowseComp, accuracy ↑67.67%69.33%66.67%53.33%50.00%

Across all four tasks, every Arbor run outperforms both baseline results reported in the table. These independent search repeats support the consistency of Arbor’s gains across runs.

Thanks for the additional results. The majority-voting comparison and the throughput numbers answer my first two questions. After reading the rest of the discussion and going back to the paper, I still have a few questions.

  1. The ablation shows that the tree and the insight feedback matter inside Arbor, but there is still no comparison with an existing tree-search research agent on the AO tasks. On MLE-Bench Lite, the any-medal gap is small once the backbone is matched, for example 81.82 vs 77.27 with Opus 4.6. Running something like MARS or ML-Master with the same backbone on a few AO tasks would make the contribution much clearer. Relatedly, the 24-point gain over the strongest prior system highlighted in the abstract compares systems with different backbones, so it is hard to tell how much of it comes from the framework itself.

  2. On BrowseComp, majority voting at a similar cost already goes from 45.33 to 62.33, and Arbor reaches 67.67, so most of the gain seems to come from extra test-time compute. I see the two-round evidence sharing described in the appendix, but the remaining gap is measured on the admission set that was used for selection, so it may be a bit optimistic. How does the same majority-voting baseline do on HLE and DeepSearchQA? Since those tasks were not used during search, this would be a clean way to see whether the transferred gains come from the harness design or from the extra compute.

  3. The new repeated runs are helpful, but they still report admission scores, which are also used for selection. So they show that the results are stable rather than giving an unbiased estimate. I would also still like to see repeats of the ablation, since each variant is a single run and the differences come down to a few competitions out of 22.

  4. How is the admission set constructed for MLE-Bench Lite, given that the private test labels are not available during search? And are the baseline numbers single runs or averages over seeds? Arbor's results are single runs, so it would help to know whether the comparison is like for like.

  5. The title and abstract present this as generalist autonomous research, but all the tasks are ML or LLM engineering problems with a fixed metric and an automatic evaluator. I think "autonomous optimization", the term the paper itself uses, describes the work better, and the framing could be toned down accordingly.

  6. Minor: despite the terminology update mentioned in the rebuttal, the paper still calls the admission set "held-out" in some places, and the task prompts call it the test split. It would be good to make this consistent.

I will take your answers into account for my final score.

Q1 & Q3

Thank you for the questions and suggestions. We considered adapting tree-search research agents such as ML-Master and MARS to autonomous optimization (AO) tasks at the outset. Arbor offers two main advantages.

  • Task generality. ML-Master embeds MLE-specific requirements, such as generating training and prediction scripts, reporting validation metrics, and producing submission.csv, into candidate generation and validity checks. These requirements constrain the accepted artifacts, experiment execution, and feedback, making direct application to tasks such as harness engineering and data synthesis difficult. We could not inspect MARS at the implementation level because its code is unavailable. Arbor defines each task through initial materials, an optimization objective, and an evaluator, allowing diverse artifacts and feedback to enter a unified research workflow. This task formulation and general-purpose design are central contributions of our work.

  • Hypothesis-level reasoning. ML-Master and MARS primarily organize search around candidate solutions, using refinement and experience reuse to guide optimization. Arbor explicitly represents research directions as hierarchical hypotheses and separately maintains code evolution. Multiple implementations can test the same hypothesis, and their results can inform shared higher-level hypotheses. This allows Arbor to aggregate evidence when deciding whether to refine, revise, or prune a research direction. For example, evidence from related experiments can help distinguish an implementation failure from a flawed hypothesis. HTR summarizes these findings into insights at multiple levels, providing an accumulating, revisable research state for long-horizon exploration.

Although directly adapting ML-Master to our AO setting would require substantial changes, we conducted additional ablations to assess the contributions of Arbor’s tree structure and insight feedback across AO tasks.

SettingOptimizer Design<br>Steps ↓Architecture Design<br>Loss ↓Terminal-Bench<br>Accuracy (%) ↑
Arbor3237.51.02877.36
w/o tree33001.03973.58
w/o insight feedback3262.51.03671.70

Full Arbor performs best on all three tasks, providing further support for both components across different AO settings.

Q2

Thank you for the suggestion. We agree that the substantial improvement from N=7 majority voting on BrowseComp motivates testing whether the evolved harness provides additional gains over multiple rollouts.

We therefore evaluated the N=7 majority-voting baseline on HLE and DeepSearchQA, neither of which participated in the BrowseComp harness search. The evolved harness remained frozen, with no further optimization on either transfer task.

MethodBrowseComp Accuracy (%)HLE Accuracy (%)DeepSearchQA Accuracy (%)
Initial harness45.3325.5061.00
N=7 majority voting62.3329.5063.00
Arbor-evolved harness67.6731.5069.00

The frozen Arbor harness outperforms N=7 majority voting by 2.00 percentage points on HLE and 6.00 points on DeepSearchQA. These results support the transferability of improvements discovered during harness search.

The larger gain on DeepSearchQA may reflect its answer format. For questions requiring a complete answer set, different rollouts may discover complementary correct items that simple answer-level voting struggles to combine. The evolved harness’s two rounds of evidence sharing may help integrate this information. This remains a possible mechanism, and its contribution requires targeted ablation.

Reviewer_3wgROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents Arbor, a tree-search-based framework for automated machine learning engineering. It organizes the exploration and iterative refinement of candidate solutions into a structured search process, aiming to guide candidate selection and allocate resources across costly ML experiments. The framework addresses the practical challenge of balancing exploration of alternative approaches with continued improvement of promising solutions. The authors evaluate Arbor on MLE-Bench and compare it with existing ML engineering agents to assess whether its search design improves end-to-end performance on realistic machine learning tasks.

Critical strengths
  1. The paper focuses on search and iterative optimization within automated machine learning (AutoML) engineering—a problem with clear practical value, particularly regarding candidate selection and resource allocation under conditions where experiments are costly.
  2. The use of MLE-Bench provides a standardized environment for evaluating the end-to-end performance of methods on ML tasks, while also facilitating comparisons with existing ML.
Critical weaknesses
  1. Tree search and this type of iterative optimization seem quite common; it is not entirely clear where the specific novelty lies.
  2. The experiments rely on MLE-Bench, a benchmark that has largely fallen out of favor. Current research tends to focus on benchmarks such as PostTrainBench, MLS-Bench, SWE-Marathon, AutoResearchExam, and RSI_Exam.
  3. Newer models have emerged since the time of this study, yet Table 2 still uses older models. It is possible that these newer models have already closed the performance gap that Arbor claims to address. It seems like the authors just resubmit an old paper again without any updates.
Questions for authors

Please refer to the weaknesses above.

8 comments

Thank you for reviewing our work. We appreciate your concerns regarding novelty, evaluation scope, and model choice. However, several judgments overlook designs and experiments already presented in the manuscript. We believe the review, in its current form, falls short of a fair and rigorous assessment, and some of its characterizations have left us feeling offended. We address these points below and ask that the assessment be reconsidered in light of the evidence. Although the review raises only three weaknesses, our response is much longer than would normally be needed to address three points, because we feel our work has been unfairly judged. Please do not take the length of this response as evidence that it was generated by AI. The frustration and sense of offense expressed here are our own. An AI cannot experience these feelings on our behalf.

Before addressing the individual weaknesses, we would first like to clarify a fundamental misunderstanding of our problem formulation in the review. The review’s strengths section frames our work as addressing AutoML. This mischaracterizes the problem studied in our paper. Both the Introduction and the opening of section 3, “Task Formulation,” explicitly define Autonomous Optimization (AO): iteratively improving a given artifact under a specified objective and evaluator. AO is not equivalent to AutoML; our experiments span model training, harness engineering, and data synthesis. We find it difficult to understand why a definition stated explicitly and emphasized throughout the manuscript is replaced by a different problem framing in the review. This raises further concerns about whether the manuscript, including its basic formulation, was read with sufficient care.

W1: Novelty

Reducing our contribution to tree search conflates a general search technique with the research mechanism we propose. Building on tree search does not make our method equivalent to tree search.

Our central idea is to guide autonomous research through a structured external research state. This state records hypotheses, artifacts, evidence, and insights, and actively guides further exploration. HTR uses a hypothesis tree to connect three decisions: what to investigate next, what to learn from experiments, and how to accumulate progress.

  1. Organizing exploration through hypotheses. HTR separates the conceptual hierarchy of hypotheses from code ancestry. Tree edges represent refinement from broad research directions to executable interventions. Experiments start from the current incumbent, so a research direction can build on artifact improvements obtained elsewhere. The tree therefore organizes research questions, not merely candidate code versions.
  2. Turning experimental outcomes into reusable knowledge. Local findings—including failures, applicability conditions, and evidence-supported explanations—are synthesized into ancestor insights. These insights guide subsequent hypothesis generation, selection, and pruning. An unsuccessful experiment can therefore change future research decisions even when its code is discarded. The tree supplies structured guidance for exploration, rather than simply storing an experiment history.
  3. Separating evidence accumulation from artifact promotion. Hypothesis-bound executors return traceable implementations, results, and insights from isolated experiments. The coordinator incorporates this evidence into the research state. Candidate code becomes the incumbent only after improving the admission metric. Knowledge can therefore accumulate even when an implementation is rejected. HTR explicitly manages both the evolving understanding of the problem and the artifact being improved.

Table 3 provides component-level evidence. Under the same backbone and wall-clock budget, the any-medal rate decreases from 81.82% for full Arbor to 63.64% without the hierarchy and 54.55% without upward insight feedback. Both variants retain access to leaf-level experimental evidence. These results support the value of how HTR organizes and synthesizes evidence to guide research, beyond simply retaining past experiments or using a tree-shaped search space.

W2: Benchmark Choice

The review states, “The experiments rely on MLE-Bench, a benchmark that has largely fallen out of favor.” Neither part of this statement holds: the first overlooks our other experiments, and the second lacks supporting evidence.

  1. Regarding the first part, we also evaluate six AO tasks, presented in the main results table, Table 1. They cover model training, harness engineering, and data synthesis. MLE-Bench Lite is reported separately in Table 2. Section 5.3 also evaluates the frozen BrowseComp-optimized harness on HLE and DeepSearchQA. Both improve without target-task optimization or feedback. Why does the review focus only on MLE-Bench without addressing these experiments?

  2. Regarding the second part, official evaluations of foundation LLMs still use MLE-Bench and its Lite version. Its use extends beyond comparisons between research frameworks to models with hundreds of billions of parameters:

    ModelTotal parametersRelease / publication dateOfficial evaluation
    MiniMax M2.7Approximately 230BReleased March 18, 2026MiniMax’s official release report includes MLE-Bench Lite results. (source)
    Gemini 3.6 FlashUndisclosedReleased July 21, 2026Google’s official model card reports MLE-Bench results. (source)
    Atria Dawn Preview744BTechnical report published September 14, 2026Shanghai AI Laboratory includes MLE-Bench Lite in its main evaluation table. (source)

    With current foundation LLMs still evaluated on this benchmark, what supports the claim that it has “largely fallen out of favor”?

  3. Listing newer benchmarks does not establish that our evaluation is inadequate. Their public release timelines are as follows:

    BenchmarkFirst public paper or resource releaseTime before the September 25, 2026 deadline
    PostTrainBenchMarch 9, 2026: initial paper200 days (source)
    MLS-BenchMay 9, 2026: initial paper139 days (source)
    SWE-MarathonJune 5, 2026: initial paper112 days (source)
    RSI-ExamAugust 28, 2026: public data and code28 days (source)
    AutoResearchExamSeptember 9, 2026: benchmark release16 days (source)

    Two of these benchmarks released their public resources less than one month before the deadline. The reviewer provides no evidence that they had already become widely and independently validated replacement standards.

    SWE-Marathon includes tasks such as library reproduction and application cloning. Long-horizon software completion does not automatically constitute AO as defined in our paper: continuously improving an existing artifact through experimental feedback. The reviewer does not explain how these tasks test our claims. Requiring SWE-Marathon as a mandatory additional evaluation is therefore even unreasonable here. (benchmark paper) We ask the reviewer to specify: which concrete claim in our paper is unsupported by the existing experiments?

W3: Model Choice

We are frustrated and offended by the claim that we simply resubmitted an old paper without updates. This is a serious allegation, and the review provides no evidence to support it. We welcome criticism of our work. We also expect that criticism to be accurate and grounded in the current submission. We address the specific concerns below.

  1. The claim of an unchanged resubmission is unsupported. The reviewer writes, “It seems like the authors just resubmit an old paper again without any updates.” No version comparison or specific evidence follows this statement. It is therefore subjective speculation. We even perceive an element of malice in this unsupported allegation. We ask the reviewer to provide a concrete comparison or withdraw the statement. The current submission should be assessed on its methodology and evidence.

  2. Our experiments already examine Arbor’s contribution across multiple backbones. Table 2 includes GPT-5.5, Claude Opus 4.6, and Gemini 3 Flash. Section 5.3 further examines performance across backbones. Our central question is straightforward: does Arbor’s research organization and experimental decision-making improve autonomous optimization with a given backbone? With GPT-5.5, Table 2 reports gold rates of 77.27% for Arbor and 46.97% for Codex. This is a difference of 30.30 percentage points. The comparison supports Arbor’s effectiveness with the same backbone.

  3. Stronger models do not automatically erase Arbor’s relative gains. New models may improve absolute performance. Their release alone does not show that Arbor’s advantages have disappeared. If a missing model materially affects the assessment, please name it. Please also explain which capability matters and which experimental conclusion its omission undermines. A general reference to “stronger models” is not enough.

  4. Model recency should be judged against the submission deadline. The table below lists the evaluated models and subsequent releases available by September 25, 2026.

    Evaluated modelRelease dateSubsequent releases before the deadline
    GPT-5.5April 23, 2026 (official record)GPT-5.6 family: July 9; GPT-6 Astra: September 3; GPT-6 Sol/Luna: September 22. (release history)
    Claude Opus 4.6February 5, 2026Opus 4.7: April 16; Opus 4.8: May 28; Opus 5: July 24; Opus 5.5: September 22. (release history)
    Gemini 3 FlashDecember 17, 2025Standard Flash successors: 3.5 Flash: May 19, 2026; 3.6 Flash: July 21; 3.7 Flash: August 13; 3.8 Flash: September 2. (release history)

    Gemini 3 Flash had no announced standard Flash successor for about five months. The next generation, Gemini 3.5 Flash, arrived on May 19, 2026. Google released other model variants during that period, but those were separate product lines. (release history)

    GPT-5.5 was only about five months old at the submission deadline. If even a model released within six months is considered too old, that is an unreasonable standard for an empirical research paper. Some later models appeared just three days before the deadline. Their release does not invalidate completed experiments. It certainly does not support the allegation that we resubmitted an old paper without updates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.