acceptodds
Under review as a conference paper at ICLR 2027

Prime Scientist: Strategic Allocation of Research Effort in Autonomous Research

Abstract

Autonomous research agents aim to automate scientific workflows, from proposing ideas to conducting experiments and analyzing results. Yet current AI and research agents can propose more directions than available resources allow them to pursue. Moreover, each attempt could consume substantial resources, requiring agents to reconsider how to invest in subsequent research. Thus, deciding how to invest research effort strategically should be a defining capability of autonomous research agents. Accordingly, we introduce PrimeScientist, which jointly determines research direction and resource investment across successive research attempts. Specifically, we formulate this challenge of strategic research effort allocation as a sequential decision problem where remaining resources should explicitly guide the research policy. We first introduce an executable plan tree that preserves competing plans and their outcomes across attempts. Building on this representation, we propose an adaptive MCTS-based allocation policy that balances exploration and exploitation using experimental feedback and remaining resources. Comprehensive evaluations across AI research, systems and code optimization, and machine learning engineering show that strategic allocation improves research quality and sample efficiency together. Across 12 AI research tasks, PrimeScientist improves average reward by 10.3% with 50.6% fewer research attempts than AutoResearch under the same resource budget. We believe making research effort allocation an explicit optimization target establishes effective resource use as a core research capability for autonomous agents to drive scientific breakthroughs at scale. Code and data are available at https://github.com/Henri-XYu02/PrimeScientist.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Autonomous Research, LLM Agents, Budget Allocation, Tree Search

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9sPCOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how an autonomous research agent should divide a shared token budget between drafting plans and executing them. PrimeScientist keeps a tree of executable plans and samples branches by their observed value, concentrating on high-value branches as the budget runs out. On 27 tasks (FIRE-Bench, AutoLab, MLE-Bench) with GPT-5 and a 1.5M-token budget, it reports better or comparable scores than AutoResearch with fewer executor invocations.

Critical strengths
  1. Charging planning and execution to one token budget is a fair and useful evaluation protocol, since many agent papers leave planning overhead uncounted.
  2. The paper is candid about its evidence: it reports the 38% planning share, per-task exceptions in the ablations, and states that the comparisons are descriptive rather than significance tests.
Critical weaknesses
  1. Limited novelty. The method combines MCTS-style search, a reflector and an executor, and its only increment is a budget-dependent sampling schedule that ignores each candidate’s expected cost, room for improvement and information value (e.g., a high-scoring direction that needs heavy training is not necessarily a better investment than a cheap, promising new one), which falls short of the claimed “strategic effort allocation”.
  2. Unvalidated pruning. Pruning affects both exploration and value backpropagation, yet there is no no-pruning ablation or threshold sensitivity analysis, so it may prematurely discard directions that score low only temporarily or due to execution randomness.
  3. Inconsistent and unreliable results. On AutoLab, AutoResearch scores higher on 5 of 8 tasks and the averages are essentially identical once Smallest Game Player is excluded, while the GPT-5 main results (Tables 1–2) rest on a single search per configuration and the GPT-5-mini repeats cannot establish their stability.
  4. Anonymity violation. The abstract on the OpenReview submission page includes a non-anonymized GitHub link, which leaks author identity and harms double-blind review.
Questions for authors
  1. Please report results with pruning disabled and for several values of δ\delta (e.g., 0.02, 0.05, 0.10) on the 8-task ablation subset, together with the number of pruned nodes per search.
  2. How often would a pruned plan have cleared the threshold on a second execution? Re-running a sample of pruned plans would quantify pruning caused by execution randomness.
  3. Can you report Tables 1 and 2 as mean and 95% confidence intervals over at least three independent GPT-5 searches per configuration?
  4. A paired per-task comparison against AutoResearch (win/tie/loss counts with a bootstrap or sign test) on each benchmark would help; is the difference significant on any of them?
  5. What happens if the selection weight also includes a per-branch cost estimate, such as the parent’s token consumption or the reflector’s predicted cost?
  6. Can you report the average token cost per attempt for each branch depth or task, so that readers can judge whether high-value branches are also the expensive ones?
Reviewer_hJ6rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies strategic research effort allocation: how an autonomous research agent should distribute effort across research directions as evidence accumulates and resources are consumed. PrimeScientist formulates plan construction and execution as a sequential decision problem under a shared token budget, keeps competing plans and their outcomes in an executable plan tree, and chooses which plan to work on with an MCTS-based policy whose selection concentrates on high-value branches as the remaining budget shrinks. Using GPT-5 through the Codex CLI with a 1.5M-token budget, it is evaluated on 27 tasks against AutoResearch, AutoLab and a single-run agent, and reports better or comparable scores with fewer research attempts.

Critical strengths

(+) The paper counts the tokens spent writing and revising plans in the same budget as the tokens spent running experiments. Its tree of written plans keeps alternatives and their results on record, so a search can return to an earlier direction and a reader can see what was tried, pruned or never run, and results report the number of attempts and runtime alongside reward.

Critical weaknesses

(---) The closest prior methods the paper cites are never compared, including MARS, AIDE, AlphaLab's Strategist and FML-Bench's AdaptiveSearch, and the only controlled baselines are a single-run agent and AutoResearch's greedy loop. Its comparison with UCT covers 8 tasks with one search each.

(---) Only the selection rule is ablated, so the contribution of the reflector, the plan tree and pruning is unknown.

(--) Main results come from one search per configuration, and on AutoLab AutoResearch matches or beats PrimeScientist's reward on 6 of 8 tasks.

(--) Branch values are unnormalized averages, so when rewards are close the budget schedule barely concentrates selection (two branches at 0.64 and 0.60 give only about 1.9 : 1 odds even at the maximum exponent), and averaging conflicts with the objective of finding the single best result.

Questions for authors
  • Does AutoResearch, prompted with the remaining budget as BATS does for tool calls, achieve similar savings in attempts?
  • On FIRE-Bench, does AutoResearch also get two executions per change with the higher score kept?
  • What happens when the reflector, the plan tree or pruning is removed, one at a time?
  • How does PrimeScientist compare with the closest cited methods, such as AIDE and MARS, on the MLE-Bench tasks?
  • Which model, agent, time budget and metric produced the AutoLab column in Table 1, and why is it listed as a single attempt?
Reviewer_1DDpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper claims that autonomous research agents can propose more directions than their budget allows them to test, so distributing effort across directions should be an explicit capability. The authors formulate plan construction and execution as a sequential decision problem under one token budget that covers both. A reflector agent proposes and revises plans stored in an executable plan tree, and a coding agent executes them. An MCTS-style policy selects which plan to expand or execute, with a sampling exponent designed to concentrate selection on high-value branches as the budget runs out.

The contributions are as follows:

  1. A formulation of strategic research effort allocation in which plan construction and complete research attempts are charged to one shared token budget.
  2. PrimeScientist, which combines an executable plan tree with a budget-adaptive selection rule, α = min(1/r, 10), where r is the remaining fraction of the budget. Each plan is a Markdown document, and each modification is a runnable diff.py.
  3. Experiments on 27 tasks showing better or comparable outcomes with fewer research attempts than AutoResearch for 26/27 of the tasks.

Validation:

  1. Main comparison (Tables 1-3). With GPT-5 and a 1.5M-token budget, the authors compare PrimeScientist with AutoResearch and a single-run agent on AutoLab, FIRE-Bench, and MLE-Bench, using one search per configuration.
  • AutoLab (8 tasks): 0.482 vs 0.443 with 10.0 vs 17.5 attempts.
  • FIRE-Bench (12 tasks): PrimeScientist reports 0.774 vs 0.702 average reward with 13.3 vs 27.0 attempts.
  • MLE-Bench (7 tasks): PrimeScientist leads on four of seven competitions.
  1. Policy ablations (Tables 6-7). With GPT-5 on eight tasks, the authors report higher mean reward for adaptive selection (0.598) than for UCT (0.551), Greedy (0.563) and Random (0.546). With GPT-5-mini on six tasks, they report higher mean reward than with fixed exponents (best 0.507).
  2. Consistency and resources (Table 8, Appendix F). Three GPT-5-mini searches on AutoLab give comparable reward (0.4363 vs 0.4359) with fewer attempts (21.6 vs 28.4).
Critical strengths
  1. The paper charges planning and execution to one token budget, which existing autoresearch systems do not. Table 9 shows planning takes 38% of tokens, a cost earlier comparisons ignore. This matters to my assessment because it is the right accounting for comparing research agents, whatever this method’s results.
  2. The executable plan tree makes research decisions traceable. Each plan is a complete document, and each modification is a runnable diff.py with a prior and rationale (Appendices H-I). This matters because auditing agent-driven research requires this kind of record.
Critical weaknesses
  1. The efficiency claim largely follows from the accounting, and the quality gains do not survive repetition. Spending 38% of tokens on planning leaves fewer for execution, so fewer attempts on 26 of 27 tasks is mostly built in. On quality, PrimeScientist loses 4 of 12 FIRE-Bench and 3 of 7 MLE-Bench tasks. Headline results come from one search per configuration, and repeated runs (Table 8) show no reward difference (0.4363 vs 0.4359). This is the main reason for my rating, since the central claim is that allocation makes attempts more productive.
  2. The experiments do not isolate the allocation policy. Random (0.546) nearly matches UCT (0.551), suggesting the reflector does most of the work. Fixing α costs at most 0.03 (0.507 vs 0.537), and adaptive leads on only 3 of 6 tasks (Table 7). The only external baseline is linear AutoResearch while the closest cited agents (AIRA, MARS, AlphaLab, AdaptiveSearch) are not compared.
Questions for authors
  1. Reflector-matched control. Can you add an AutoResearch arm that uses the same reflector and plan prompts but keeps a single linear trajectory, under the same token budget? This would separate better plans from branching and allocation.
  2. Seeds and run parity. Can you report at least three GPT-5 searches per configuration for Tables 1-3, with paired confidence intervals? Why do fixed-α arms have two searches rather than three (Table 7)? On FIRE-Bench, does AutoResearch also get best-of-two executions?
  3. Ablation task selection. How were the tasks in Tables 6-7 chosen? Why were SECA Hallucination and Gaussian Blur, the two tasks where Random led in Table 6, excluded from Table 7?
  4. Baselines and budgets. Can you compare against at least one cited tree-search agent, such as AIRA-MCTS or AdaptiveSearch, at a matched budget?
  5. The paper argues fewer attempts are valuable because experiments are costly, but only tokens are measured. Can you report execution compute (e.g., GPU hours) for both methods, or compare them at the same number of attempts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.