- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how an autonomous research agent should divide a shared token budget between drafting plans and executing them. PrimeScientist keeps a tree of executable plans and samples branches by their observed value, concentrating on high-value branches as the budget runs out. On 27 tasks (FIRE-Bench, AutoLab, MLE-Bench) with GPT-5 and a 1.5M-token budget, it reports better or comparable scores than AutoResearch with fewer executor invocations.
Critical strengths
- Charging planning and execution to one token budget is a fair and useful evaluation protocol, since many agent papers leave planning overhead uncounted.
- The paper is candid about its evidence: it reports the 38% planning share, per-task exceptions in the ablations, and states that the comparisons are descriptive rather than significance tests.
Critical weaknesses
- Limited novelty. The method combines MCTS-style search, a reflector and an executor, and its only increment is a budget-dependent sampling schedule that ignores each candidate’s expected cost, room for improvement and information value (e.g., a high-scoring direction that needs heavy training is not necessarily a better investment than a cheap, promising new one), which falls short of the claimed “strategic effort allocation”.
- Unvalidated pruning. Pruning affects both exploration and value backpropagation, yet there is no no-pruning ablation or threshold sensitivity analysis, so it may prematurely discard directions that score low only temporarily or due to execution randomness.
- Inconsistent and unreliable results. On AutoLab, AutoResearch scores higher on 5 of 8 tasks and the averages are essentially identical once Smallest Game Player is excluded, while the GPT-5 main results (Tables 1–2) rest on a single search per configuration and the GPT-5-mini repeats cannot establish their stability.
- Anonymity violation. The abstract on the OpenReview submission page includes a non-anonymized GitHub link, which leaks author identity and harms double-blind review.
Questions for authors
- Please report results with pruning disabled and for several values of (e.g., 0.02, 0.05, 0.10) on the 8-task ablation subset, together with the number of pruned nodes per search.
- How often would a pruned plan have cleared the threshold on a second execution? Re-running a sample of pruned plans would quantify pruning caused by execution randomness.
- Can you report Tables 1 and 2 as mean and 95% confidence intervals over at least three independent GPT-5 searches per configuration?
- A paired per-task comparison against AutoResearch (win/tie/loss counts with a bootstrap or sign test) on each benchmark would help; is the difference significant on any of them?
- What happens if the selection weight also includes a per-branch cost estimate, such as the parent’s token consumption or the reflector’s predicted cost?
- Can you report the average token cost per attempt for each branch depth or task, so that readers can judge whether high-value branches are also the expensive ones?