- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a benchmark in which an autoresearch agent may modify only the inference policy of a frozen LLM, with the goal of improving the reward-cost frontier on tau^2-bench, Terminal-Bench 2.1, and SWE-bench Verified. With a 50-task development set and a 12-hour budget, three frontier agents optimizing their own backbones obtain development gains of up to 10.4 points that shrink to at most 2.7 points on 817 held-out tasks, and all three center their search on risk-gated sampling chosen early in the session. Controlled variations show that a target with more headroom (Qwen3-Coder-Next) allows a 10-19% token reduction at matched reward, while a 24-hour budget or access to 200 papers does not improve the held-out frontier.
Critical strengths
-
The problem of agentic test-time scaling matters both for reducing cost and for raising the achievable performance of a fixed model. Restricting the agent to the inference policy of a frozen model gives fast iterations and allows gains to be attributed to the method itself.
-
The evaluation separates a small development set from a large held-out set and reports the negative results directly. The finding that development gains do not transfer to held-out tasks is useful for the autoresearch community, and this study is costly for academic groups to run.
Critical weaknesses
-
It is unclear that why the development set has only 50 tasks, since the variance at this size is clearly very large.
-
The iterating on sampling strategies is not reasonable enough. It would overfit very easily, so the resulting methods become task-specific.
-
The three benchmarks are too saturated, with scores generally in the 80-90 range, so the authors can only focus on efficiency. A harder, more reasoning-heavy benchmark might give more signal.
Questions for authors
-
Why was the development set limited to 50 tasks, and was this driven by cost? Do you have an estimate of how the dev-to-held-out gap changes with a larger development set?
-
How much of the dev-to-held-out gap is explained by measurement noise alone, compared with task-specific fitting by the agents?
-
Have you considered harder, more reasoning-heavy benchmarks where reward has more headroom? The Qwen3-Coder-Next result suggests that headroom matters.
Thank you for your thoughtful feedback. We address your concerns as follows.
Weakness 1 and Question 1: The development set size follows from the 12-hour budget: each iteration evaluating the method on all three benchmarks takes 1-1.5 hours of wall-clock time, which leaves room for roughly 8-12 iterations in a session. We rerun two gpt-5.5 autoresearch sessions by doubling the size of the dev set to 100 problems, and the held-out set problem is therefore decreased by 50 problems. We report the dev set and held-out set results with standard deviation in the following table:
The results show that even with a larger dev set, the dev set +1.7% gain over the single sample still shrinks to 0.4% than simple baselines, while best-of-n’s gain is consistent.
Weakness 2: Optimizing inference-time policies is well established, including sampling and aggregation [1, 2], iterative refinement [3, 4], and automated agent/workflow optimization [5, 6]. We intentionally fix the model and harness to isolate improvements from inference-policy design. This also represents a practical deployment setting in which developers access a fixed model through an API, including locally hosted endpoints, and optimize how model calls are organized. To assess task-specific overfitting, we evaluated the delivered policies on separate held-out tasks across multiple benchmarks and reported the development-to-held-out gap.
Question 2: The delivered gpt-5.5 and opus-5 methods contain no explicit task-specific hints. The gpt-5.6-sol method modifies its Terminal-Bench prompts. By removing the task-specific optimization, its dev set gain dropped by 2.6%, standard deviations observed in the held-out set is around 1.3%, while the dev-held-out gap is 7.7%. Therefore, these results support our finding that development gains do not fully transfer to held-out tasks.
Weakness 3 and Question 3: We further conduct an autoresearch session using gpt-5.6-sol working on swe-rebench, which is a harder benchmark. We split it into a dev set (30 problems) and a held-out set (81 problems), each problem we sample 5 times.
The agent improves the single sample on dev set by 2% with 1.17x cost, while this improvement shrinks to -1.2% with 1.21x cost. This shows that development gains can also fail to transfer in a setting with substantially lower baseline reward, which is consistent with our main findings.
Thank you again for your thoughtful feedback and we would be happy to discuss any further questions.
References
[1] Wang et al. Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.
[2] Aggarwal et al. Let’s Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs. EMNLP 2023.
[3] Madaan et al. Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS 2023.
[4] Shinn et al. Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS 2023.
[5] Hu et al. Automated Design of Agentic Systems. ICLR 2025.
[6] Zhang et al. AFlow: Automating Agentic Workflow Generation. ICLR 2025.