acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking LLM Agents on Pushing the Test-Time Reward-Cost Frontier

Abstract

LLM agents can now autonomously form hypotheses, modify code, run experiments, and interpret results—a workflow commonly known as autoresearch. A prominent goal of this paradigm is to use LLM agents to improve LLMs themselves. Existing autoresearch testbeds typically pursue this goal through model training. This makes each research iteration slow and expensive, leaving agents with limited opportunities to develop methods that are practical and transferable to larger-scale settings within a fixed time budget. Moreover, because agents are often given broad control over data, training pipelines, and engineering choices, final performance may depend as much on auxiliary capabilities—such as finding benchmark-relevant training data—as on the ability to conduct research. We instead introduce a controlled benchmark for autoresearch at test time. It asks an agent to push the reward–cost frontier of a frozen LLM on agentic tasks solely by reorganising its inference-time computation: achieving a higher reward at the same cost or the same reward at a lower cost. This setting enables faster research iterations, and any method discovered can be applied directly to already-trained models without additional training. We build a controlled evaluation on -bench, Terminal-Bench 2.1, and SWE-bench Verified, using a small development set for autoresearch and a separate held-out set for evaluation. During autoresearch, coding agents may change only how the given LLM is used at inference time. We find that (1) frontier agents cannot yet push the reward–cost frontier of their own backbone models, even when they appear to do so on the development set; (2) different agents converge to similar methods; and (3) agents retain the broad research direction they choose at the outset, even when given access to extensive relevant literature.

Primary area:
datasets and benchmarks
Keywords:
Autoresearch, LLM Agents, Test-Time Scaling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qbfAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a benchmark in which an autoresearch agent may modify only the inference policy of a frozen LLM, with the goal of improving the reward-cost frontier on tau^2-bench, Terminal-Bench 2.1, and SWE-bench Verified. With a 50-task development set and a 12-hour budget, three frontier agents optimizing their own backbones obtain development gains of up to 10.4 points that shrink to at most 2.7 points on 817 held-out tasks, and all three center their search on risk-gated sampling chosen early in the session. Controlled variations show that a target with more headroom (Qwen3-Coder-Next) allows a 10-19% token reduction at matched reward, while a 24-hour budget or access to 200 papers does not improve the held-out frontier.

Critical strengths
  1. The problem of agentic test-time scaling matters both for reducing cost and for raising the achievable performance of a fixed model. Restricting the agent to the inference policy of a frozen model gives fast iterations and allows gains to be attributed to the method itself.

  2. The evaluation separates a small development set from a large held-out set and reports the negative results directly. The finding that development gains do not transfer to held-out tasks is useful for the autoresearch community, and this study is costly for academic groups to run.

Critical weaknesses
  1. It is unclear that why the development set has only 50 tasks, since the variance at this size is clearly very large.

  2. The iterating on sampling strategies is not reasonable enough. It would overfit very easily, so the resulting methods become task-specific.

  3. The three benchmarks are too saturated, with scores generally in the 80-90 range, so the authors can only focus on efficiency. A harder, more reasoning-heavy benchmark might give more signal.

Questions for authors
  1. Why was the development set limited to 50 tasks, and was this driven by cost? Do you have an estimate of how the dev-to-held-out gap changes with a larger development set?

  2. How much of the dev-to-held-out gap is explained by measurement noise alone, compared with task-specific fitting by the agents?

  3. Have you considered harder, more reasoning-heavy benchmarks where reward has more headroom? The Qwen3-Coder-Next result suggests that headroom matters.

1 comment

Thank you for your thoughtful feedback. We address your concerns as follows.

Weakness 1 and Question 1: The development set size follows from the 12-hour budget: each iteration evaluating the method on all three benchmarks takes 1-1.5 hours of wall-clock time, which leaves room for roughly 8-12 iterations in a session. We rerun two gpt-5.5 autoresearch sessions by doubling the size of the dev set to 100 problems, and the held-out set problem is therefore decreased by 50 problems. We report the dev set and held-out set results with standard deviation in the following table:

MethodDev Reward (%)Dev Output tokensHeld-out Reward (%)Held-out Output tokens
Single sample83.1 ± 3.813,586 ± 1,07682.4 ± 1.513,422 ± 1,132 ×1.00
Best-of-n, trajectory level, n=285.628,82384.628,244 ×2.10
Best-of-n, trajectory level, n=485.657,32685.755,912 ×4.17
Autoresearch frontier84.8 ± 1.619,345 ± 1,94382.8 ± 1.223,840 ± 3,895 ×1.78

The results show that even with a larger dev set, the dev set +1.7% gain over the single sample still shrinks to 0.4% than simple baselines, while best-of-n’s gain is consistent.

Weakness 2: Optimizing inference-time policies is well established, including sampling and aggregation [1, 2], iterative refinement [3, 4], and automated agent/workflow optimization [5, 6]. We intentionally fix the model and harness to isolate improvements from inference-policy design. This also represents a practical deployment setting in which developers access a fixed model through an API, including locally hosted endpoints, and optimize how model calls are organized. To assess task-specific overfitting, we evaluated the delivered policies on separate held-out tasks across multiple benchmarks and reported the development-to-held-out gap.

Question 2: The delivered gpt-5.5 and opus-5 methods contain no explicit task-specific hints. The gpt-5.6-sol method modifies its Terminal-Bench prompts. By removing the task-specific optimization, its dev set gain dropped by 2.6%, standard deviations observed in the held-out set is around 1.3%, while the dev-held-out gap is 7.7%. Therefore, these results support our finding that development gains do not fully transfer to held-out tasks.

Weakness 3 and Question 3: We further conduct an autoresearch session using gpt-5.6-sol working on swe-rebench, which is a harder benchmark. We split it into a dev set (30 problems) and a held-out set (81 problems), each problem we sample 5 times.

MethodDev Reward (%)Dev Output tokensHeld-out Reward (%)Held-out Output tokens
Single sample53.3 ± 3.816,923 ± 300 ×1.0052.6 ± 3.118,732 ± 482 ×1.00
Best-of-n, trajectory level, n=256.734,469 ×2.0453.738,063 ×2.03
Best-of-n, trajectory level, n=456.768,934 ×4.0758.075,964 ×4.06
Autoresearch frontier55.3 ± 4.119,749 ± 610 ×1.1751.4 ± 1.222,606 ± 943 ×1.21

The agent improves the single sample on dev set by 2% with 1.17x cost, while this improvement shrinks to -1.2% with 1.21x cost. This shows that development gains can also fail to transfer in a setting with substantially lower baseline reward, which is consistent with our main findings.

Thank you again for your thoughtful feedback and we would be happy to discuss any further questions.

References

[1] Wang et al. Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.

[2] Aggarwal et al. Let’s Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs. EMNLP 2023.

[3] Madaan et al. Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS 2023.

[4] Shinn et al. Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS 2023.

[5] Hu et al. Automated Design of Agentic Systems. ICLR 2025.

[6] Zhang et al. AFlow: Automating Agentic Workflow Generation. ICLR 2025.

Reviewer_TjBMOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies the ability of LLM agents to optimize how test-time computation of agents is organized. By running controlled evaluations of 3 frontier models (GPT 5.5, GPT 5.6 Sol, Claude Opus 5) on 3 benchmarks (τ2\tau^2-bench, Terminal-Bench 2.1, and SWE-Bench Verified), the paper demonstrates that, while agents are able to optimize target rewards and output tokens for their own backbone on a restricted development set, these gains rarely transfer to held-out settings. Qualitative analysis of the optimizer trajectories additionally shows that agents often converge on similar methods (e.g. risk-gated sampling) and keep directions they choose early on in their trajectories. The authors run further experiments that show (a) that agents can optimize the test-time strategy for one other weaker model, (b) that longer optimization budgets do not reliably translate into better outcomes, and (c) that providing optimizers with additional context on test-time scaling strategies does not help them succeed.

Critical strengths
  • Interesting take on a timely topic. The authors study a restricted form of harness optimization, where interventions are limited to how a model inference policy is configured. While several papers have studied more general forms of harness optimization, this restriction is novel and interesting.

  • Experimental breadth. The authors attempt to address several questions across models and benchmarks, including both baseline optimization feasibility experiments, as well as more targeted ablations for optimization budget, context, and target model.

Critical weaknesses

Major

  • Evaluation noise. The authors only repeat inner target agent evaluations for τ\tau-Bench. These evaluations are stochastic and as the authors note on L201 pg. 4, single sample run accuracies can vary significantly. This is likely true of output token counts as well. It is unclear whether the reported gains from the baselines in Table 3 are significant or within noise; lack of significance may indicate lack of headroom rather than lack of optimizer ability. Reported gains or losses in token counts on both the development and held-out splits (Table 3 and Table 6) are hard to justify without any repeats. Additionally, each outer optimization run is conducted only twice: what is the range of outcomes for each optimizer configuration and how representative are the reported averages?

  • Experimental design. In section 4.3, the authors test whether providing optimizer agents with access to 200 papers on test-time sampling techniques can improve their ability to optimize targets. Could the authors show that at least one of these techniques can indeed improve the target? It is unclear whether the result should be interpreted as (1) the task has no optimization headroom, (2) the papers are insufficient, or (3) agents are simply not good optimizers, even in the presence of conditioning information.

  • Baseline implementation of best of NN. I have doubts about the quality of the best-of-NN baselines. Typically best of NN sampling is conducted on entire rollouts, where an LLM-based selector can take into account the final state of the rollout. This can be used to select a majority answer or "best" answer as deemed by an LLM judge. Best of NN on partial sequences, which is what the authors implement, seems peculiar and less likely to work. In Table 3, gains from increasing NN are non-monotonic in some cases; while this could be due to noise, I am not convinced that these baselines are fair.

  • Choice of inference policy restriction. My understanding is that the inference policy only determines how each turn in a multi-turn agent rollout is generated. This is a very specific way of organizing test-time computation. Alternatives could involve sampling entire trajectories or multiple turns at a time.

    • Could the authors first justify why they use this particular choice versus whole-trajectory sampling?
    • The authors mention that "Best-of-n samples n candidate messages at each turn and asks the target model to select one." What signals does the target model have to select messages from using only partial rollouts? Could the authors provide examples to ground this choice?
  • Framing against related work. One concern I have about this paper is that it almost entirely ignores recent related work on automated harness optimization. Works such as ADAS (mentione), AFlow, GEPA, VeRO, MetaHarness, and ACE all assume that the underlying LLM is frozen and instead optimize (possibly some restricted form of) the program that invokes it for test-time tasks. An auto-research pipeline that optimizes "how [an agent's] test-time computation is organised" is a subclass of more general harness optimization and can be compared against these other choices. Morever, several of these works discuss overfitting in the context of harness optimization; the paper's finding that development set gains rarely transfer to a held-out set could perhaps be mitigated by simply introducing a validation set, as some of the above papers do.

Minor

  • Framing of training vs. non-training approaches. The authors claim that auto-research applied to model training is expensive and use this to motivate auto-research for test-time computation. While this may be true of certain cases, I would argue that inference-time computation can be as expensive depending on the setting. As such, cost is not a strong a priori motivation for avoiding training.
Questions for authors
  1. What are the results for the baselines on the held out set reported in Table 4? Are any gains seen for best-of-N?
  2. What methods other than risk-gated sampling and a model-based judge do agents attempt?
  3. I am concerned that the task as presented has little to no optimization headroom (scores already in the 80-90 range and marginal improvement from more test-time computation). Is it possible to show that there exists some method other than best-of-N that leads to improvements over the baseline?
2 comments
AuthorsOpenReview ↗

Author Response 1/2

Thank you for your thoughtful feedback. We address your concerns as follows.

Weakness 1 and Question 1: Evaluation noise and held-out set results.

We have added a turn-level best-of-n baseline for Table 4 using gpt-5.5 and gpt-5.6-sol. Also we provide a trajectory-level best-of-n baseline and sample 5 more times on the held-out set to give a better noise estimation (we report standard deviation).

MethodGPT-5.5 Reward (%)GPT-5.5 Output tokensGPT-5.6-sol Reward (%)GPT-5.6-sol Output tokens
Single sample82.4 ± 1.513,143 ± 987 (1.00×)83.9 ± 1.16,349 ± 235 (1.00×)
Autoresearch frontier80.1 ± 1.210,617 ± 445 (0.81×)85.4 ± 1.310,358 ± 296 (1.63×)

Weakness 2: Experimental design

We present the 200 papers to the agent, however we found that agents do not extensively try the provided paper methods, instead agents either tend to summarize an overall high-level idea from the given papers and propose its own idea, which is usually a conservative, risk-aware-like idea or try one or two methods in the beginning and then stop further exploring the given papers in its subsequent experiment period. In the observed sessions, access to the papers led to limited implementation and evaluation of additional methods and did not improve held-out performance.

Weakness 3 and Question 1: Implementation of best of n and its results

Our original turn-level best-of-n baseline follows the per-turn interface used by the autoresearch agents. We have therefore added trajectory-level best-of-n and report both variants on the held-out set below.

MethodGPT-5.5 Reward (%)GPT-5.5 Output tokensGPT-5.6-sol Reward (%)GPT-5.6-sol Output tokens
Single sample82.4 ± 1.513,143 ± 987 (1.00×)83.9 ± 1.16,349 ± 235 (1.00×)
Turn-level best-of-n, n=283.325,319 (1.93×)83.313,017 (2.05×)
Turn-level best-of-n, n=482.744,644 (3.40×)85.525,084 (3.95×)
Trajectory-level best-of-n, n=284.327,713 (2.11×)87.412,901 (2.03×)
Trajectory-level best-of-n, n=485.354,880 (4.18×)89.225,788 (4.06×)
Autoresearch frontier80.1 ± 1.210,617 ± 445 (0.81×)85.4 ± 1.310,358 ± 296 (1.63×)

Trajectory-level best-of-n improves held-out reward for both models: best-of-2 gains 1.9 and 3.5 percentage points, and best-of-4 gains 2.9 and 5.3 points, for GPT-5.5 and GPT-5.6-sol, respectively. It also achieves higher reward than the corresponding turn-level baseline at similar token costs. These results provide a stronger reference comparison.

Weakness 4: Choice of inference policy restriction.

Thank you for raising this point. We chose the per-turn interface to enable fine-grained control over the reward–cost tradeoff: a policy can adapt its computation at each turn, generate multiple candidate outputs, and revise or select among them before returning an action. We have also added trajectory-level best-of-n results to provide a broader baseline comparison.

Weakness 5: Framing against related work.

Thank you for these suggestions. Our benchmark studies a restricted form of harness optimization: only the per-turn inference policy is editable, while the model, benchmark prompts, tools, and environment remain fixed. This setting reflects applications where users access a hosted model without control over its weights or serving infrastructure. We will expand the related-work discussion to include ADAS, AFlow, GEPA, VeRO, MetaHarness, and ACE, and clarify our focus on an inference-policy optimization. Regarding overfitting issues, agents can utilize the dev set provided for research and validation subsets and determine how to select generalizable methods. We leave this choice to the agent to evaluate its experimental methodology, and we finally evaluate the agent's delivered policies on held-out tasks to assess generalization beyond the agent’s reported development gains.

AuthorsOpenReview ↗

Author Response 2/2

Question 2: We have summarized all the methods explored by the agents in the table below, grouped by high-level idea:

IdeaMethodDescription
When to spend extra computeTurn-type routingDifferent sample counts for dialogue, structured-tool and shell turns
Consequence gatesExtra samples only for irreversible or episode-ending actions, or after a first proposal with a tool call
Schedule gatesExtra samples only within or after a fixed range of turns
Adaptive early stoppingSample in growing batches (1, 2, 4, ...) and stop once they agree
Repair-triggered resamplingResample only empty, malformed, truncated or repeated failed proposals
Choosing without a judgeAgreement votingExact or structural match of proposed actions
Medoid selectionThe candidate most similar to the others
Strict-majority vetoRequired before ending an episode
Validity-based electionPrefer well-formed, valid candidates
Confidence tie-breakMean token log-probability (local target only)
Refining one candidateSelf-critique and revisionKeep-or-revise critique, then regenerate
Reply auditCheck user-facing replies for omitted facts or totals
Length and budget controlOutput-token capsPer benchmark, uncapped retry on truncation
Context-length fallbackShorter caps after context-length errors
Episode token budgetsBrevity instruction past a threshold; forced finish at a cap or on repetition
Generic guidanceOperating checklistse.g. verify outputs before declaring completion
Per-turn conversation signalsStall detectionRepeated failed commands and stalls, recomputed from the supplied history

Overall, the dominant pattern is a combination of (1) and (2): sample more on turns judged risky, then aggregate by agreement. No session tried within-turn lookahead (simulating future steps before acting) or compressed or summarised the supplied conversation, although both are permitted by the interface.

Question 3: We have shown above that trajectory best-of-n serves as a strong frontier in performance-cost trade-off. Regarding the problem of limited headroom, we further conduct an autoresearch session using gpt-5.6-sol working on swe-rebench, which is a harder benchmark. We split it into a dev set (30 problems) and a held-out set (81 problems), each problem we sample 5 times.

MethodDev Reward (%)Dev Output tokensHeld-out Reward (%)Held-out Output tokens
Single sample53.3 ± 3.816,923 ± 300 ×1.0052.6 ± 3.118,732 ± 482 ×1.00
Best-of-n, trajectory level, n=256.734,469 ×2.0453.738,063 ×2.03
Best-of-n, trajectory level, n=456.768,934 ×4.0758.075,964 ×4.06
Autoresearch frontier55.3 ± 4.119,749 ± 610 ×1.1751.4 ± 1.222,606 ± 943 ×1.21

The agent improves the single sample on dev set by 2% with 1.17x cost, while this improvement shrinks to -1.2% with 1.21x cost. This shows that development gains can also fail to transfer in a setting with substantially lower baseline reward, which is consistent with our main findings.

Minor: Framing of training vs. non-training approaches.

Our motivation concerns users for whom training or updating a large model is impractical, particularly when access is limited to a hosted API. We will further emphasize that our benchmark studies training-free optimization of inference policies with fixed model weights.

If our additional results and clarifications address your concerns, we would be grateful if you would consider raising your score. Thank you for your time and effort!

Reviewer_Z1iKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper benchmarks coding agents that optimize the inference policy of a frozen model on multi-turn tasks. Agents develop policies on 50 tasks before evaluation on 817 held-out tasks. The experiments examine whether reward and token cost gains transfer, and how research time and literature access affect the search.

Critical strengths

The held-out evaluation shows how much the apparent research gains shrink on new tasks. The per-turn interface provides a controlled setting that differs from the broader harness edits in Meta-Harness and offline replay in AutoTTS [1, 2]. Appendix C is particularly useful: a selected Terminal-Bench score falls from 100 to 86.7 percent after pooling 90 episodes, and a same-day control leaves an aggregate gain of only 0.6 points.

[1] Meta-Harness: End-to-End Optimization of Model Harnesses. https://arxiv.org/abs/2603.28052v1 [2] LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling. https://arxiv.org/abs/2605.08083v2

Critical weaknesses
  1. The main conclusion needs a held-out comparison with the stated baselines. Section 2.1 defines the reference frontier using single-sample decoding and best-of-n, but Table 4 and Appendix D report only single-sample baselines. GPT-5.6-sol raises reward from 78.6 to 81.3 percent while using 81% more tokens. GPT-5.5 uses fewer tokens but also earns less reward. Both tradeoffs could be useful at different budgets. The results show that the development gains shrink substantially, but they do not establish whether these policies improve on best-of-n at comparable cost.

  2. The interpretation of limited exploration needs to account for the interface. Section 3.3 points to the absence of trajectory-level search and context management. Appendix A forbids executing tools inside the policy and carrying private memory across external turns, so the policy cannot try alternative sequences of real tool interactions before choosing its next action. Branching over model outputs and processing the supplied conversation remain possible. The discussion should distinguish methods excluded by these rules from permitted alternatives that the agents failed to explore.

Questions for authors
  1. How do the delivered policies compare with held-out best-of-n at similar token costs, and what criterion determines whether the frontier improved?
  2. Does trajectory-level search in Section 3 mean hypothetical continuations within a turn, or branches containing real tool executions?
1 comment

Thank you for your thoughtful feedback. We address your concerns below.

Weakness 1 and Question 1: We have added held-out best-of-n baselines for GPT-5.5 and GPT-5.6-sol to Table 4, including both turn-level and trajectory-level variants. We also sample 5 more times on the held-out sets to better estimate evaluation noise (we report standard deviation).

MethodGPT-5.5 Reward (%)GPT-5.5 Output tokensGPT-5.6-sol Reward (%)GPT-5.6-sol Output tokens
Single sample82.4 ± 1.513,143 ± 987 (1.00×)83.9 ± 1.16,349 ± 235 (1.00×)
Turn-level best-of-n, n=283.325,319 (1.93×)83.313,017 (2.05×)
Turn-level best-of-n, n=482.744,644 (3.40×)85.525,084 (3.95×)
Trajectory-level best-of-n, n=284.327,713 (2.11×)87.412,901 (2.03×)
Trajectory-level best-of-n, n=485.354,880 (4.18×)89.225,788 (4.06×)
Autoresearch frontier80.1 ± 1.210,617 ± 445 (0.81×)85.4 ± 1.310,358 ± 296 (1.63×)

GPT-5.5 reduces both reward and token cost. GPT-5.6-sol achieves a more favorable reward outcome, improving reward by 1.5 percentage points at 1.63× cost, while trajectory-level best-of-2 improves reward by 3.5 percentage points at 2.03× cost.

We use “autoresearch frontier” to refer to the final policies delivered by the autoresearch agent. Following prior work [1, 2], a policy is non-dominated if no other policy achieves at least the same reward at no greater cost, with a strict improvement in either dimension. In practice, matching costs precisely is difficult because generation lengths are stochastic [3, 4]. We therefore assess frontier improvement through higher reward at no greater cost, or lower cost without reduced reward, consistent with [5]. The delivered policies remain non-dominated and their development-set gains substantially weaken on held-out tasks.

Weakness 2 and Question 2: Thank you for highlighting this distinction. Branching over model outputs, context management, and processing the supplied conversation within a turn are permitted. All agents explored branching over candidate outputs, but none explored context management or within-turn branching over hypothetical continuations. These are the permitted alternatives we intended to discuss. Trajectory-level search involving real tool executions is excluded because tool calls change the environment state, which the policy may not be able to restore reliably. Such search also requires control over the harness beyond the inference-policy interface. We have revised Section 3.3 in the updated manuscript to make this distinction explicit.

If our additional results and clarifications address your concerns, we would be grateful if you would consider raising your score. Thank you for your time and effort!

References

[1] Wu et al. Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving. ICLR 2025.

[2] Saad-Falcon et al. An Architecture Search Framework for Inference-Time Techniques. ICML 2025.

[3] Wu et al. ARM: Adaptive Reasoning Model. NeurIPS 2025.

[4] Zeng et al. Pushing Test-Time Scaling Limits of Deep Search with Asymmetric Verification. ICLR 2026.

[5] Manvi et al. Zero-Overhead Introspection for Adaptive Test-Time Compute. ICLR 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.