acceptodds
Under review as a conference paper at ICLR 2027

Beyond Scripted Search: Sample-Efficient Reward Discovery via Agentic Black-box Optimization

Abstract

Designing dense reward functions for low-level reinforcement learning (RL) control remains difficult. Recent work uses large language models (LLMs) to iteratively generate and refine reward functions using policy-training feedback within scripted search algorithms. However, evaluating each candidate requires a full RL training run, making sample efficiency a central challenge for reward search on complex control tasks. To address this limitation, we propose an Agentic Reward Black-box Optimization (ARBO) framework, in which an LLM agent builds the search strategy at run time from an evaluation history maintained as its persistent workspace. The evaluation history comprises two components: observations maintained by the evaluation oracle, including candidate scores, per-term training curves, and error tracebacks; and an agent-maintained belief that records diagnoses and intended next steps. The agent queries both with tools and generates the next batch of reward candidates, rather than generating them in a single pass from a fixed prompt. Across four control domains, ARBO achieves gains of 29.9% in manipulation success rate and 192.8% in power-grid score over baseline means under a shared evaluation budget. Ablations examine each component’s contribution and sensitivity to backbone choice.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Reward Design, Blackbox Optimization

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_p8NCOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors want to cut the number of full RL training runs needed when an LLM searches for reward functions. Their approach replaces the fixed outer loop of methods like Eureka with a coding agent. All past candidates, scores, reward-term curves and error logs sit as files in a workspace, and the agent reads them with bash and writes the next batch of reward code. A simple external schedule decides when to explore and when to exploit. The method is tested on 19 tasks across dexterous manipulation, power-grid operation, locomotion and traffic-signal control, against five LLM reward-search baselines with the same evaluation budget and backbone.

Critical strengths

The empirical work is solid. The evaluation covers four quite different domains, every method shares the same budget, backbone and training setup, and the best rewards are retrained on five fresh seeds, where ARBO's rewards mostly hold up. The appendix is thorough, with the prompts, configurations, per-candidate cost and code all provided. The codex comparison is also useful. A stock coding agent given the same task loses most of the gain on the power grid (−0.82) and GraspAndPlace (−0.64), so the improvement is not just from swapping in an agent.

Critical weaknesses
  1. The headline numbers overstate the gain. The 29.9% and 192.8% in the abstract are measured against the mean of all baselines, not the strongest one. Against the best baseline the gain is about 16% on Bi-DexHands (0.918 vs Eureka's 0.792) and about 28% on the power grid (52.5 vs ShinkaEvolve's 41.1). Figure 6d also shows the final result is slightly worse than the baseline mean on traffic (0.95×) and only 1.1× on locomotion. The clear wins are on manipulation and the grid, and the abstract should say so.

  2. The idea itself is simple, and the paper dresses it up more than it needs to. At its core, the method replaces Eureka's search loop with a coding agent that can read a log directory and keep a notes file. The "belief" that the paper compares to a Bayesian posterior is an analysis.md file the agent rewrites each round. Equations 2 to 11 mostly put symbols on a standard engineering loop rather than adding anything that is used later. This doesn't make the results wrong, but it limits what the paper teaches beyond "a capable agent with full logs beats a fixed prompt template."

  3. Some baseline and ablation results are hard to trust. R⋆ was reimplemented by the authors since no code exists, and it scores 0.000 on CatchAbreast with a flat curve in Figure 12, which looks like an implementation problem rather than a real result. The ablations cover only four tasks, and the paper does not say how many search runs each one uses.

Questions for authors

Even if the questions below are answered well, I would not raise my score to 4. The experiments are careful, but the contribution is an engineering recipe, and its conceptual framing claims more than the method delivers.

Can you confirm the R⋆ reimplementation is working on CatchAbreast, for example by checking it against any numbers reported in the original paper?

How many search runs does each ablation in Figure 7 use?

Can the abstract report the gain against the strongest baseline in each domain alongside the gain against the baseline mean?

5 comments

Since there was no response from the authors, my questions are still open. The main one is whether the R⋆ reimplementation works on CatchAbreast, where it scores 0.000 with a flat curve. If that baseline is broken, the comparison is weaker than it looks. It's also still unclear how many search runs each ablation uses, and the abstract still reports gains against the baseline mean rather than the strongest baseline. Without answers to these, I'm less confident in the results than when I first gave my score, so I'm lowering it to 2 for now. I'd be glad to revisit if the authors are able to address these points.

We highly appreciate the insightful suggestions to help us improve the paper. We apologize for the late response, since the new experiments took time to complete. Below we respond to each point.


W1 – Gain against the baseline mean

W1. The headline numbers overstate the gain. The 29.9% and 192.8% in the abstract are measured against the mean of all baselines, not the strongest one. Against the best baseline the gain is about 16% on Bi-DexHands (0.918 vs Eureka's 0.792) and about 28% on the power grid (52.5 vs ShinkaEvolve's 41.1). Figure 6d also shows the final result is slightly worse than the baseline mean on traffic (0.95×) and only 1.1× on locomotion. The clear wins are on manipulation and the grid, and the abstract should say so.

Thank you for pointing this out. The abstract now reports the gain against the strongest baseline, on manipulation and the power grid. On locomotion and traffic ARBO matches the baselines, and Section 1 now says so. We have made this change in the abstract and Section 1.


W2 – Simple idea and framing

W2. The idea itself is simple, and the paper dresses it up more than it needs to. At its core, the method replaces Eureka's search loop with a coding agent that can read a log directory and keep a notes file. The "belief" that the paper compares to a Bayesian posterior is an analysis.md file the agent rewrites each round. Equations 2 to 11 mostly put symbols on a standard engineering loop rather than adding anything that is used later. This doesn't make the results wrong, but it limits what the paper teaches beyond "a capable agent with full logs beats a fixed prompt template."

Thank you for the comment. ARBO is deliberately designed as a simple system. The contribution we intend to establish is more specific than replacing a fixed prompt with a coding agent. Instead, we aim to explore how to organize an agent’s access to accumulated experimental evidence when each new evaluation requires policy training. We have revised the paper on the points below:

  • What the agent replaces. To clarify, the agent replaces the hand-designed search algorithm. Existing methods fix both a selection rule and an expansion rule before the search, and ARBO lets the agent build both at every round. To improve clarity and accuracy, we have made this change in the abstract, Section 1, and Eq. 3.

  • Why a simple design. We agree the design is simple but not the problem. The problem we address is letting the agent itself decide which candidates to generate next, instead of following a scripted search. Handing the search to a common coding agent (Codex) does not solve it (Fig. 7). A concurrent work, Dream-RSI [1], solves a similar problem with a similar approach. It changes the search policy only after enough evidence has accumulated, and it aims to get past a plateau rather than to save samples. AgentPilot [2], released after our submission, shares this motivation and helps an agent choose a better next move from its past trials. ARBO pursues the same goal in reward discovery, where sample efficiency matters even more. RSI can improve efficiency in other ways, for example by locating failures in execution traces [3,4]. Apart from code errors, a reward's learning failure (not the errors of the code itself) can only be checked by running another full training. We have made this change in Section 1.

  • The Bayesian analogy and the equations. We agree the analogy needs a more careful statement. We only mean that the design is inspired by Bayesian optimization. A surrogate revises its fit after each evaluation, and our agent revises a free-text belief after each round. The equations are meant to put the five baselines and ARBO in one template (Eq. 3, Table 9, Eq. 4). We have made this change in the abstract and Section 1.


[1] Zheng et al. Dream-RSI: Recursive Self-Improvement through Evolving Worlds. arXiv:2609.14858, 2026.

[2] HKUDS. AgentPilot: Better Next Moves for Your Agent. GitHub repository, https://github.com/HKUDS/AgentPilot, released November 2026.

[3] Xia et al. RRSI: Regularized Recursive Self-Improvement of Agent Harnesses. arXiv:2609.24972, 2026.

[4] Lee et al. Recursive Harness Self-Improvement. arXiv:2607.15524, 2026.


Continued from our previous comment (2/2).


W3 – R* and ablation results

W3. Some baseline and ablation results are hard to trust. R⋆ was reimplemented by the authors since no code exists, and it scores 0.000 on CatchAbreast with a flat curve in Figure 12, which looks like an implementation problem rather than a real result. The ablations cover only four tasks, and the paper does not say how many search runs each one uses.

Thank you for the concern. We went through the per-candidate training records of the R* CatchAbreast run.

Table R1: Per-candidate training records of the R* run on CatchAbreast, 128 candidates.

StatisticValue
Candidates that compiled and trained to completion126 (2 failed at launch)
Candidates whose proposed reward increased during training126 / 126
Candidates with task success > 0 at any point in training0 / 126
Ground-truth reward, max over training (median across candidates)0.0036
Right-hand reach term at convergence (median)0.88
Lift term at convergence (median)0.24
Goal-distance term at convergence (median)0.055
Candidates containing the lift module127 / 128
Generations8

PPO optimized every reward R* wrote. The rewards are satisfied by lifting the object with the throwing hand, so the object never reaches the goal 0.6 m away in the other hand. The lift module is inherited through crossover in 127 of 128 candidates. The archive stays in this local optimum.

In the paper, we also retrained the reward R* selected with five training seeds, 3000 iterations each, under the search configuration.

Table R2: Best success rate over training of the reward R* selected on CatchAbreast.

Seed 0 (search)Seed 1Seed 2Seed 3Seed 4Seed 5
0.0000.0000.0000.0000.0000.000

The result does not depend on the training seed. The same reimplementation is the best method on SwingCup and CatchUnderarm (Table 1), so the 0.000 is specific to this task.

On the ablations, the four tasks are one per domain, GraspAndPlace, Case14, Cassie Flat, and Cologne8. We ran additional searches during the rebuttal, and each variant now uses three search runs (Q2). We have made this change in Fig. 7 and its caption.


Q1 – R* on CatchAbreast

Q1. Can you confirm the R⋆ reimplementation is working on CatchAbreast, for example by checking it against any numbers reported in the original paper?

Yes. 126 of 128 candidates trained to completion and PPO optimized every proposed reward (Table R1). The 0.000 is the score of R*'s rewards, not a pipeline failure. R* did not report Bi-DexHands and released no code, so there is no published number to compare.


Q2 – Runs per ablation

Q2. How many search runs does each ablation in Figure 7 use?

Thank you for the question. Each ablation variant uses three search runs per task. We have made this change in Fig. 7 and its caption.


Q3 – Gain against the strongest baseline

Q3. Can the abstract report the gain against the strongest baseline in each domain alongside the gain against the baseline mean?

Yes. We have made this change in the abstract.

Reviewer_rAVVOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes ARBO, an agentic black-box optimization framework for discovering reward functions for low-level RL control. Prior LLM-based methods such as Eureka/REvolve use a fixed search rule, i.e., just one rule decides which past candidates the LLM sees, and another decides how it edits them. ARBO instead has an LLM coding agent decide at each round what to look at and what to write next. The agent uses the full recorded evaluation history which has two parts- observation and a belief. The reported gains are shown to be +29.9% for manipulation success and +192.8% for power-grid score over the baseline mean.

Critical strengths

S1. Well-motivated problem. The core problem is well motivated. Each candidate reward costs a full RL training run, so sample efficiency matters a lot in reward search. Automatic reward design is also an active topic in RL post-training of LLMs; this paper targets low-level control and manipulation instead. It would be useful to discuss whether the two setups share dynamics, and whether they could (or couldn’t) benefit from the same design principles.

S2. A clear conceptual link to Bayesian optimization, plus evidence that fixed search rules fall short. The Bayesian-optimization analogy is a good conceptual connection: the belief acts as a revisable posterior over past observations, and the agent acts as an informal acquisition step. It makes the idea easier to follow. The analysis in Sec. 3 of where fixed search rules fall short is also appreciated.

S3. Candidate-quality analysis that separates proposal quality from final performance. Fig. 5 is an important analysis. It looks at the quality of every proposed candidate rather than only the best-so-far score. This isolates the effect of the proposal mechanism from downstream performance, and it directly supports the claim that ARBO spends fewer costly evaluations on low-quality candidates.

Critical weaknesses

W1. The trade-off between added LLM computation and saved evaluations isn’t discussed. An important missing discussion is the trade-off between the extra LLM computation needed to generate each new candidate at run time and the sample efficiency gained from evaluating fewer candidates. The paper measures search cost only by the number of evaluated candidates, on the grounds that policy training dominates every other cost.

W2. It’s unclear why the gains are largest on sparse-reward tasks and only appear with reasoning backbones. The paper’s explanation for why ARBO helps most on sparse-reward tasks is one sentence: unsuccessful candidates “require diagnosis beyond their evaluation scores.” There is no analysis to support this. It’s also unclear why the method only helps with backbones that support reasoning. Is it that the other models are not as good at understanding the textual feedback (curves, tracebacks), which limits the method’s usefulness?

Questions for authors
  1. (W1) Can you report wall-clock time or total compute per search, including the agent’s LLM inference, for ARBO and the baselines? Is there a per-candidate training cost below which ARBO’s advantage disappears?

  2. (W2) Does the backbone result reflect reasoning ability or overall capability? Can you add a strong non-reasoning model and run at least one baseline under the same alternative backbones?

  3. (S1) Do you expect the design principles (a full history the oracle owns, a belief the agent maintains, an external explore/exploit schedule) to carry over to reward or rubric design for LLM post-training, where evaluation is also expensive?

  4. How much do the empirical results vary across independent search runs (different seeds or LLM sampling)?

5 comments

We highly appreciate the insightful suggestions to help us improve the paper. We apologize for the late response, since the new experiments took time to complete. Below we respond to each point.


W1 – LLM computation vs. saved evaluations

W1. The trade-off between added LLM computation and saved evaluations isn’t discussed. An important missing discussion is the trade-off between the extra LLM computation needed to generate each new candidate at run time and the sample efficiency gained from evaluating fewer candidates. The paper measures search cost only by the number of evaluated candidates, on the grounds that policy training dominates every other cost.

Thank you for raising this. We agree the paper should have discussed it. ARBO spends more LLM computation per candidate, but this cost is small next to the policy training it saves. Table R1 reports the cost per candidate. It is the same table as in our response to Reviewer ouMy (W1 / Q1).

Table R1: Cost per candidate on Bi-DexHands. Policy training is the mean over the ten tasks (Table 6).

MethodPrefill tokensDecode tokensLLM time (s)Training (min)LLM share of cost
ARBO (ours)176,9614,99595602.6%
RF-Agent10,7839,99081602.2%
REvolve3,9403,48860601.6%
Eureka3,9193,51162601.7%
R*3,3916,132106602.9%
ShinkaEvolve19,2427,726134603.6%
  • The LLM time is minor. ARBO spends 45× Eureka's prefill tokens, but prefill is cheap in time. Its LLM time is 95 s against 62 s per candidate, and the LLM side stays below 4% of a candidate's cost for every method.
  • The saved evaluations outweigh the extra LLM time. ARBO at 64 candidates already exceeds every baseline at 128 (Fig. 4). This supports a reduction in candidate evaluations required to reach comparable average performance.
  • The gain does not come from more computation. w/o belief and retrieval (GA) spend more tokens than the full system and still lose. AutoResearch w/ Codex spends 1.9× the prefill and loses most of the grid gain (Fig. 7). The cost of every ablation setting is in our response to Reviewer ouMy (W3 / Q3).

We have made this change in Tables 7 and 8, Appendix B.


W2 – Sparse tasks and backbones

W2. It’s unclear why the gains are largest on sparse-reward tasks and only appear with reasoning backbones. The paper’s explanation for why ARBO helps most on sparse-reward tasks is one sentence: unsuccessful candidates “require diagnosis beyond their evaluation scores.” There is no analysis to support this. It’s also unclear why the method only helps with backbones that support reasoning. Is it that the other models are not as good at understanding the textual feedback (curves, tracebacks), which limits the method’s usefulness?

Thank you for pointing this out. We agree the paper should explain where the gain comes from. We explain both cases below.

  • Sparse tasks. A sparse task returns a coarse end-of-episode score, so most candidates score near the floor. A fixed rule selects by score. When the scores are all near the floor it has nothing to select on, and the search stalls. ARBO's agent revises its search strategy at every round, so it leaves the floor. The more candidates sit near the floor, the larger the gain (Fig. 8).
  • Backbones. The gain needs a model that can work as an agent, not a reasoning mode. The strong backbones read the environment code and the observations before writing a reward, and almost all of their candidates train. The weak backbones barely read anything, and few of their candidates train (Fig. 9). Qwen3-32B is a reasoning model and still fails. We also ran an additional experiment for Q2. With reasoning turned off, DeepSeek-V4-Flash still works as an agent and ARBO still outperforms Eureka. This experiment is reported in Q2 and Fig. 9c.

We have made this change in Section 5.5 and Fig. 9c.


Continued from our previous comment (2/2).


Q1 – Compute per search and break-even cost

Q1. (W1) Can you report wall-clock time or total compute per search, including the agent’s LLM inference, for ARBO and the baselines? Is there a per-candidate training cost below which ARBO’s advantage disappears?

Thank you for the question. Per 128-candidate search, the LLM side is 3.4 hours for ARBO against 2.2 hours for Eureka, while policy training is about 128 hours on average for either on Bi-DexHands. Policy training remains the main overhead of the search. We have made this change in Table 7, Appendix B.


Q2 – Reasoning or overall capability

Q2. (W2) Does the backbone result reflect reasoning ability or overall capability? Can you add a strong non-reasoning model and run at least one baseline under the same alternative backbones?

Thank you for the suggestion. DeepSeek-V4-Flash is already a Fig. 7c backbone with reasoning on. We ran ARBO and Eureka with reasoning on and off on GraspAndPlace, 128 candidates.

Table R2: Best success rate on GraspAndPlace with DeepSeek-V4-Flash, after the given number of candidates. Bold marks the best within each reasoning setting.

MethodReasoning163248648096112128
ARBOon (Fig. 7c)0.9510.9650.9650.9650.9700.9760.9760.976
Eurekaon0.1960.3800.4470.8000.8000.8000.8000.829
ARBOoff0.4200.4200.4860.4860.6730.6730.8280.828
Eurekaoff0.4330.4330.4920.4920.5640.5640.5640.564

ARBO stays ahead of Eureka under the same backbone, by 0.15 with reasoning on and by 0.26 with reasoning off. Reasoning lifts both methods, and ARBO's gain comes from the design, not the reasoning.

Table R3: Behaviour of ARBO with DeepSeek-V4-Flash on GraspAndPlace, per attempt.

Reasoning onReasoning off
Reads of environment code9.69.4
Reads of observations6.93.2
Candidates that train95%100%

Without reasoning the agent still reads environment code and every candidate trains, matching the strong backbones in W2. It consults past candidates about half as often. Our wording in the paper separated the backbones into reasoning and non-reasoning, which is not accurate. The result tracks overall capability, not the reasoning model. We have made this change in Sections 5.4 and 5.5 and Fig. 9c.


Q3 – Transfer to LLM post-training

Q3. (S1) Do you expect the design principles (a full history the oracle owns, a belief the agent maintains, an external explore/exploit schedule) to carry over to reward or rubric design for LLM post-training, where evaluation is also expensive?

Thank you for the question. We expect so. None of the three principles is specific to control. Agentic RL post-training is a harder case of the same problem. One reward candidate costs a full post-training run and the reward is usually a single outcome signal per trajectory, exactly the sparse and expensive regime where ARBO's gain is largest (W2). We have made this change in the limitations in Section 6.


Q4 – Variation across searches

Q4. How much do the empirical results vary across independent search runs (different seeds or LLM sampling)?

Thank you for asking. We repeated the complete search on the four ablation tasks, three times per variant, each with new LLM sampling (Fig. 7). Across the three runs, the standard error of each variant is 1–7% of ARBO's gain on Cologne8, 3–13% on GraspAndPlace, 5–16% on Cassie Flat and 7–24% on Case14. We have made this change in Fig. 7 and its caption.

Reviewer_rAVVOpenReview ↗

Response to rebuttal

Thanks for your responses and additional experiments in the short period of time. My major concerns are sufficiently addressed, and I acknowledge the paper's revisions with the new edits. I don't have any further concerns at the moment, and happy to raise my score.

Reviewer_ouMyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ARBO, an LLM agent with tools and a persistent workspace for searching reinforcement-learning reward code. The agent actively inspects candidate code, task scores, training curves, and errors, maintains analysis notes, and proposes new candidates; an external scheduler controls exploration and exploitation. Each candidate is evaluated by training a policy from scratch and measuring a separate task objective. Experiments compare ARBO with five reward-search methods across manipulation, power-grid operation, locomotion, and traffic control. With a shared LLM backbone and candidate budget, ARBO substantially improves manipulation and grid performance. History-access and scheduling ablations examine the contributions of its components.

Critical strengths
  1. The agent actively inspects candidate code, training curves, and errors rather than modifying only candidates selected by a fixed script. Observation-removal and fixed-retrieval ablations demonstrate that complete history and its access mechanism have practical effects on search performance.

  2. ARBO achieves a mean manipulation success rate of 0.918 versus Eureka’s 0.792, and a grid score of 52.5 versus ShinkaEvolve’s 41.1. Under a shared backbone and candidate budget, these results support more effective use of expensive reward evaluations.

Critical weaknesses
  1. ARBO uses approximately 45× Eureka’s prefill tokens and 6× its decode tokens per candidate. Matching candidate counts does not exclude benefits from additional LLM compute. Token-matched and end-to-end cost comparisons are missing, leaving practical efficiency unclear.

  2. Search uses policy-training seed 0, while seeds 1–5 only retrain selected rewards. This does not capture randomness in LLM sampling and complete search trajectories, leaving the reproducibility of rankings and search gains unestablished.

  3. Observation-removal, belief-removal, and fixed-retrieval ablations also change information availability, context, and tool use. Without compute matching, contributions from active diagnosis, additional history, and additional reasoning compute are difficult to separate.

  4. Exploration and exploitation remain manually scheduled, with five initial exploration rounds, a 20-round stagnation threshold, and up to five re-exploration rounds. Grid performance substantially depends on this schedule, but threshold sensitivity is insufficiently tested, leaving agentic search and scheduling contributions intertwined.

Questions for authors
  1. Does ARBO retain its advantage under matched LLM-token budgets or total cost? Can existing results quantify performance against combined LLM-inference and policy-training costs?

  2. Were complete searches independently repeated, including LLM sampling and different policy-training seeds? If so, how stable are the rankings and gains in Table 1 and Figure 6?

  3. Are token expenditures comparable across the history-access ablations in Section 5.4? Can existing results distinguish benefits of the access mechanism from additional context and reasoning compute?

4 comments

We sincerely thank the reviewer for the really careful review to help us improve the experiments. We apologize for the late response, since the new experiments took time to complete. Below we respond to each point.


W1, Q1 – Matched tokens or total cost

W1. ARBO uses approximately 45× Eureka’s prefill tokens and 6× its decode tokens per candidate. Matching candidate counts does not exclude benefits from additional LLM compute. Token-matched and end-to-end cost comparisons are missing, leaving practical efficiency unclear.

Q1. Does ARBO retain its advantage under matched LLM-token budgets or total cost? Can existing results quantify performance against combined LLM-inference and policy-training costs?

Thank you for raising this. The LLM cost is minor next to policy training, and under total cost ARBO reaches the strongest baseline's result at about half the cost. Extra LLM tokens alone do not produce the gain, as w/o belief, retrieval (GA) and AutoResearch w/ Codex each spend more tokens than ARBO and still lose to it (Table R2, W3, Q3). Table R1 reports the cost per candidate.

Table R1: Cost per candidate on Bi-DexHands. Policy training is the mean over the ten tasks (Table 6).

MethodPrefill tokensDecode tokensLLM time (s)Training (min)LLM share of cost
ARBO (ours)176,9614,99595602.6%
RF-Agent10,7839,99081602.2%
REvolve3,9403,48860601.6%
Eureka3,9193,51162601.7%
R*3,3916,132106602.9%
ShinkaEvolve19,2427,726134603.6%
  • The LLM time is minor. ARBO's prefill is 45× Eureka's, but prefill is cheap in time. Its LLM time is 95 s against 62 s, and the LLM side stays below 4% of a candidate's cost for every method.
  • Under total cost ARBO needs about half the candidates. ARBO at 64 candidates already exceeds every baseline at 128 (Fig. 4). This supports a reduction in candidate evaluations required to reach comparable average performance.
  • ARBO's decode tokens were counted repeatedly in the submission. Our submitted Table 7 charged the decode of a full generation to every agent call, which produced the 6× figure. ARBO's decode is 1.4× Eureka's. Its extra cost is prefill, since every agent call re-sends its history.

We have made this change in Table 7, Appendix B, and AI use statement.


W2, Q2 – Repeated searches

W2. Search uses policy-training seed 0, while seeds 1–5 only retrain selected rewards. This does not capture randomness in LLM sampling and complete search trajectories, leaving the reproducibility of rankings and search gains unestablished.

Q2. Were complete searches independently repeated, including LLM sampling and different policy-training seeds? If so, how stable are the rankings and gains in Table 1 and Figure 6?

Thank you for the question. We repeated the complete search on the four ablation tasks, three times per variant, each with new LLM sampling (Fig. 7). Across the three runs, the standard error of each variant is 1–7% of ARBO's gain on Cologne8, 3–13% on GraspAndPlace, 5–16% on Cassie Flat and 7–24% on Case14. We did not repeat the main comparison. Each of the 19 tasks is an independent search with its own LLM sampling. On Bi-DexHands, ARBO ranks first on 7 of the 10 tasks and second on the other 3 (Table 1). Repeating every method on every task three times would take over 8 months on our node. We have made this change in Fig. 7 and its caption.

We would like to kindly solicit the reviewer's understanding that repeating every experiment many times is beyond our compute budget. We chose instead to spend our budget on breadth to support our claim.


Continued from our previous comment (2/2).


W3, Q3 – Compute across ablations

W3. Observation-removal, belief-removal, and fixed-retrieval ablations also change information availability, context, and tool use. Without compute matching, contributions from active diagnosis, additional history, and additional reasoning compute are difficult to separate.

Q3. Are token expenditures comparable across the history-access ablations in Section 5.4? Can existing results distinguish benefits of the access mechanism from additional context and reasoning compute?

Thank you for the comment. The retrieval ablations separate the access mechanism from extra context and compute. retrieval (embed) and retrieval (GA) draw on the same full history and spend about the same tokens as ARBO (Table R2). They differ only in that a fixed rule, not the agent, chooses what to read, and they still limit the gain (Fig. 7a). Every method and every ablation receives the same oracle output per candidate (score, per-term training curves, traceback, Section 4). We have made this change in Section 5.2.

On compute, Table R2 reports the cost per candidate of every ablation setting.

Table R2: Cost per candidate of each ablation setting, averaged over the four ablation tasks.

SettingPrefill tokensDecode tokensLLM time (s)
ARBO (full)124,1472,78249
w/o belief151,2592,86951
w/o observations94,0832,93352
retrieval (embed)119,6812,96352
retrieval (GA)133,3823,14755
w/o explore112,9843,22258
w/o exploit124,6482,77649
AutoResearch w/ Codex237,7861,58027
  • Tokens are comparable across the history-access settings. Their prefill ranges from 94k to 151k tokens per candidate and their LLM time from 49 to 55 s.
  • The gain comes from the design, not from more computation. w/o belief and retrieval (GA) spend more tokens than the full system and still lose. AutoResearch w/ Codex spends 1.9× the prefill and loses most of the grid gain (Fig. 7).

We have made this change in Table 8, Appendix B.


W4 – Manual schedule

W4. Exploration and exploitation remain manually scheduled, with five initial exploration rounds, a 20-round stagnation threshold, and up to five re-exploration rounds. Grid performance substantially depends on this schedule, but threshold sensitivity is insufficiently tested, leaving agentic search and scheduling contributions intertwined.

Thank you for the comment. We agree the current schedule design is a manual setting. To separate the schedule from the agent, we tested it at its two extremes, w/o explore and w/o exploit, on all four ablation tasks with three complete searches each (Fig. 7b). Both variants keep the agent and change only the schedule, and both lose part of the gain on the harder tasks. We also ran AutoResearch w/ Codex, which removes the framework and keeps only an agent, and it loses most of the gain on the grid. We state the manual schedule and its untested sensitivity as a limitation, and leave internalizing the schedule into the LLM agent to future work. We have made this change in the limitations in Section 6.

Thank you for the authors' detailed response, which has addressed most of my concerns. Although the contribution is incremental, overall I consider this paper to be a solid piece of engineering work. I have thus raised my score to 3.

Thank you very much for your careful review to help us improve the paper. In the remaining time, we will do our best to conduct more searches on a subset of the main-comparison tasks for W2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.