- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors want to cut the number of full RL training runs needed when an LLM searches for reward functions. Their approach replaces the fixed outer loop of methods like Eureka with a coding agent. All past candidates, scores, reward-term curves and error logs sit as files in a workspace, and the agent reads them with bash and writes the next batch of reward code. A simple external schedule decides when to explore and when to exploit. The method is tested on 19 tasks across dexterous manipulation, power-grid operation, locomotion and traffic-signal control, against five LLM reward-search baselines with the same evaluation budget and backbone.
Critical strengths
The empirical work is solid. The evaluation covers four quite different domains, every method shares the same budget, backbone and training setup, and the best rewards are retrained on five fresh seeds, where ARBO's rewards mostly hold up. The appendix is thorough, with the prompts, configurations, per-candidate cost and code all provided. The codex comparison is also useful. A stock coding agent given the same task loses most of the gain on the power grid (−0.82) and GraspAndPlace (−0.64), so the improvement is not just from swapping in an agent.
Critical weaknesses
-
The headline numbers overstate the gain. The 29.9% and 192.8% in the abstract are measured against the mean of all baselines, not the strongest one. Against the best baseline the gain is about 16% on Bi-DexHands (0.918 vs Eureka's 0.792) and about 28% on the power grid (52.5 vs ShinkaEvolve's 41.1). Figure 6d also shows the final result is slightly worse than the baseline mean on traffic (0.95×) and only 1.1× on locomotion. The clear wins are on manipulation and the grid, and the abstract should say so.
-
The idea itself is simple, and the paper dresses it up more than it needs to. At its core, the method replaces Eureka's search loop with a coding agent that can read a log directory and keep a notes file. The "belief" that the paper compares to a Bayesian posterior is an analysis.md file the agent rewrites each round. Equations 2 to 11 mostly put symbols on a standard engineering loop rather than adding anything that is used later. This doesn't make the results wrong, but it limits what the paper teaches beyond "a capable agent with full logs beats a fixed prompt template."
-
Some baseline and ablation results are hard to trust. R⋆ was reimplemented by the authors since no code exists, and it scores 0.000 on CatchAbreast with a flat curve in Figure 12, which looks like an implementation problem rather than a real result. The ablations cover only four tasks, and the paper does not say how many search runs each one uses.
Questions for authors
Even if the questions below are answered well, I would not raise my score to 4. The experiments are careful, but the contribution is an engineering recipe, and its conceptual framing claims more than the method delivers.
Can you confirm the R⋆ reimplementation is working on CatchAbreast, for example by checking it against any numbers reported in the original paper?
How many search runs does each ablation in Figure 7 use?
Can the abstract report the gain against the strongest baseline in each domain alongside the gain against the baseline mean?
5 comments
We highly appreciate the insightful suggestions to help us improve the paper. We apologize for the late response, since the new experiments took time to complete. Below we respond to each point.
W1 – Gain against the baseline mean
W1. The headline numbers overstate the gain. The 29.9% and 192.8% in the abstract are measured against the mean of all baselines, not the strongest one. Against the best baseline the gain is about 16% on Bi-DexHands (0.918 vs Eureka's 0.792) and about 28% on the power grid (52.5 vs ShinkaEvolve's 41.1). Figure 6d also shows the final result is slightly worse than the baseline mean on traffic (0.95×) and only 1.1× on locomotion. The clear wins are on manipulation and the grid, and the abstract should say so.
Thank you for pointing this out. The abstract now reports the gain against the strongest baseline, on manipulation and the power grid. On locomotion and traffic ARBO matches the baselines, and Section 1 now says so. We have made this change in the abstract and Section 1.
W2 – Simple idea and framing
W2. The idea itself is simple, and the paper dresses it up more than it needs to. At its core, the method replaces Eureka's search loop with a coding agent that can read a log directory and keep a notes file. The "belief" that the paper compares to a Bayesian posterior is an analysis.md file the agent rewrites each round. Equations 2 to 11 mostly put symbols on a standard engineering loop rather than adding anything that is used later. This doesn't make the results wrong, but it limits what the paper teaches beyond "a capable agent with full logs beats a fixed prompt template."
Thank you for the comment. ARBO is deliberately designed as a simple system. The contribution we intend to establish is more specific than replacing a fixed prompt with a coding agent. Instead, we aim to explore how to organize an agent’s access to accumulated experimental evidence when each new evaluation requires policy training. We have revised the paper on the points below:
-
What the agent replaces. To clarify, the agent replaces the hand-designed search algorithm. Existing methods fix both a selection rule and an expansion rule before the search, and ARBO lets the agent build both at every round. To improve clarity and accuracy, we have made this change in the abstract, Section 1, and Eq. 3.
-
Why a simple design. We agree the design is simple but not the problem. The problem we address is letting the agent itself decide which candidates to generate next, instead of following a scripted search. Handing the search to a common coding agent (Codex) does not solve it (Fig. 7). A concurrent work, Dream-RSI [1], solves a similar problem with a similar approach. It changes the search policy only after enough evidence has accumulated, and it aims to get past a plateau rather than to save samples. AgentPilot [2], released after our submission, shares this motivation and helps an agent choose a better next move from its past trials. ARBO pursues the same goal in reward discovery, where sample efficiency matters even more. RSI can improve efficiency in other ways, for example by locating failures in execution traces [3,4]. Apart from code errors, a reward's learning failure (not the errors of the code itself) can only be checked by running another full training. We have made this change in Section 1.
-
The Bayesian analogy and the equations. We agree the analogy needs a more careful statement. We only mean that the design is inspired by Bayesian optimization. A surrogate revises its fit after each evaluation, and our agent revises a free-text belief after each round. The equations are meant to put the five baselines and ARBO in one template (Eq. 3, Table 9, Eq. 4). We have made this change in the abstract and Section 1.
[1] Zheng et al. Dream-RSI: Recursive Self-Improvement through Evolving Worlds. arXiv:2609.14858, 2026.
[2] HKUDS. AgentPilot: Better Next Moves for Your Agent. GitHub repository, https://github.com/HKUDS/AgentPilot, released November 2026.
[3] Xia et al. RRSI: Regularized Recursive Self-Improvement of Agent Harnesses. arXiv:2609.24972, 2026.
[4] Lee et al. Recursive Harness Self-Improvement. arXiv:2607.15524, 2026.
Continued from our previous comment (2/2).
W3 – R* and ablation results
W3. Some baseline and ablation results are hard to trust. R⋆ was reimplemented by the authors since no code exists, and it scores 0.000 on CatchAbreast with a flat curve in Figure 12, which looks like an implementation problem rather than a real result. The ablations cover only four tasks, and the paper does not say how many search runs each one uses.
Thank you for the concern. We went through the per-candidate training records of the R* CatchAbreast run.
Table R1: Per-candidate training records of the R* run on CatchAbreast, 128 candidates.
| Statistic | Value |
|---|---|
| Candidates that compiled and trained to completion | 126 (2 failed at launch) |
| Candidates whose proposed reward increased during training | 126 / 126 |
| Candidates with task success > 0 at any point in training | 0 / 126 |
| Ground-truth reward, max over training (median across candidates) | 0.0036 |
| Right-hand reach term at convergence (median) | 0.88 |
| Lift term at convergence (median) | 0.24 |
| Goal-distance term at convergence (median) | 0.055 |
| Candidates containing the lift module | 127 / 128 |
| Generations | 8 |
PPO optimized every reward R* wrote. The rewards are satisfied by lifting the object with the throwing hand, so the object never reaches the goal 0.6 m away in the other hand. The lift module is inherited through crossover in 127 of 128 candidates. The archive stays in this local optimum.
In the paper, we also retrained the reward R* selected with five training seeds, 3000 iterations each, under the search configuration.
Table R2: Best success rate over training of the reward R* selected on CatchAbreast.
| Seed 0 (search) | Seed 1 | Seed 2 | Seed 3 | Seed 4 | Seed 5 |
|---|---|---|---|---|---|
| 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
The result does not depend on the training seed. The same reimplementation is the best method on SwingCup and CatchUnderarm (Table 1), so the 0.000 is specific to this task.
On the ablations, the four tasks are one per domain, GraspAndPlace, Case14, Cassie Flat, and Cologne8. We ran additional searches during the rebuttal, and each variant now uses three search runs (Q2). We have made this change in Fig. 7 and its caption.
Q1 – R* on CatchAbreast
Q1. Can you confirm the R⋆ reimplementation is working on CatchAbreast, for example by checking it against any numbers reported in the original paper?
Yes. 126 of 128 candidates trained to completion and PPO optimized every proposed reward (Table R1). The 0.000 is the score of R*'s rewards, not a pipeline failure. R* did not report Bi-DexHands and released no code, so there is no published number to compare.
Q2 – Runs per ablation
Q2. How many search runs does each ablation in Figure 7 use?
Thank you for the question. Each ablation variant uses three search runs per task. We have made this change in Fig. 7 and its caption.
Q3 – Gain against the strongest baseline
Q3. Can the abstract report the gain against the strongest baseline in each domain alongside the gain against the baseline mean?
Yes. We have made this change in the abstract.
Since there was no response from the authors, my questions are still open. The main one is whether the R⋆ reimplementation works on CatchAbreast, where it scores 0.000 with a flat curve. If that baseline is broken, the comparison is weaker than it looks. It's also still unclear how many search runs each ablation uses, and the abstract still reports gains against the baseline mean rather than the strongest baseline. Without answers to these, I'm less confident in the results than when I first gave my score, so I'm lowering it to 2 for now. I'd be glad to revisit if the authors are able to address these points.