- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RoboArchitect proposes an automated approach to improving robot task execution by optimizing the control programs surrounding a frozen vision-language-action (VLA) policy, without updating model weights. A coding LLM analyzes execution trajectories, diagnoses failures, and iteratively modifies the harness, retaining changes that improve performance on an optimization set. The framework incorporates auxiliary tools, programmatic control, and persistent memory to support execution and reuse experience across episodes. Experiments across multiple simulation benchmarks show improvements in both unseen-scene and unseen-task settings, with aggregate scores increasing from 9.7% to 43.3% and from 12.7% to 40.2%, respectively. However, the overall effectiveness remains limited, and the experiments do not isolate the contribution of persistent memory or adequately establish the independent advantage of automated optimization over carefully designed harnesses. The lack of real-robot evaluation also leaves its applicability to physical environments unverified.
Critical strengths
1. A relevant and practical problem formulation. The paper explores an alternative to updating policy weights by automatically optimizing the control programs surrounding a frozen VLA. This is a useful direction for embodied AI, as it can improve task execution through program-level changes without requiring additional policy training. The proposed approach also makes the optimization process interpretable by exposing changes to control logic and tool use.
2. Evaluation across multiple benchmarks and generalization settings. The method is evaluated on several simulation benchmarks with frozen policies, including both unseen-scene and unseen-task settings. The reported improvements over the initial harness are substantial in aggregate, and the qualitative examples illustrate how program modifications can address specific execution failures. These results provide evidence that harness optimization can improve the practical performance of existing robot policies.
Critical weaknesses
1. Limited overall effectiveness. While RoboArchitect improves performance over the initial harness, the aggregate scores remain modest, reaching 43.3 on unseen scenes and 40.2 on unseen tasks. The results do not yet demonstrate that LLM-based harness optimization can consistently unlock the broader potential of LLMs for embodied tasks.
2. Missing causal ablation of persistent memory. Persistent memory is a central component of the proposed approach, yet the paper does not directly quantify its independent contribution to task performance. It is therefore unclear whether memory provides substantial benefits beyond iterative program optimization.
3. Insufficient component attribution and fair comparisons. The experiments do not adequately disentangle the effects of programmatic control, auxiliary tools, persistent memory, and iterative optimization. Comparisons against a carefully designed harness under matched tool access and evaluation budgets are also missing, making the independent advantage of automated harness optimization difficult to establish.
4. Lack of real-robot evaluation. All evaluations are conducted in simulation. Without physical robot experiments, it remains unclear whether the reported gains transfer to real-world settings with perception noise, contact uncertainty, and execution errors.
Questions for authors
1.What is the performance difference between the full method and variants with persistent memory disabled or with historical experience unavailable? Can the authors provide results for both unseen-scene and unseen-task settings to isolate the contribution of memory?
-
How much of the improvement comes from iterative optimization, programmatic control, and auxiliary tool use, respectively? Do the authors have controlled ablations that isolate these factors while keeping the other components fixed?
-
How does RoboArchitect compare with a carefully engineered harness using the same frozen policy, tool access, and evaluation budget? Such a comparison would help establish whether the gains arise specifically from automated optimization rather than from stronger control logic or additional tools.
-
Have the authors evaluated RoboArchitect on physical robots? If not, what evidence supports the transfer of the reported improvements to real-world deployment, particularly for programmatic actions and persistent memories that may be sensitive to execution errors and environmental changes?
4 comments
Response to reviewer oLyn (Part 2/2)
[Q3] What is the performance difference with persistent memory disabled, on both unseen scenes and unseen tasks?
[A3] Our persistent memory plays an important role in harness performance, because it keeps learning on new episodes. We added this ablation as Fig. 6 and the paragraph "Persistent memory keeps learning on new episodes." in App. C.
By design, the memory in RoboArchitect is a persistent memory across episodes: it holds what the robot learned from its own earlier attempts, for example the grasp heights that actually lifted objects on WidowX (Fig. 4a), which later episodes read on other objects. Resetting it every episode removes exactly this mechanism, leaving each episode only what it wrote itself.
Still, we ablated the memory of the evolved harnesses in three modes: reset every episode; frozen after the harness's optimization ("training"), so that every held-out episode starts from the memory built on the training episodes and nothing carries between held-out episodes; and disabled altogether:
| held-out success (%) | SimplerEnv WidowX × general X-VLA | VLABench × general X-VLA (progress score) | LIBERO-Pro-10 × OpenVLA-7B |
|---|---|---|---|
| unseen scenes | unseen scenes | unseen scenes | |
| persistent memory (Tab. 1) | 87.5 | 52.6 | 26.0 |
| reset every episode | 79.2 | 51.3 | 24.0 |
| frozen after training | 79.2 | 53.5 | 14.0 |
| memory off | 79.2 | 51.3 | 24.0 |
The persistent memory stands out as the best mode on WidowX and LIBERO-Pro, as it carries what the robot learned in earlier episodes into later decisions; on VLABench, every mode stays close to the others. The harness maintains this memory itself during rollouts, so it adapts to the actual scenes and tasks of the held-out set. A memory frozen after training cannot adapt in this way.
[Q4] How large are the overall gains of RoboArchitect?
[A4] RoboArchitect brings large absolute gains on held-out scenes and tasks, turning frozen policies that nearly fail on them into capable agents. We added the breakdown of these gains by episode as Tab. 8 in App. D.
On average, the evolved harnesses raise held-out success by +33.6 points on unseen scenes and +27.5 points on unseen tasks over the same frozen policies, and solve 34.8% of the held-out episodes that the policies fail (Tab. 8).
These gains come exactly where finetuned policies break down. As reported in LIBERO-Pro, OpenVLA, π0 and π0.5, all above 90% on standard LIBERO, collapse to near zero once object positions change, and the bare policies on our swapped LIBERO-10 suite agree (0.0 to 10.0 on unseen scenes). Without new demonstrations or weight updates, the evolved harness brings the same finetuned policies back to 24.0 to 40.0.
The evolved harness can go past finetuning. Around the general X-VLA, it outperforms the X-VLA finetuned on that very benchmark, on VLABench under both protocols and on WidowX unseen scenes (52.6 vs 35.2, 72.5 vs 32.1, and 87.5 vs 70.8). It also outperforms hand-designed agent harnesses around the same policy by a substantial margin ([A2]).
Our claim is not that an LLM can now solve embodied tasks by itself. It is that the agent around a frozen policy can be designed and improved automatically, offline, without new demonstrations or weight updates, and that this unlocks substantial performance on setups the policy was not trained for.
[Q5] Evaluation of RoboArchitect on physical robots: what evidence supports transfer to real-world deployment?
[A5] The evolved harnesses rely only on what a real robot also provides, so we expect them to transfer; a real-robot evaluation is our next step (App. A). The evidence:
-
they read only what a real robot also has: camera images, depth, calibration, the robot's own readings, and the instruction;
-
their routines rely on measured geometry and handle perception and execution errors, such as checking grasps from depth, reaching in from the side when something stands overhead, and lifting clear when commanded and actual motion disagree, which help real robots just as much;
-
memory is indexed by measured quantities, such as an object's height, which a real robot measures the same way;
-
SimplerEnv, used for WidowX and the Google robot, was built to track real-robot results, and on it the evolved harnesses work well for both robots (Tabs. 1 and 2).
Thank you for the detailed rebuttal and the additional experiments. The absolute gains (+33.6 / +27.5 points) and the three repeated evolutions convince me that the improvement over the bare policy is real and does not depend on a single run. Fig. 5 and the one-shot results also show that iteration and trace-based diagnosis both matter.
My main concerns, however, are only partly addressed. The memory ablation covers unseen scenes but not unseen tasks, and the effects it shows are within evaluation noise. On WidowX the gap is 4 episodes out of 48, and on LIBERO-Pro it is 1 out of 50. On VLABench, frozen memory scores above persistent memory, and every gap there is smaller than the spread from re-evaluating the same harness. On LIBERO-Pro, memory frozen after training is the worst mode, so any benefit seems to come from accumulating memory over the held-out episodes rather than from carrying training experience forward. For a contribution named in the title, this support is thin.
The component ablations are each a single run, compared against evolution 1, the best of the three full-method runs on LIBERO-Pro (40.0 vs. 26.0 and 30.0). The no-VLA and success-only variants (both 26.0) therefore fall within the full method’s own variance. On VLABench unseen tasks, the full harness uses code on every step, so its gap to the no-VLA variant reflects variance between runs, not a contribution from the VLA. On LIBERO-Pro unseen tasks, the no-VLA harness outperforms the full one, which contradicts the claim that the harness finds the optimal split of work. The VoLo and CaP-X comparison is also not matched in budget: those baselines receive no setup-specific tuning, and RoboArchitect’s own one-shot harness scores below both of them.
I am keeping my score at 2. I would be open to raising it if the final version reports memory results on unseen tasks, compares the ablations against the mean of the repeated evolutions, and tones down the memory and “optimal split” claims to match the evidence.
Thank you for acknowledging the absolute gains, their robustness across repeated harness evolutions, and the advantages of iterative refinement and trace-grounded optimization (Tab. 1, Fig. 5).
Regarding VoLo and CaP-X, we evaluated their official implementations on the same held-out LIBERO-Pro-10 episodes, and both codebases support LIBERO-family evaluation. Our method features iterative self-improvement rather than one-shot agent design. As acknowledged above, the iterative improvements beyond the one-shot harness are an important part of what RoboArchitect achieves.
Regarding memory, RoboArchitect's persistent memory is designed to accumulate and reuse experience across episodes during inference. Resetting it every episode, freezing it after "training", or disabling it removes this defining functionality during held-out evaluation. The observed benefit from accumulating memory during held-out episodes is therefore consistent with its intended design. Moreover, our contribution is that these memory mechanisms emerge automatically, without any predefined memory structures or rules. Fig. 4 illustrates intuitive learned memories that directly inform later control decisions, while Tab. 3 shows how frequently they are accessed and yield actionable information. Alongside the diverse control routines in Figs. 3 & 9, these findings demonstrate how automated harness optimization discovers useful agent mechanisms that work together, rather than relying on manually specified designs.
Similarly, our analysis of the division of work between the VLA, code, and tools is intended to reveal what emerges from automated optimization, rather than to claim an optimal split in every setup. As Fig. 7 illustrates, different setups evolve very different divisions of control, ranging from relying primarily on the VLA to replacing it entirely with code. RoboArchitect discovers these arrangements from rollout evidence, without any manually designed template prescribing when to use the policy or code. This dynamically brings together VLA policies and code-as-policy, traditionally treated as separate paradigms, within a single automatically optimized agent, thus substantially reducing the need to hand-design their collaboration.
Response to reviewer oLyn (Part 1/2)
All figure, table and section numbers refer to the revised paper. We marked our changes in blue.
[Q1] How much of the improvement comes from iterative optimization, programmatic control and auxiliary tool use, respectively?
[A1] Each component contributes, and the evolved harness itself decides how much each carries in each setup. We added these ablations as Tab. 4 and Fig. 5 in App. C, and the dynamic split of control between the policy and code as Fig. 7 in §4.2.
We removed each component with everything else fixed (for ablating persistent memory: see [A3]):
Neither the policy alone nor code alone is the best design everywhere: the evolved harness sets the split of work for each setup on its own (Fig. 7), and iterative optimization on full rollout recordings lifts both setups. Under unseen tasks on LIBERO-Pro, a harness built around the policy could inherit its overfitting to the original LIBERO tasks, and the code-as-policy agent evolved without the VLA could even generalize better (Tab. 4), another design within RoboArchitect's search space. Iterative refinement also beats the one-shot harness design.
[Q2] How does RoboArchitect compare with a carefully engineered harness with the same policy, tool access and budget?
[A2] Our evolved harness outperforms the hand-designed harnesses VoLo and CaP-X by a substantial margin, around the same policy and with comparable tools. We added this comparison as Tab. 5 and the paragraph "Auto-evolved harnesses outperform hand-designed robot harnesses." in App. C, with the full setup in App. D.
We evaluated VoLo and CaP-X with comparable set of tools from their official code on the same held-out episodes, on LIBERO-Pro-10 with RLinf-π0.5, a setup directly supported by their codebase:
These hand-designed harnesses also keep a frontier model (Claude) inside every rollout, whereas RoboArchitect uses the frontier model only offline, to optimize the harness, and acts with the VLA policy and much smaller tools alone. So RoboArchitect is also more efficient to deploy, needing no Claude call during rollouts.
These comparisons show the advantage of optimizing the harness for every setup over fixing one hand-designed harness: the evolved harness is tailored to each setup and its own challenges. This optimization is exactly what RoboArchitect contributes, as a significant step towards automated, flexible harness design beyond any fixed manual one.