acceptodds
Under review as a conference paper at ICLR 2027

RoboArchitect: Automating the Design of Robotic Agents with Persistent Memory

Abstract

Recent work shows that a frozen robot policy performs far better inside an agent harness, where control code decides when to query the policy, consult vision tools, or command the arm directly. Yet today these harnesses are built by hand and redesigned for each new robot or benchmark. We show that harness design can be automated, offering a strong alternative to finetuning on a novel setup. In RoboArchitect, a coding model observes a robot's rollouts, diagnoses failures, and iteratively revises the harness around the frozen policy. The harness evolves on a set of episodes for its optimization and is evaluated on held-out scenes and tasks. Without updating any model weights, RoboArchitect raises average success from 9.7% to 43.3% on unseen scenes (4.5x) and from 12.7% to 40.2% on unseen tasks (3.2x), across multiple benchmarks and policies. The gains are largest where the benchmark is far from the policy's training data: wrapping a general-purpose policy, the evolved harness yields a specialist that often outperforms even the finetuned version. Intuitive control strategies are automatically designed, such as lifting a jammed arm clear or reaching in horizontally when an obstacle blocks the way from above. The harness also evolves its own persistent memory, carrying learned experience, such as effective grasp depths, into later decisions. These results suggest that substantial headroom in robotic manipulation lies in the agent harness, and its design can be effectively automated and optimized without weight updates.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robotic agents, automated optimization, robotic memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oLynOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

RoboArchitect proposes an automated approach to improving robot task execution by optimizing the control programs surrounding a frozen vision-language-action (VLA) policy, without updating model weights. A coding LLM analyzes execution trajectories, diagnoses failures, and iteratively modifies the harness, retaining changes that improve performance on an optimization set. The framework incorporates auxiliary tools, programmatic control, and persistent memory to support execution and reuse experience across episodes. Experiments across multiple simulation benchmarks show improvements in both unseen-scene and unseen-task settings, with aggregate scores increasing from 9.7% to 43.3% and from 12.7% to 40.2%, respectively. However, the overall effectiveness remains limited, and the experiments do not isolate the contribution of persistent memory or adequately establish the independent advantage of automated optimization over carefully designed harnesses. The lack of real-robot evaluation also leaves its applicability to physical environments unverified.

Critical strengths

1. A relevant and practical problem formulation. The paper explores an alternative to updating policy weights by automatically optimizing the control programs surrounding a frozen VLA. This is a useful direction for embodied AI, as it can improve task execution through program-level changes without requiring additional policy training. The proposed approach also makes the optimization process interpretable by exposing changes to control logic and tool use.

2. Evaluation across multiple benchmarks and generalization settings. The method is evaluated on several simulation benchmarks with frozen policies, including both unseen-scene and unseen-task settings. The reported improvements over the initial harness are substantial in aggregate, and the qualitative examples illustrate how program modifications can address specific execution failures. These results provide evidence that harness optimization can improve the practical performance of existing robot policies.

Critical weaknesses

1. Limited overall effectiveness. While RoboArchitect improves performance over the initial harness, the aggregate scores remain modest, reaching 43.3 on unseen scenes and 40.2 on unseen tasks. The results do not yet demonstrate that LLM-based harness optimization can consistently unlock the broader potential of LLMs for embodied tasks.

2. Missing causal ablation of persistent memory. Persistent memory is a central component of the proposed approach, yet the paper does not directly quantify its independent contribution to task performance. It is therefore unclear whether memory provides substantial benefits beyond iterative program optimization.

3. Insufficient component attribution and fair comparisons. The experiments do not adequately disentangle the effects of programmatic control, auxiliary tools, persistent memory, and iterative optimization. Comparisons against a carefully designed harness under matched tool access and evaluation budgets are also missing, making the independent advantage of automated harness optimization difficult to establish.

4. Lack of real-robot evaluation. All evaluations are conducted in simulation. Without physical robot experiments, it remains unclear whether the reported gains transfer to real-world settings with perception noise, contact uncertainty, and execution errors.

Questions for authors

1.What is the performance difference between the full method and variants with persistent memory disabled or with historical experience unavailable? Can the authors provide results for both unseen-scene and unseen-task settings to isolate the contribution of memory?

  1. How much of the improvement comes from iterative optimization, programmatic control, and auxiliary tool use, respectively? Do the authors have controlled ablations that isolate these factors while keeping the other components fixed?

  2. How does RoboArchitect compare with a carefully engineered harness using the same frozen policy, tool access, and evaluation budget? Such a comparison would help establish whether the gains arise specifically from automated optimization rather than from stronger control logic or additional tools.

  3. Have the authors evaluated RoboArchitect on physical robots? If not, what evidence supports the transfer of the reported improvements to real-world deployment, particularly for programmatic actions and persistent memories that may be sensitive to execution errors and environmental changes?

4 comments
AuthorsOpenReview ↗

Response to reviewer oLyn (Part 1/2)

All figure, table and section numbers refer to the revised paper. We marked our changes in blue.

[Q1] How much of the improvement comes from iterative optimization, programmatic control and auxiliary tool use, respectively?

[A1] Each component contributes, and the evolved harness itself decides how much each carries in each setup. We added these ablations as Tab. 4 and Fig. 5 in App. C, and the dynamic split of control between the policy and code as Fig. 7 in §4.2.

We removed each component with everything else fixed (for ablating persistent memory: see [A3]):

held-out success (%)LIBERO-Pro-10 × RLinf-π0.5VLABench × general X-VLA (progress score)
unseen scenesunseen scenes
VLA policy2.07.3
RoboArchitect (Tab. 1)40.052.6
without iteration (one-shot design, Tab. 1)2.032.8
without the VLA (code and tools only)26.052.2
without the auxiliary tools (VLA and code only)14.051.8
without full rollout recordings (success results only)26.036.6

Neither the policy alone nor code alone is the best design everywhere: the evolved harness sets the split of work for each setup on its own (Fig. 7), and iterative optimization on full rollout recordings lifts both setups. Under unseen tasks on LIBERO-Pro, a harness built around the policy could inherit its overfitting to the original LIBERO tasks, and the code-as-policy agent evolved without the VLA could even generalize better (Tab. 4), another design within RoboArchitect's search space. Iterative refinement also beats the one-shot harness design.

[Q2] How does RoboArchitect compare with a carefully engineered harness with the same policy, tool access and budget?

[A2] Our evolved harness outperforms the hand-designed harnesses VoLo and CaP-X by a substantial margin, around the same policy and with comparable tools. We added this comparison as Tab. 5 and the paragraph "Auto-evolved harnesses outperform hand-designed robot harnesses." in App. C, with the full setup in App. D.

We evaluated VoLo and CaP-X with comparable set of tools from their official code on the same held-out episodes, on LIBERO-Pro-10 with RLinf-π0.5, a setup directly supported by their codebase:

LIBERO-Pro-10 × RLinf-π0.5held-out success (%)Claude use per held-out episode
unseen scenesunseen taskscallscost (USD)
VLA policy2.00.000
VoLo with its tools (around RLinf-π0.5)4.02.512.90.10
CaP-X (code only, no VLA)6.02.59.91.05
RoboArchitect (around RLinf-π0.5)40.010.000

These hand-designed harnesses also keep a frontier model (Claude) inside every rollout, whereas RoboArchitect uses the frontier model only offline, to optimize the harness, and acts with the VLA policy and much smaller tools alone. So RoboArchitect is also more efficient to deploy, needing no Claude call during rollouts.

These comparisons show the advantage of optimizing the harness for every setup over fixing one hand-designed harness: the evolved harness is tailored to each setup and its own challenges. This optimization is exactly what RoboArchitect contributes, as a significant step towards automated, flexible harness design beyond any fixed manual one.

AuthorsOpenReview ↗

Response to reviewer oLyn (Part 2/2)

[Q3] What is the performance difference with persistent memory disabled, on both unseen scenes and unseen tasks?

[A3] Our persistent memory plays an important role in harness performance, because it keeps learning on new episodes. We added this ablation as Fig. 6 and the paragraph "Persistent memory keeps learning on new episodes." in App. C.

By design, the memory in RoboArchitect is a persistent memory across episodes: it holds what the robot learned from its own earlier attempts, for example the grasp heights that actually lifted objects on WidowX (Fig. 4a), which later episodes read on other objects. Resetting it every episode removes exactly this mechanism, leaving each episode only what it wrote itself.

Still, we ablated the memory of the evolved harnesses in three modes: reset every episode; frozen after the harness's optimization ("training"), so that every held-out episode starts from the memory built on the training episodes and nothing carries between held-out episodes; and disabled altogether:

held-out success (%)SimplerEnv WidowX × general X-VLAVLABench × general X-VLA (progress score)LIBERO-Pro-10 × OpenVLA-7B
unseen scenesunseen scenesunseen scenes
persistent memory (Tab. 1)87.552.626.0
reset every episode79.251.324.0
frozen after training79.253.514.0
memory off79.251.324.0

The persistent memory stands out as the best mode on WidowX and LIBERO-Pro, as it carries what the robot learned in earlier episodes into later decisions; on VLABench, every mode stays close to the others. The harness maintains this memory itself during rollouts, so it adapts to the actual scenes and tasks of the held-out set. A memory frozen after training cannot adapt in this way.

[Q4] How large are the overall gains of RoboArchitect?

[A4] RoboArchitect brings large absolute gains on held-out scenes and tasks, turning frozen policies that nearly fail on them into capable agents. We added the breakdown of these gains by episode as Tab. 8 in App. D.

On average, the evolved harnesses raise held-out success by +33.6 points on unseen scenes and +27.5 points on unseen tasks over the same frozen policies, and solve 34.8% of the held-out episodes that the policies fail (Tab. 8).

These gains come exactly where finetuned policies break down. As reported in LIBERO-Pro, OpenVLA, π0 and π0.5, all above 90% on standard LIBERO, collapse to near zero once object positions change, and the bare policies on our swapped LIBERO-10 suite agree (0.0 to 10.0 on unseen scenes). Without new demonstrations or weight updates, the evolved harness brings the same finetuned policies back to 24.0 to 40.0.

The evolved harness can go past finetuning. Around the general X-VLA, it outperforms the X-VLA finetuned on that very benchmark, on VLABench under both protocols and on WidowX unseen scenes (52.6 vs 35.2, 72.5 vs 32.1, and 87.5 vs 70.8). It also outperforms hand-designed agent harnesses around the same policy by a substantial margin ([A2]).

Our claim is not that an LLM can now solve embodied tasks by itself. It is that the agent around a frozen policy can be designed and improved automatically, offline, without new demonstrations or weight updates, and that this unlocks substantial performance on setups the policy was not trained for.

[Q5] Evaluation of RoboArchitect on physical robots: what evidence supports transfer to real-world deployment?

[A5] The evolved harnesses rely only on what a real robot also provides, so we expect them to transfer; a real-robot evaluation is our next step (App. A). The evidence:

  • they read only what a real robot also has: camera images, depth, calibration, the robot's own readings, and the instruction;

  • their routines rely on measured geometry and handle perception and execution errors, such as checking grasps from depth, reaching in from the side when something stands overhead, and lifting clear when commanded and actual motion disagree, which help real robots just as much;

  • memory is indexed by measured quantities, such as an object's height, which a real robot measures the same way;

  • SimplerEnv, used for WidowX and the Google robot, was built to track real-robot results, and on it the evolved harnesses work well for both robots (Tabs. 1 and 2).

Thank you for the detailed rebuttal and the additional experiments. The absolute gains (+33.6 / +27.5 points) and the three repeated evolutions convince me that the improvement over the bare policy is real and does not depend on a single run. Fig. 5 and the one-shot results also show that iteration and trace-based diagnosis both matter.

My main concerns, however, are only partly addressed. The memory ablation covers unseen scenes but not unseen tasks, and the effects it shows are within evaluation noise. On WidowX the gap is 4 episodes out of 48, and on LIBERO-Pro it is 1 out of 50. On VLABench, frozen memory scores above persistent memory, and every gap there is smaller than the spread from re-evaluating the same harness. On LIBERO-Pro, memory frozen after training is the worst mode, so any benefit seems to come from accumulating memory over the held-out episodes rather than from carrying training experience forward. For a contribution named in the title, this support is thin.

The component ablations are each a single run, compared against evolution 1, the best of the three full-method runs on LIBERO-Pro (40.0 vs. 26.0 and 30.0). The no-VLA and success-only variants (both 26.0) therefore fall within the full method’s own variance. On VLABench unseen tasks, the full harness uses code on every step, so its gap to the no-VLA variant reflects variance between runs, not a contribution from the VLA. On LIBERO-Pro unseen tasks, the no-VLA harness outperforms the full one, which contradicts the claim that the harness finds the optimal split of work. The VoLo and CaP-X comparison is also not matched in budget: those baselines receive no setup-specific tuning, and RoboArchitect’s own one-shot harness scores below both of them.

I am keeping my score at 2. I would be open to raising it if the final version reports memory results on unseen tasks, compares the ablations against the mean of the repeated evolutions, and tones down the memory and “optimal split” claims to match the evidence.

Thank you for acknowledging the absolute gains, their robustness across repeated harness evolutions, and the advantages of iterative refinement and trace-grounded optimization (Tab. 1, Fig. 5).

Regarding VoLo and CaP-X, we evaluated their official implementations on the same held-out LIBERO-Pro-10 episodes, and both codebases support LIBERO-family evaluation. Our method features iterative self-improvement rather than one-shot agent design. As acknowledged above, the iterative improvements beyond the one-shot harness are an important part of what RoboArchitect achieves.

Regarding memory, RoboArchitect's persistent memory is designed to accumulate and reuse experience across episodes during inference. Resetting it every episode, freezing it after "training", or disabling it removes this defining functionality during held-out evaluation. The observed benefit from accumulating memory during held-out episodes is therefore consistent with its intended design. Moreover, our contribution is that these memory mechanisms emerge automatically, without any predefined memory structures or rules. Fig. 4 illustrates intuitive learned memories that directly inform later control decisions, while Tab. 3 shows how frequently they are accessed and yield actionable information. Alongside the diverse control routines in Figs. 3 & 9, these findings demonstrate how automated harness optimization discovers useful agent mechanisms that work together, rather than relying on manually specified designs.

Similarly, our analysis of the division of work between the VLA, code, and tools is intended to reveal what emerges from automated optimization, rather than to claim an optimal split in every setup. As Fig. 7 illustrates, different setups evolve very different divisions of control, ranging from relying primarily on the VLA to replacing it entirely with code. RoboArchitect discovers these arrangements from rollout evidence, without any manually designed template prescribing when to use the policy or code. This dynamically brings together VLA policies and code-as-policy, traditionally treated as separate paradigms, within a single automatically optimized agent, thus substantially reducing the need to hand-design their collaboration.

Reviewer_wry9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work proposed ROBOARCHITECT, an agentic harness that observes a robot’s rollouts, diagnoses failures, and iteratively revises the harness around the frozen policy. Experimental results show that ROBOARCHITECT gets significant improvement across several simulation embodied benchmarks specifically on out-of-domain scenarios.

Critical strengths
  1. The proposed optimization loop is broadly applicable: it is applied to seven frozen policies across five simulation benchmarks (Tab. 1), and further evolves a single unified harness that works across four benchmarks and three robot embodiments (Tab. 2).
  2. The evolved harnesses bring large absolute gains over the bare policies, raising mean held-out success from 9.7% to 43.3% on unseen scenes and from 12.7% to 40.2% on unseen tasks (Tab. 1). The gains are largest in cross-benchmark and out-of-domain settings, e.g., X-VLA on VLABench unseen tasks goes from 1.2% to 72.5%.
  3. The evaluation design is sound: harnesses are optimized on training episodes and evaluated on held-out episodes under two protocols, unseen scenes and unseen tasks, which directly tests whether the evolved code generalizes rather than overfitting to the optimization episodes.
Critical weaknesses
  1. Most of the contribution comes from the frontier coding model, not from the method. What the authors actually design is the optimization loop, the harness interface and tool server, the prompt (App. E), and the evaluation protocol. Every control routine and memory structure the paper highlights (Figs. 3–7, App. B) is written by claude-opus-5. The loop itself (run rollouts, let a coding agent analyze traces and revise the code) follows prior auto-research work on self-improving agents (§2), and its acceptance rule is taken from Zhang et al. (2026a;b) (§3). So it's unclear what the methodological contribution is beyond "a strong coding model, given good tools and traces, can write a good robot agent."
  2. Only one coding model and one loop design are tested. All results use claude-opus-5 at maximum effort, so we can't tell whether the gains come from the loop or from this specific model. I'd like to see (a) the same loop with weaker or open-source coding models, and (b) loop ablations with the same coding model, e.g., giving only success rates instead of full traces, or accepting every revision instead of only improving ones. If the gains hold with a weaker model, or if the trace-based diagnosis clearly matters, that would show the loop itself contributes.
  3. There is no direct-control baseline. The paper never compares against a model that controls the robot directly, e.g., a VLM or LLM that outputs actions at every step, or a code-as-policy agent built by the same loop with tools.vla removed. The code-as-policy agents in the paper only appear when the harness happens to drop the VLA during evolution (§4.2, App. B.1). Without a controlled direct-control comparison, it's unclear whether wrapping a frozen policy, which is the paper's central framing, is actually better than letting the coding model control the arm itself.
  4. The only baselines are bare policies, so the gains can't be attributed to the automated design. Every baseline in Tab. 1 is a bare policy, either the same VLA or its finetuned version, which sees only its own standard inputs. The harness gets much more: depth maps where available, camera intrinsics and extrinsics, a pixel-to-world geometry module, ready-made motion commands (App. E), and up to six auxiliary models (Molmo2-8B or Qwen3-VL-8B, Qwen3-30B, Grounding-DINO, SAM 3, GraspGen). So the improvement could come from the extra tools and 3D information rather than from the automated design itself. The paper also doesn't compare against any existing hand-designed agent, such as VoLo, HarnessVLA, or CaP-X (all cited in §2), under the same tools. Without this comparison, it's unclear whether the automated loop produces better agents than a human engineer would, which is the paper's main motivation (§1).
  5. The held-out sets are small, each setup is evolved only once, and no uncertainty is reported. Tab. 4 uses 10 trials per task on LIBERO-Pro-10, RoboCerebra, and VLABench, so the unseen-task held-out sets have only 30–50 episodes. For example, π0.5's unseen-task gain on LIBERO-Pro-10 (0.0 → 2.5%) is a single episode out of 40, and MolmoAct-2's 15.0% is 6/40, with a 95% interval of roughly 7–29%. The coding model is also stochastic, but each setup reports a single evolution of up to 5 iterations, so we don't know how much the final harness varies across runs. Since one evolution costs under $150 (App. C), I'd like to see at least three independent runs per setup, with CIs on the held-out scores. Also, the "+345%" and "4.5×" headline numbers are relative gains over baselines that are often exactly 0.0, and the means mix VLABench progress scores with binary success rates (§4.1). Please report absolute differences and keep VLABench separate.
Questions for authors
  1. Harness vs. finetuning. The abstract frames harness design as an alternative to finetuning. But in the unified setting, the finetuned X-VLA inside a harness reaches 47.0% held-out, vs. 28.8% for the general X-VLA inside a harness (Tab. 2). On SimplerEnv WidowX, the "87.5% vs. 70.8% finetuned" comparison pits a harness that ignores the VLA entirely against a bare policy. Can you run the finetuned X-VLA inside an evolved harness on VLABench and WidowX (Tab. 1), so the comparison holds the tools fixed?

  2. Benchmark leakage. During optimization, the coding model can read the simulator source and scoring code and replay episodes (App. E), and the only safeguard is the prompt's rules.

  • Were the final harnesses checked for simulator-state access or task-specific constants? How many revisions were rejected or edited by the authors?
  • The RoboCerebra memory stores the benchmark's 150-step scene-reset period (App. B.2). Do you consider this a general skill? How much of the RoboCerebra gain disappears without it?
  • Was the prompt, the tool set, or the iteration budget tuned while looking at held-out results?
  1. Memory and iteration. Persistent memory and iterative refinement are the paper's two titular claims.
  • What is the held-out performance of each final harness with remember/recall disabled? Tab. 3 shows memory is read, not that it helps.
  • In the unified harness, iteration lowers held-out WidowX from 45.8% to 24.0% (Tab. 2), and RoboCasa-365 unseen scenes shows no gain (27.9 → 27.8). Would a held-out-based stopping rule avoid this?

Others see weaknesses and limitations.

5 comments
AuthorsOpenReview ↗

Response to reviewer wry9 (Part 1/5)

All figure, table and section numbers refer to the revised paper. We marked our changes in blue.

[Q1] Does the design of RoboArchitect's optimization matter, e.g. success rates only instead of full rollout recordings?

[A1] Full rollout recordings matter, as they mirror a human engineer's observation and debugging process, in which every harness revision is grounded. We added this ablation as Fig. 5 and the paragraph "Full rollout recordings ground every revision, as RoboArchitect mirrors a human engineer's observations." in App. C.

A numeric success rate alone discards this rich evidence of how the robot behaved:

held-out success (%)VLABench × general X-VLA (progress score)LIBERO-Pro-10 × RLinf-π0.5
unseen scenesunseen scenes
VLA policy7.32.0
RoboArchitect given success results only36.626.0
RoboArchitect given full rollout recordings (Tab. 1)52.640.0

RoboArchitect still improves past the VLA policy even with a numeric success rate as its only signal, and full rollout recordings take it much further.

[Q2] How does RoboArchitect compare with a model that controls the robot directly?

[A2] RoboArchitect evolves a dynamic collaboration between the VLA policy and code-as-policy with tools, adjusted to suit each setup: the VLA carries the motions it executes well, and code-as-policy steps in where it fails. We added this comparison as Tab. 4 and the paragraph "The VLA policy, code and tools combined in RoboArchitect form a much more expressive design space than code or the policy alone." in App. C.

We compared it with RoboArchitect with the VLA policy removed, so that the coding model writes the whole controller from code and tools:

held-out success (%)LIBERO-Pro-10 × RLinf-π0.5VLABench × general X-VLA (progress score)
unseen scenesunseen tasksunseen scenesunseen tasks
VLA policy2.00.07.31.2
RoboArchitect without the VLA (code and tools only)26.025.052.258.8
RoboArchitect with the VLA and auxiliary tools (Tab. 1)40.010.052.672.5

In RoboArchitect, the VLA policy is one tool among the others, and the optimization decides whether and how to use it from rollout evidence. The VLA policy, code and tools together form a more expressive design space than code or the policy alone. Where the policy is competent, as RLinf-π0.5 on LIBERO-Pro under unseen scenes, the harness keeps it for the motions it executes well and lets code take only the decisions it gets wrong. Under unseen tasks, a harness built around this policy could inherit its overfitting to the original LIBERO tasks, and the code-as-policy agent evolved without the VLA generalizes better. Where the policy is not competent, as the general X-VLA on VLABench, the harness drops it on its own (Fig. 7 shows the split of work in every setup).

A frontier model that directly controls the arm at every step also adds latency and cost to every step. RoboArchitect uses the coding model only offline, to optimize the harness, so the evolved harness acts with the VLA policy and smaller tools alone, without that extra inference cost.

AuthorsOpenReview ↗

Response to reviewer wry9 (Part 2/5)

[Q3] Do the gains come from the extra tools rather than the automated design? How does RoboArchitect compare with hand-designed agents given the same tools?

[A3] The gains come from the automated design: given comparable tools, the hand-designed harnesses VoLo and CaP-X stay close to the VLA policy, while our evolved harness outperforms them by a substantial margin. We added this comparison as Tab. 5 and the paragraph "Auto-evolved harnesses outperform hand-designed robot harnesses." in App. C, with the full setup in App. D.

We evaluated VoLo and CaP-X with comparable set of tools from their official code on the same held-out episodes, on LIBERO-Pro-10 with RLinf-π0.5, a setup directly supported by their codebase:

LIBERO-Pro-10 × RLinf-π0.5held-out success (%)Claude use per held-out episode
unseen scenesunseen taskscallscost (USD)
VLA policy2.00.000
VoLo with its tools (around RLinf-π0.5)4.02.512.90.10
CaP-X (code only, no VLA)6.02.59.91.05
RoboArchitect (around RLinf-π0.5)40.010.000

These hand-designed harnesses also keep a frontier model (Claude) inside every rollout, whereas RoboArchitect uses the frontier model only offline, to optimize the harness, and acts with the VLA policy and much smaller tools alone. So RoboArchitect is also more efficient to deploy, needing no Claude call during rollouts.

These comparisons show the advantage of optimizing the harness for every setup over fixing one hand-designed harness: the evolved harness is tailored to each setup and its own challenges. This optimization is exactly what RoboArchitect contributes, as a significant step towards automated, flexible harness design beyond any fixed manual one.

[Q4] Does RoboArchitect also work around a finetuned X-VLA policy on VLABench and WidowX??

[A4] RoboArchitect also lifts finetuned X-VLA policies with an automatically optimized harness in most setups, as it does for the other finetuned VLAs in Tab. 1. We added this comparison as Tab. 6 and to the paragraph "An evolved harness turns a general policy into a specialist." in §4.2.

Below are the harnesses evolved around the X-VLA checkpoints finetuned on each benchmark, besides the general X-VLA:

held-out success (%)VLABench (progress score)SimplerEnv WidowX
unseen scenesunseen tasksunseen scenesunseen tasks
general X-VLA7.31.20.00.0
general X-VLA + final evolved harness52.672.587.566.7
finetuned X-VLA35.232.170.887.5
finetuned X-VLA + final evolved harness39.532.589.677.1

Whether the VLA policy is finetuned or not, RoboArchitect improves on the VLA alone in most setups. The exception is WidowX under unseen tasks, where the finetuned X-VLA alone already succeeds on most held-out episodes. On VLABench, the general checkpoint even outperforms the finetuned one once both are wrapped in an evolved harness. A general, unfinetuned VLA leaves RoboArchitect more room to design the harness, whereas a finetuned checkpoint, though stronger on its own, can overfit to the benchmark and leaves less room to steer it. Indeed, around every finetuned checkpoint the evolved harness executes the VLA's actions on most steps, while around the general checkpoint it designs the whole controller itself. Still, it finds an effective harness around finetuned models too, as Tab. 1 already shows for MolmoAct-2, OpenVLA-7B, π0.5, RLinf-π0.5 and SmolVLA.

So we do not see the general and the finetuned VLA as any design constraint for RoboArchitect: as in [A2], our framework adapts flexibly to policies of every training status (Tab. 1).

AuthorsOpenReview ↗

Response to reviewer wry9 (Part 3/5)

[Q5] What is the held-out performance of each final harness with memory disabled?

[A5] Our persistent memory plays an important role in harness performance, because it keeps learning on new episodes. We added this ablation as Fig. 6 and the paragraph "Persistent memory keeps learning on new episodes." in App. C.

By design, the memory in RoboArchitect is a persistent memory across episodes: it holds what the robot learned from its own earlier attempts, for example the grasp heights that actually lifted objects on WidowX (Fig. 4a), which later episodes read on other objects. Resetting it every episode removes exactly this mechanism, leaving each episode only what it wrote itself.

Still, we ablated the memory of the evolved harnesses in three modes: reset every episode; frozen after the harness's optimization ("training"), so that every held-out episode starts from the memory built on the training episodes and nothing carries between held-out episodes; and disabled altogether:

held-out success (%)SimplerEnv WidowX × general X-VLAVLABench × general X-VLA (progress score)LIBERO-Pro-10 × OpenVLA-7B
unseen scenesunseen scenesunseen scenes
persistent memory (Tab. 1)87.552.626.0
reset every episode79.251.324.0
frozen after training79.253.514.0
memory off79.251.324.0

The persistent memory stands out as the best mode on WidowX and LIBERO-Pro, as it carries what the robot learned in earlier episodes into later decisions; on VLABench, every mode stays close with the others. The harness maintains this memory itself during rollouts, so it adapts to the actual scenes and tasks of the held-out set. A memory frozen after training cannot adapt in this way.

[Q6] Do the gains hold with an open-weight coding model?

[A6] RoboArchitect does not require a proprietary coding model: it also works with an open-weight one. We added these results as Tab. 7 and the paragraph "RoboArchitect with an open-weight coding model." in App. C.

With the open-weight Qwen3-Coder-Next (80B) as its coding model, RoboArchitect can also evolve harnesses that outperform the VLA policy on held-out episodes (e.g., on LIBERO-Pro-10 under unseen scenes and on VLABench under unseen tasks):

held-out success (%)LIBERO-Pro-10 × RLinf-π0.5VLABench × general X-VLA (progress score)
unseen scenesunseen tasksunseen scenesunseen tasks
VLA policy2.00.07.31.2
RoboArchitect with Qwen3-Coder-Next10.00.07.32.5
RoboArchitect with claude-opus-5 (Tab. 1)40.010.052.672.5

Naturally, a stronger coding model takes the optimization much further. Also note that the hand-designed baselines keep a proprietary frontier model inside every rollout ([A3]), whereas RoboArchitect uses its coding model only offline, so whichever coding model it uses, the evolved harness acts without it.

[Q7] Why accept only the revisions that improve, instead of every revision?

[A7] Our acceptance rule keeps every revision building on the best harness so far. Single revisions often regress on the training episodes (the rejected revisions in Figs. 13 to 15), and the rule keeps such a regression from becoming the base of the next revision.

Reviewer_uvBMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper uses a coding model (Claude Code) to write and revise a control "harness" around a frozen VLA policy. The harness can call the VLA, an LLM/VLM, Grounding-DINO, SAM 3, Molmo2 pointing, GraspGen, calibrated depth geometry, and a cross-episode memory. Over at most 5 iterations, the coding model reads rollout traces from training episodes and edits the harness, keeping an edit only if training success improves. On held-out scenes and tasks across five simulated benchmarks and seven policies, mean success goes from 9.7% to 43.3% (scenes) and from 12.7% to 40.2% (tasks). The paper's headline claim is that an evolved harness around a frozen policy is an alternative to finetuning, and that it can beat the finetuned checkpoint (VLABench, WidowX).

Critical strengths
  1. Automated harness search for VLAs is a timely question, and the held-out protocol is designed sensibly. Splitting evaluation into unseen-scene and unseen-task sets, with a stated prompt (App. E) that forbids hardcoded coordinates and reading simulator state, makes the gains much more credible than training-set curves alone would.
  2. The gains where the VLA stays in the loop are large and clearly reported. On LIBERO-Pro-10, in-distribution LIBERO policies go from roughly 0–10% to 24–40% on unseen scenes (Tab. 1), and the per-revision examples (Figs. 3–7) show what changed.
Critical weaknesses

In short: the main claim, that a harness around a frozen policy is an alternative to finetuning, is confounded with "a coding agent plus foundation perception models solves simulated tasks," and every number comes from a single evolution run with no variance. Adding evaluation to fill in the gaps for these points would potentially have me increase my score.

  1. The experiments that would separate the harness-around-a-policy claim from code-as-policy were not run.

    • What was run: (a) the bare VLA; (b) the coding-agent harness around it with the full tool stack, one-shot and iterative; (c) public finetuned checkpoints, bare, with no harness or tools.
    • Why this does not support the claim: The two settings where the harness beats finetuning (X-VLA on VLABench and WidowX, Tab. 1) are the two where the harness ignores the VLA and becomes a pure code-as-policy agent (§4.2, §B, Fig. 6i). Those comparisons pit depth, calibrated cameras, SAM 3, Grounding-DINO and GraspGen against an RGB-only policy, and the frozen policy contributes nothing. The only matched comparison in the paper, the same unified harness around the general vs. the finetuned X-VLA (Tab. 2), favors finetuning: 47.0 vs. 28.8 held-out mean (WidowX 74.0 vs. 24.0, Google 70.9 vs. 45.6, VLABench 35.1 vs. 21.6).
    • Should have been run: (i) the same loop with no policy (tools only), which gives the code-as-policy baseline; (ii) the same loop with the policy but no perception/grasp tools, which isolates what a harness adds to the policy itself; (iii) the harness around the finetuned checkpoint wherever one exists in Tab. 1, so harness and finetuning are compared under the same tools; (iv) one existing harness or code-as-policy system (HarnessVLA, VoLo or CaP-X) given the same tools. For each setup, also report the fraction of executed actions that come from the VLA.
  2. Every result is one stochastic run, scored on small held-out sets, with no uncertainty.

    • What was run: one evolution per setup (claude-opus-5, ≤5 iterations, edits accepted on training success), and each final harness evaluated once. Held-out sets are small (10 trials per task on LIBERO-Pro-10; π0.5's +2.5 on unseen tasks is one episode). The headline means are relative percentages over near-zero baselines (+345%, +2228%), and they average VLABench progress scores together with success rates. The one place iteration can be checked against a second number goes the wrong way: the unified harness on WidowX falls from 45.8 (one-shot) to 24.0 (iterative) on held-out while training success rises (Tab. 2).
    • Should have been run: at least 3 independent evolutions per setup, reporting the mean ± std across evolutions plus binomial CIs over episodes, with absolute differences replacing the relative-percent means.
Questions for authors
  1. What does the same loop reach with the VLA removed, and with the perception/grasp tools removed, on at least LIBERO-Pro-10 and VLABench? This is the result that would establish the harness-around-a-policy claim.
  2. With the harness around the finetuned checkpoint in Tab. 1 (as in Tab. 2), does the general policy plus harness still beat the finetuned policy plus harness anywhere?
  3. How much do results vary across repeated evolutions of the same setup?
  4. Memory accumulates during the held-out run itself (App. E). What is held-out success with memory reset every episode, or frozen after training?
5 comments
AuthorsOpenReview ↗

Response to reviewer uvBM (Part 1/4)

All figure, table and section numbers refer to the revised paper. We marked our changes in blue.

[Q1] How does RoboArchitect perform with its components ablated, i.e. without the VLA policy, or without the perception and grasp tools?

[A1] RoboArchitect evolves a dynamic collaboration between the VLA policy and code-as-policy with tools, adjusted to suit each setup: the VLA carries the motions it executes well, and code-as-policy steps in where it fails. We added this ablation to the revised paper as Tab. 4 and the paragraph "The VLA policy, code and tools combined in RoboArchitect form a much more expressive design space than code or the policy alone." in App. C.

In RoboArchitect, the VLA policy is one tool among the others, and the optimization decides from rollout evidence whether and how to use it. The VLA policy, code and tools together form a strictly larger and more expressive design space than code alone or the policy alone. The harness can still become a pure code-as-policy agent, as it does for the general X-VLA (the pretrained checkpoint, never finetuned on these benchmarks) on VLABench and WidowX (Fig. 9i). But it can also keep the policy for the motions it executes well and take over only the decisions the policy gets wrong (Fig. 3a and 3d). Which of these is better, and how they work together, is a property of each setup, grounded in its own evidence, and finding and optimizing it is exactly what RoboArchitect automates.

To show the roles of the VLA policy and the auxiliary tools directly, we evolved harnesses with the VLA policy removed (code and tools only) or with the perception and grasp tools removed (VLA and code only), taking LIBERO-Pro-10 with RLinf-π0.5 and VLABench with the general X-VLA as example setups:

held-out success (%)LIBERO-Pro-10 × RLinf-π0.5VLABench × general X-VLA (progress score)
unseen scenesunseen tasksunseen scenesunseen tasks
VLA policy2.00.07.31.2
RoboArchitect: code and tools (no VLA)26.025.052.258.8
RoboArchitect: VLA and code (no auxiliary tools)14.00.051.841.3
RoboArchitect: VLA, code and auxiliary tools (Tab. 1)40.010.052.672.5

On LIBERO-Pro, where RLinf-π0.5 was finetuned on the original LIBERO tasks, the VLA policy remains useful under unseen scenes, even though on its own it fails to generalize to the swapped scenes (Tab. 1). So the final agent benefits greatly from both the VLA policy and the tools. Under unseen tasks, a harness built around the VLA only (no auxiliary tools) could inherit the policy's overfitting to original LIBERO tasks, and the code-as-policy agent evolved without the VLA generalizes better.

On VLABench, where the general X-VLA was never trained on the benchmark, RoboArchitect already drops it on its own and becomes a pure code-as-policy agent (§4.2). So without the VLA, it arrives at a code-as-policy agent of the same strength under unseen scenes. The grounding tools matter most on unseen tasks, e.g., selecting a specific drink out of a fridge relies on precise perception and visual grounding, which only these tools provide.

Note that even with no auxiliary tools, the harness evolved around the VLA still substantially improves held-out success over the VLA policy alone on most setups. So an evolved harness adds a great deal to the policy by itself, through programmatic code that adjusts the policy's actions.

This is exactly why we treat the VLA as one tool rather than as a "cornerstone". Neither the policy alone nor code alone is the best design everywhere, and the evolved harness sets the split of work for each setup on its own (see also [A4] for discussion on this split).

AuthorsOpenReview ↗

Response to reviewer uvBM (Part 2/4)

[Q2] Does RoboArchitect also work around a finetuned X-VLA policy, and how does a harness around a finetuned VLA compare with one around a general VLA?

[A2] RoboArchitect also lifts finetuned X-VLA policies with an automatically optimized harness in most setups, as it does for the other finetuned VLAs in Tab. 1. We added this comparison as Tab. 6 and to the paragraph "An evolved harness turns a general policy into a specialist." in §4.2.

Below are the harnesses evolved around the X-VLA checkpoints finetuned on each benchmark, besides the general X-VLA:

held-out success (%)VLABench (progress score)SimplerEnv WidowX
unseen scenesunseen tasksunseen scenesunseen tasks
general X-VLA7.31.20.00.0
general X-VLA + final evolved harness52.672.587.566.7
finetuned X-VLA35.232.170.887.5
finetuned X-VLA + final evolved harness39.532.589.677.1

Whether the VLA policy is finetuned or not, RoboArchitect improves on the VLA alone in most setups. The exception is WidowX under unseen tasks, where the finetuned X-VLA alone already succeeds on most held-out episodes. On VLABench, the general checkpoint even outperforms the finetuned one once both are wrapped in an evolved harness. A general, unfinetuned VLA leaves RoboArchitect more room to design the harness, whereas a finetuned checkpoint, though stronger on its own, can overfit to the benchmark and leaves less room to steer it. Indeed, around every finetuned checkpoint the evolved harness executes the VLA's actions on most steps, while around the general checkpoint it designs the whole controller itself. Still, it finds an effective harness around finetuned models too, as Tab. 1 already shows for MolmoAct-2, OpenVLA-7B, π0.5, RLinf-π0.5 and SmolVLA.

So we do not see the general and the finetuned VLA as any design constraint for RoboArchitect: as in [A1], our framework adapts flexibly to policies of every training status (Tab. 1).

[Q3] How does RoboArchitect compare with existing hand-designed harnesses given comparable set of tools?

[A3] Our evolved harness outperforms the hand-designed harnesses VoLo and CaP-X by a significant margin, around the same policy and with comparable tools. We added this comparison as Tab. 5 and the paragraph "Auto-evolved harnesses outperform hand-designed robot harnesses." in App. C.

We evaluated VoLo and CaP-X with comparable set of tools from their official code on the same held-out episodes, on LIBERO-Pro-10 with RLinf-π0.5, a setup directly supported by their codebase:

LIBERO-Pro-10 × RLinf-π0.5held-out success (%)Claude use per held-out episode
unseen scenesunseen taskscallscost (USD)
VLA policy2.00.000
VoLo with its tools (around RLinf-π0.5)4.02.512.90.10
CaP-X (code only, no VLA)6.02.59.91.05
RoboArchitect (around RLinf-π0.5)40.010.000

These hand-designed harnesses also keep a frontier model (Claude) inside every rollout, whereas RoboArchitect uses the frontier model only offline, to optimize the harness, and acts with the VLA policy and much smaller tools alone. So RoboArchitect is also more efficient to deploy, needing no Claude call during rollouts.

These comparisons show the advantage of optimizing the harness for every setup over fixing one hand-designed harness: the evolved harness is tailored to each setup and its own challenges. This optimization is exactly what RoboArchitect contributes, as a significant step towards automated, flexible harness design beyond any fixed manual one.

AuthorsOpenReview ↗

Response to reviewer uvBM (Part 3/4)

[Q4] What fraction of the executed actions comes from the VLA policy, and what fraction from code?

[A4] RoboArchitect sets the division of labor between the VLA and code automatically, for each setup, instead of following a fixed template, which is another advantage over hand-designed harnesses ([A3]). We added analyses of this split as Fig. 7 and to the paragraph "The harness dynamically balances policy and programmatic control." in §4.2. Below is the split of work in every final evolved harness on its held-out episodes, grouped by which side drives the robot mostly:

held-out steps (%)VLA policy successVLA action as returnedVLA action postprocessedcode action
the VLA policy keeps control
RoboCasa-365 × RLDX-1, unseen scenes27.993.20.76.1
RoboCasa-365 × RLDX-1, unseen tasks73.590.27.02.8
RoboCerebra × SmolVLA, unseen scenes20.090.75.83.5
LIBERO-Pro-10 × MolmoAct-2, unseen scenes10.085.02.013.0
LIBERO-Pro-10 × RLinf-π0.5, unseen scenes2.084.92.412.7
LIBERO-Pro-10 × π0.5, unseen tasks0.096.70.72.6
LIBERO-Pro-10 × MolmoAct-2, unseen tasks2.577.73.418.9
RoboCerebra × π0.5, unseen tasks24.072.96.121.0
LIBERO-Pro-10 × π0.5, unseen scenes8.071.26.622.2
the VLA policy and code share the work
RoboCerebra × SmolVLA, unseen tasks26.055.910.333.8
RoboCerebra × π0.5, unseen scenes22.055.65.438.9
LIBERO-Pro-10 × RLinf-π0.5, unseen tasks0.043.216.340.5
LIBERO-Pro-10 × OpenVLA-7B, unseen tasks0.037.711.850.5
code takes over
LIBERO-Pro-10 × SmolVLA, unseen scenes0.00.07.093.0
LIBERO-Pro-10 × SmolVLA, unseen tasks0.06.45.388.3
LIBERO-Pro-10 × OpenVLA-7B, unseen scenes0.05.713.780.6
VLABench × general X-VLA, unseen scenes7.30.00.0100.0
VLABench × general X-VLA, unseen tasks1.20.00.0100.0
SimplerEnv WidowX × general X-VLA, unseen scenes0.00.00.0100.0
SimplerEnv WidowX × general X-VLA, unseen tasks0.00.00.0100.0

We see that

  • Where the VLA policy is competent, it keeps control, as for RLDX-1 on RoboCasa, SmolVLA on RoboCerebra unseen scenes, and MolmoAct-2 and RLinf-π0.5 on LIBERO-Pro unseen scenes; code steps in only for the decisions the policy gets wrong, such as lifting a jammed arm clear (Fig. 3a).

  • Where the VLA policy fails completely, code takes over, as for SmolVLA and OpenVLA-7B on LIBERO-Pro unseen scenes, and the general X-VLA on VLABench and WidowX.

  • The harness can also postprocess the VLA's actions, such as rescaling the contact force to latch a door on RoboCasa (Fig. 3c): the code keeps the VLA in charge but calibrates it, achieving a form of organic collaboration.

[Q5] How much do results vary across repeated evolutions of the same setup?

[A5] RoboArchitect consistently lifts performance across repeated evolutions. Taking LIBERO-Pro-10 with RLinf-π0.5 and VLABench with the general X-VLA as example setups, we repeated the whole evolution twice more:

held-out success (%)LIBERO-Pro-10 × RLinf-π0.5VLABench × general X-VLA (progress score)
unseen scenesunseen scenes
VLA policy2.07.3
evolution 1 (Tab. 1)40.052.6
evolution 226.060.4
evolution 330.059.3
mean32.057.4

Every evolution lifts held-out success far above the VLA policy, by +30.0 and +50.1 points on average, so the gains of RoboArchitect do not rest on any single evolution.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

We thank all reviewers and the Area Chairs for reviewing our paper.


We summarize the key results and additional evidence presented in our revised paper and rebuttal:

  1. Substantial, robust gains over frozen VLA policies. RoboArchitect improves average held-out success by 33.6 and +27.5 points on unseen scenes and tasks, with consistent improvements across three independent harness evolutions and repeated evaluations. The evolved harness can also outperform finetuning in several settings, even when both the general and finetuned policies are wrapped in harnesses. (Tabs. 1, 6, 8; §4.2; [uvBM Q2, Q5–6], [wry9 Q4, Q8], [oLyn Q4])

  2. RoboArchitect's optimization design is effective. Iterative refinement improves beyond one-shot agent design, while full rollout recordings produce substantially stronger harnesses than success signals alone. RoboArchitect can also evolve harnesses with an open-weight coding model, showing that our automated optimization framework works beyond proprietary frontier models. (Tab. 1, Fig. 5, Tab. 7; §4.2; App. C; [wry9 Q1, Q6], [oLyn Q1])

  3. Automated harness optimization outperforms existing hand-designed agents. Our evolved harness substantially outperforms VoLo and CaP-X on the same held-out LIBERO-Pro episodes with comparable tools, while requiring no frontier-model calls during inference. (Tab. 5; App. C; [uvBM Q3], [wry9 Q3], [oLyn Q2])

  4. Useful control strategies and persistent memories emerge automatically. RoboArchitect discovers diverse ways to dynamically divide control between VLA policies and code, alongside memory mechanisms that accumulate and reuse experience without predefined structures or rules. Additional component ablations further examine how the VLA, code, tools, and memory contribute to the evolved harness. (Figs. 3–4, 6–7, Tabs. 3–4; §4.2–4.4, App. C; [uvBM Q1, Q4, Q7], [wry9 Q2, Q5], [oLyn Q1, Q3])

  5. A unified harness works across different robot embodiments. RoboArchitect automatically evolves a single harness that operates simultaneously across several benchmarks and robot embodiments. (Tab. 2, Fig. 12; §4.2; [wry9])


We also highlight key strengths of our work acknowledged by reviewers:

  1. A timely and practical research direction: Automating agent harness design offers a promising alternative to manually engineering controllers or finetuning policy weights, while making the control decisions interpretable. ([uvBM], [oLyn])

  2. Broad evaluation with sound generalization protocols: Our evaluation spans diverse VLA policies and benchmarks, with held-out unseen scenes and tasks providing evidence of generalization beyond the episodes used for optimization. ([uvBM], [wry9], [oLyn])

  3. Substantial performance gains with interpretable control routines: Our evolved harnesses achieve large improvements over bare VLA policies, while addressing concrete execution failure patterns through intuitive control logic. ([uvBM], [wry9], [oLyn])


In light of the above, we believe RoboArchitect makes a significant contribution toward automating the design and improvement of robotic agents. Our extensive empirical evidence shows that automated optimization can discover effective control strategies, policy-code collaboration, and persistent memory mechanisms without updating model weights.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.