- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work proposes RoboFoundry, an embodied agent framework that formulates the system supporting a foundation model as the policy and makes this system the unit of evolution. RoboFoundry localizes system-level bottlenecks as capability gaps in execution traces, edits only the responsible surface, and promotes generalized edits that pass held-out verification to the general system. Across three embodied simulation benchmarks, RoboFoundry improves embodied decision-making across foundation models of varying capability and outperforms existing methods on long-horizon memory and on manipulation under changes in object positions and task specifications. In real-robot experiments on a bimanual platform and a Unitree G1 humanoid, RoboFoundry transfers evolved skills to unseen objects and continues to evolve after deployment, as in a nesting-doll task where edits made between trials raise success to 70% over ten trials against 30% for the model alone.
Critical strengths
RoboFoundry offers a promising structure that turns LLM-written task logic into robot actions and confines each revision to the component responsible for a failure. Embedding it in a learning loop that promotes reusable rules across tasks makes it a plausible basis for larger systems spanning many tasks and robots.
Extensive real-world experiments and multi-axis analyses substantiate that the evolved system transfers across robots and keeps improving after deployment, with gains that hold across backbones.
Critical weaknesses
In the EmbodiedBench experiments, the variant that removes part of RoboFoundry keeps a fixed general system and applies task-level repairs alone, which still rely on capability diagnosis. As a result, it remains less clear how much capability diagnosis itself adds beyond revising the system directly from failure records. Environment interactions are matched across variants, although reporting the inference used for diagnosis and editing would make the comparison easier to interpret.
While task-level repairs are validated through execution, promotion to the general system relies on a model judgment on recorded traces from other tasks rather than on executing the proposed updates on held-out tasks. Although such a judgment is a reasonable low-cost proxy, the paper does not establish how reliably it predicts actual transfer performance or whether promoted updates introduce regressions on other tasks. This weakens the support for the claim that system updates are validated.
Questions for authors
How much model inference do the diagnosis and editing steps use compared with the baselines, and how well do the scores that decide which edits enter the general system predict actual execution outcomes?
The per-task tables show where RoboFoundry gains most, but what do the tasks with large gains have in common compared with those with small or negative gains?
How often are rules deleted or replaced over the course of evolution?
Do the authors view this framework as a long-term architecture even as foundation models evolve to internalize the abilities that its system components currently provide? For example, if a model could diagnose its own failures and revise its behavior on its own, which part of the method would become the main bottleneck, and would the save-retrieve-use memory operations and the six capability categories for diagnosing failures still be justified?
5 comments
Reply to Reviewer 4SWF - Part 2
We acknowledge that positive transfer does not guarantee the absence of regressions on previously successful tasks. The current promotion check does not explicitly test cross-capability retention. In Section 3.1, we have clarified this and also report the inference overhead of diagnosis and editing separately from the matched environment-interaction budget: “We further evaluate the promotion verifier by…”.
Together, these experiments support two key components of RoboFoundry: capability-guided attribution improves repair success, and the trace-based verifier predicts which general-system edits are more likely to transfer successfully.
Q1.2: Inference and Promotion Calls
For EmbodiedBench, all methods use model inference during task execution under the same environment interaction budget. Compared with the raw-backbone baselines, RoboFoundry additionally performs one outer-loop Revise call after each rollout for capability diagnosis and system editing.
For a complete sweep of 1128 scored tasks, this corresponds to 1128 additional model calls, or one extra call per task rather than per environment step. Each evolution call has a budget of 4096 input and 4096 output tokens, corresponding to a maximum budget of 9.24M additional tokens under this one-call-per-revision setting.
Q2: Characteristics of High- and Low-Gain Tasks
We thank the reviewer for this question. We observe a pattern in the per-task results. Tasks with large gains tend to benefit from reusable reasoning, planning, and recovery strategies. For example, EB-Habitat Spatial improves across all five backbones, with GPT-5.5 increasing from 32.0% to 100.0% and GLM5.3-Flash from 34.0% to 94.0%. As illustrated in Appendix A.1.1, RoboFoundry can turn failures into reusable rules, such as observation checkpoints and feedback-guided recovery, that benefit subsequent tasks.
In contrast, small or negative gains often appear in tasks requiring scene-specific visual grounding and precise action refinement. For example, EB-Manipulation Visual decreases for four of the five backbones, including GPT-5.5 (58.3% to 41.7%) and Qwen3.7-Plus (61.1% to 44.4%). Such tasks may benefit less from general system updates, which can also interfere with previously effective task-specific strategies.
In Section 3.1, we have added this analysis to clarify that RoboFoundry tends to benefit most from failures that can be turned into reusable system-level improvements, while scene-specific failures may show smaller gains or even regressions: “Larger gains tend to occur in tasks…”.
Q3: Rule Update During Evolution
We thank the reviewer for this question. We analyze rule changes over the full EmbodiedBench on GPT-6 Astra, Qwen3.7-Plus, and GLM5.3-Flash. For every promoted update, we record each rule-level operation as addition, replacement, or deletion. Each operation is counted separately.
| Backbone | Rule Operations | Addition | Replacement | Deletion |
|---|---|---|---|---|
| GPT-6 Astra | 142 | 101 (71.1%) | 29 (20.4%) | 12 (8.5%) |
| Qwen3.7-Plus | 166 | 116 (69.9%) | 35 (21.1%) | 15 (9.0%) |
| GLM5.3-Flash | 153 | 109 (71.2%) | 31 (20.3%) | 13 (8.5%) |
| Overall | 461 | 326 (70.7%) | 95 (20.6%) | 40 (8.7%) |
Across all three backbones, 20.6% of rule operations replace existing rules, while 8.7% delete them. Together, 29.3% of operations modify or remove existing knowledge rather than simply adding new rules.
RoboFoundry updates existing rules when later execution feedback suggests a better correction, while deletion removes rules that are no longer useful or become inconsistent with new experience. These results show that RoboFoundry does not simply accumulate rules, but continuously revises its persistent general system as new execution experience arrives.
Q4: Long-Term Role of RoboFoundry
We thank the reviewer for this question. We view RoboFoundry as a system-level architecture, rather than a temporary solution for weak foundation models. Our results already support this: even with the stronger GPT-6 Astra backbone, RoboFoundry improves EmbodiedBench from 71.9% to 78.0%, showing that system-level evolution remains useful as foundation models become stronger. Similarly, Qwen3.7-Plus + RoboFoundry reaches 70.3%, close to GPT-5.5 + RoboFoundry at 72.7%, showing that system evolution benefits backbones of different strengths.
Reply to Reviewer 4SWF - Part 3
If future models can diagnose and repair failures internally, the main bottleneck will shift from diagnosis itself to which improvements should persist, how they generalize across tasks and embodiments, and how to prevent regressions from system updates. These challenges remain important even when the backbone becomes more capable.
The Save–Retrieve–Utilize operations represent the basic process of managing persistent experience, regardless of whether memory is external or integrated into the model. Similarly, the six capability categories provide an interpretable structure for locating and improving failures, but they are not intended to be fixed or exhaustive. They can be refined or replaced as models develop stronger internal capabilities.
Thus, the core idea of RoboFoundry remains the same: execution experience should lead to persistent, validated, and reusable system improvements, while the specific diagnosis and memory mechanisms can evolve with foundation models.
Reply to Reviewer 4SWF - Part 1
W1, W2, and Q1.1: Capability Diagnosis (Attribution) and Promotion Judgment (Verifier)
We thank the reviewer for these suggestions. We have added experiments to directly evaluate capability attribution and the reliability of the promotion verifier, as shown in Table 2 (Counterfactual attribution and verifier validation on EmbodiedBench).
1. Capability Attribution
To evaluate whether capability diagnosis improves repair, we add a counterfactual attribution experiment across all four EmbodiedBench suites using GPT-6 Astra, Qwen3.7-Plus, and GLM5.3-Flash.
For every initially failed task, we compare RoboFoundry's diagnosed capability with two randomly selected alternative capabilities (CT1 and CT2). The alternative conditions intentionally direct editing toward different capabilities, while keeping the same failure trace, pre-edit system, editing budget, and execution budget. This isolates the effect of capability-guided targeting rather than additional environment interaction.
The results show a clear difference: diagnosis-guided edits achieve approximately 3× higher repair success than edits targeting randomly selected alternative capabilities. This supports the importance of capability attribution and targeted editing rather than revising arbitrary components from failure traces.
2. Promotion Verifier
At the task level, RoboFoundry validates candidate repairs through actual environment re-execution. A repair is accepted only when it resolves the original failure.
For general-system promotion, the verifier uses recorded successful repairs from other tasks with the same diagnosed capability. For each held-out trace, the candidate general edit generates a new task-level repair, which is compared against the previously validated repair and its execution feedback. The system policy assigns a trace-based score rx, and Eq. (3) averages the score differences to compute Δcap. A candidate is accepted for promotion when Δcap≥0. We have clarified the scoring criteria in Section 2.3: “This score is based on the recorded successful repair and…”.
To test whether this judgment predicts actual execution outcomes, we have added a verifier ablation using the successful task-level repairs from the attribution experiment above. For each repair, we generate a candidate general edit and evaluate it using Eq. (3). We then execute both accepted and rejected candidates on the corresponding held-out tasks, comparing their performance against the pre-promotion system.
Statistical significance is assessed using a two-sided Fisher's test comparing positive-transfer rates between accepted and rejected candidates. Across the three backbones, candidates accepted by the verifier are approximately 3× more likely to achieve positive transfer in real held-out execution than rejected candidates (78.7% vs. 25.5%). This provides direct evidence that the promotion judgment is a useful proxy for actual transfer, rather than relying solely on model-generated scores.