- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SEAR, a framework and benchmark for evaluating self-evolving robotic agents that improve through code revision, persistent skills and memories, or VLA fine-tuning. It combines code-as-policy (CaP), agent-as-policy (AaP), and auto-research (AR) settings with budget-matched controls, a five-tier evaluation framework, an isolated judge, and replicated agent lineages. Experiments across five simulated environment families and physical robot tasks examine the benefits and failures of experience inheritance, differences between code-based and learned policies, and limitations of current robotic self-improvement.
Critical strengths
The paper studies a focused but relevant benchmarking problem: distinguishing genuine improvement from repeated attempts, retained task-specific information, and other evaluation confounders. The proposed combination of inherited-library controls, held-out evaluation, and independent lineages is useful for studying robotic agents that improve through interaction. The experiments also yield informative findings beyond model rankings, particularly that inherited skills can propagate errors, that learning trajectories can diverge substantially, and that code-based policies and distilled VLAs have different robustness profiles.
Critical weaknesses
W1. Incomplete experimental validation of the benchmark's stated evaluation dimensions. Although SEAR aims to distinguish experience-driven improvement from repeated sampling and task-specific adaptation, the corresponding controls are not consistently applied across its evaluation settings. In T3/T4, agents may explore target scenes and rewrite their policies after the task instruction is disclosed, but no matched no-inheritance control is reported under the same exploration budget. Thus, the reported performance does not isolate the contribution of previously accumulated experience. The physical CaP/AaP comparison also uses different numbers of evaluation scenes (10 versus 5) and unequal execution times, limiting the interpretation of the reported ranking reversal. In addition, the paper proposes resource accounting as a central evaluation component but does not present a complete cross-paradigm ledger of token use, GPU-hours, and other costs. These gaps leave important claimed evaluation dimensions insufficiently validated.
W2. Evaluation inconsistencies weaken the reliability of key benchmark conclusions. Appendix G acknowledges that some agents accessed information from static task files, including exact object geometry and hidden goal descriptions, but the effect of these leaks on the main findings is not adequately quantified. In Table 2, the headline RoboLab TEST comparison uses lineage results collected before a tool-frame correction and no-library controls collected after it; the more closely matched validation comparison in Table 6 does not fully resolve the headline TEST mismatch. Moreover, the LIBERO-RIG CaP evaluation includes a control-frequency axis that was not actually applied by its harness, yet this axis is included in the reported eleven-axis macro score (Table 7 and Figure 3). These issues matter directly for a benchmark whose contribution rests on evaluation validity and consistent comparisons.
Questions for authors
-
Can the authors provide a budget-matched no-inheritance baseline for T3/T4 to distinguish the effect of accumulated experience from target-scene exploration and the post-disclosure policy rewrite?
-
How do the main RoboLab TEST comparisons change when all methods use the same corrected tool-frame configuration, and how robust are the conclusions after excluding lineages affected by privileged-information leakage?
-
Why is the unapplied control-frequency axis included in the headline LIBERO-RIG CaP macro score, and can the authors report aggregate results over only the perturbations actually applied?
-
Can the authors provide the complete resource ledger and a concrete release plan for the runnable benchmark, judge, and evaluation scripts?
5 comments
Thank you for the substantial revisions. Most of my concerns have been addressed.
However, Appendix G acknowledges that the main Lineage-versus-Vanilla results were obtained with access to static asset files during development, whereas the released benchmark will prohibit such access. It remains unclear whether the reported inheritance gains hold under the final protocol. I therefore maintain my rating.
Thank you for the follow-up. Our results do not rest on whether the asset files are readable. SEAR shows how self-evolution and test-time scaling work for LLM agents on robot tasks, and each comparison holds file access fixed across its arms. Below we explain why the comparison stands and why the library gain does not depend on reading these files.
Lineage and Vanilla are compared on common ground. Static scene and asset files were readable during development in every arm, alongside the same tasks, budget, and judge. The judge grades every policy on held-out TEST layouts that no agent read, and no graded RoboLab policy opens a scene or asset file when the judge runs it (0 of 592 graded Opus Lineage and Vanilla policies). Each lineage step is graded exactly like its Vanilla counterpart.
The library gain is not concentrated where the files are used. We scanned the three Opus RoboLab libraries. Some notes tell later tasks to read the scene file first, and a few skills store object poses for scenes that several tasks share; under this protocol, that is inherited experience like any other note. Few graded policies rely on it: 27 of 293 Lineage policies, on 23 of the 105 tasks, take hard-coded poses from a scene file (3 of 299 Vanilla). The Lineage − Vanilla gain is +7.5 points on those 23 tasks and +5.9 on the other 82.
On hardware, where no such files exist, the loop still works. A simulator always differs from the real world, in physics and in how scenes reset, and it always exposes some state a physical robot never has. The only strictly sealed setting is therefore the real one, which SEAR includes: a physical robot has no scene or asset file to read. Three code-as-policy lineages, one per model, start from an empty library, practice five tasks in order for up to 12 hours each, carry their workspace from task to task, and are graded as frozen policies on operator-staged scenes. Fable masters all four tasks that have a mastery rule in 17.3 hours of practice and scores 70.2% on the staged scenes (Section 4.5). Hardware also shows what simulation cannot: mastering a task in practice does not guarantee success on staged scenes, and switching from frozen code to live execution reverses the model ranking. The agent benchmarks closest to ours (MLE-bench, RE-Bench, PaperBench) grade only in software.
The release makes file access an explicit setting. The default, used in the paper, keeps the files readable. A strict setting removes them from the agent's sandbox, as on a physical robot. Every result is labeled with the setting it used.
Rebuttal
Thank you for the careful review. You identified concrete inconsistencies; the revised PDF corrects each of them.
W1 / Q1. Budget-matched no-inheritance baseline for T3/T4. We have this control, and it changes our claim: T3/T4 measure exploration, not inheritance. We ran Opus self-exploration lineages that differ from the donor-seeded runs only in starting with an empty library (one per family; Robosuite has no donor-seeded run). TEST % of the frozen exploration policy:
Starting empty matched (within one SD) or beat every donor; on LIBERO the donor library even hurts. We therefore remove the Section 4.1 sentence that T3/T4 scores track the donor library, redefine T3/T4 in Section 3.5 as exploration plus the rewrite once the instruction is revealed, add this table to Appendix A, and keep T1 as the inheritance result, where paired tests support it. Exploration may visit the evaluation scenes, so T3/T4 test an unknown instruction, not unseen scenes; Section 3.4 now says so. A RoboLab library swap shows that library content matters there: GPT-5.6-sol scores 60.6 with an Opus lineage's library and 46.9 without one (response to cYKS, Q5).
W2 / Q2. RoboLab TEST under a common tool frame; robustness to excluding leakage-affected lineages. The matched comparison holds, and none of the access paths changes it. All three arms use the same tool frame over the same 105 tasks: Opus 68.0 ± 3.5 (Lineage) vs. 61.9 ± 1.0 (Vanilla) vs. 57.1 (best-of-3); matched gain +6.1, paired 95% CI [+1.2, +11.3]. This TEST comparison replaces the validation comparison of the submitted Table 6, and the footnote that said some runs predated the tool-frame fix is corrected. On RoboLab, no reported lineage used hidden goals, the success predicate answers only after the episode ends, and no agent read a TEST layout. Asset files and the other families are covered in the response to cYKS (Q1).
W2 / Q3. Why is the unapplied control-frequency axis in the LIBERO-RIG macro? It should not have been; the axis is now removed from every LIBERO-RIG result. The code-as-policy harness runs the rig at a fixed 20 Hz, so it never applied that fault. Over the ten applied axes, LIBERO→RIG T2 changes from 68.7 to 66.0 ± 0.2, and LIBERO-PRO→RIG from 68.1 to 65.3 ± 1.9. The VLA results already excluded it.
W1 / Q4. Complete resource ledger and release plan. Both are in the revision. Appendix C has the ledger, the totals, and the VLA and hardware cost tables (response to cYKS, Q3). The conclusion states the release: SEAR under the MIT license, including the harness, the judge, LIBERO-RIG, and the reduced SEAR-Lite protocol (response to cYKS, Q1 and Q6).
W1. Physical CaP/AaP comparison. AaP did not draw easier scenes. We agree the comparison cannot support a causal claim, and the revision presents it as two recovery strategies, each from one lineage per model, and the abstract now says so. AaP's five scenes are a subset of CaP's ten, and CaP on those scenes scores slightly lower (Fable 64.0, GPT-6-astra 36.4). On the same scenes, GPT-6-astra's live mode beats its frozen code on 11 of 20 placement pairs and loses on none; Fable's two modes do not differ. Appendix L now states the episode times (median 12.8 vs. 3.8 minutes of robot time) and the Block Stacking stopping rules, which differ between modes.
We hope these changes address the concerns, and we would be glad to clarify anything further.