acceptodds
Under review as a conference paper at ICLR 2027

SEAR: Benchmarking Self-Evolving Agentic Robotics

Abstract

Robots deployed in the real world need to improve from their own experience. We introduce SEAR (Self-Evolving Agentic Robotics), a framework and benchmark for robotic agents that get better with use, whether by rewriting their code, skills, and memories or by training a vision-language-action (VLA) model. Evaluating self-improvement is harder than evaluating a fixed policy, and SEAR addresses five specific problems. First, an agent that tries more often may look like it is learning when it is only sampling more, so we compare against best-of-N baselines with the same budget. Second, getting better at a practiced task is not the same as gen- eralizing, so we test on a five-level ladder: held-out initial states, unseen tasks, a second physics engine, and real hardware. Third, agents can exploit privileged simulator state, so success is scored by a judge the agent cannot access. Fourth, results can depend on the path an agent happens to take, so we run several inde- pendent copies of each agent from the same starting point. Fifth, code-writing and model-training agents spend very different resources, so we track compute along- side wall-clock time. We evaluate on five families of simulated tasks and five physical tasks. Inherited skills raise success from 61.9% to 72.3% on the hardest simulated family, yet inherited mistakes can persist across later tasks. A distilled VLA succeeds under camera faults that disable its own teacher. On hardware, the strongest simulated agent finishes last, and switching from frozen code to live ex- ecution reverses the ranking of the two models tested in both modes. SEAR is the first benchmark that evaluates self-evolving agents on robot tasks under rigorous controls, and we hope it gives future work on this direction a common playground and shapes the community’s understanding of this area.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Physical AI, robotics, self-evolving agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ujzhOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SEAR, a framework and benchmark for evaluating self-evolving robotic agents that improve through code revision, persistent skills and memories, or VLA fine-tuning. It combines code-as-policy (CaP), agent-as-policy (AaP), and auto-research (AR) settings with budget-matched controls, a five-tier evaluation framework, an isolated judge, and replicated agent lineages. Experiments across five simulated environment families and physical robot tasks examine the benefits and failures of experience inheritance, differences between code-based and learned policies, and limitations of current robotic self-improvement.

Critical strengths

The paper studies a focused but relevant benchmarking problem: distinguishing genuine improvement from repeated attempts, retained task-specific information, and other evaluation confounders. The proposed combination of inherited-library controls, held-out evaluation, and independent lineages is useful for studying robotic agents that improve through interaction. The experiments also yield informative findings beyond model rankings, particularly that inherited skills can propagate errors, that learning trajectories can diverge substantially, and that code-based policies and distilled VLAs have different robustness profiles.

Critical weaknesses

W1. Incomplete experimental validation of the benchmark's stated evaluation dimensions. Although SEAR aims to distinguish experience-driven improvement from repeated sampling and task-specific adaptation, the corresponding controls are not consistently applied across its evaluation settings. In T3/T4, agents may explore target scenes and rewrite their policies after the task instruction is disclosed, but no matched no-inheritance control is reported under the same exploration budget. Thus, the reported performance does not isolate the contribution of previously accumulated experience. The physical CaP/AaP comparison also uses different numbers of evaluation scenes (10 versus 5) and unequal execution times, limiting the interpretation of the reported ranking reversal. In addition, the paper proposes resource accounting as a central evaluation component but does not present a complete cross-paradigm ledger of token use, GPU-hours, and other costs. These gaps leave important claimed evaluation dimensions insufficiently validated.

W2. Evaluation inconsistencies weaken the reliability of key benchmark conclusions. Appendix G acknowledges that some agents accessed information from static task files, including exact object geometry and hidden goal descriptions, but the effect of these leaks on the main findings is not adequately quantified. In Table 2, the headline RoboLab TEST comparison uses lineage results collected before a tool-frame correction and no-library controls collected after it; the more closely matched validation comparison in Table 6 does not fully resolve the headline TEST mismatch. Moreover, the LIBERO-RIG CaP evaluation includes a control-frequency axis that was not actually applied by its harness, yet this axis is included in the reported eleven-axis macro score (Table 7 and Figure 3). These issues matter directly for a benchmark whose contribution rests on evaluation validity and consistent comparisons.

Questions for authors
  1. Can the authors provide a budget-matched no-inheritance baseline for T3/T4 to distinguish the effect of accumulated experience from target-scene exploration and the post-disclosure policy rewrite?

  2. How do the main RoboLab TEST comparisons change when all methods use the same corrected tool-frame configuration, and how robust are the conclusions after excluding lineages affected by privileged-information leakage?

  3. Why is the unapplied control-frequency axis included in the headline LIBERO-RIG CaP macro score, and can the authors report aggregate results over only the perturbations actually applied?

  4. Can the authors provide the complete resource ledger and a concrete release plan for the runnable benchmark, judge, and evaluation scripts?

5 comments

Thank you for the careful review. You identified concrete inconsistencies; the revised PDF corrects each of them.

W1 / Q1. Budget-matched no-inheritance baseline for T3/T4. We have this control, and it changes our claim: T3/T4 measure exploration, not inheritance. We ran Opus self-exploration lineages that differ from the donor-seeded runs only in starting with an empty library (one per family; Robosuite has no donor-seeded run). TEST % of the frozen exploration policy:

FamilyNo libraryDonor-seeded (paper; same frozen policy)
RoboLab (120 tasks)34.419.1–28.6 (four donors)
LIBERO43.226.9 ± 7.6
LIBERO-PRO10.314.6 ± 9.5
LIBERO-RIG23.024.3 ± 11.4

Starting empty matched (within one SD) or beat every donor; on LIBERO the donor library even hurts. We therefore remove the Section 4.1 sentence that T3/T4 scores track the donor library, redefine T3/T4 in Section 3.5 as exploration plus the rewrite once the instruction is revealed, add this table to Appendix A, and keep T1 as the inheritance result, where paired tests support it. Exploration may visit the evaluation scenes, so T3/T4 test an unknown instruction, not unseen scenes; Section 3.4 now says so. A RoboLab library swap shows that library content matters there: GPT-5.6-sol scores 60.6 with an Opus lineage's library and 46.9 without one (response to cYKS, Q5).

W2 / Q2. RoboLab TEST under a common tool frame; robustness to excluding leakage-affected lineages. The matched comparison holds, and none of the access paths changes it. All three arms use the same tool frame over the same 105 tasks: Opus 68.0 ± 3.5 (Lineage) vs. 61.9 ± 1.0 (Vanilla) vs. 57.1 (best-of-3); matched gain +6.1, paired 95% CI [+1.2, +11.3]. This TEST comparison replaces the validation comparison of the submitted Table 6, and the footnote that said some runs predated the tool-frame fix is corrected. On RoboLab, no reported lineage used hidden goals, the success predicate answers only after the episode ends, and no agent read a TEST layout. Asset files and the other families are covered in the response to cYKS (Q1).

W2 / Q3. Why is the unapplied control-frequency axis in the LIBERO-RIG macro? It should not have been; the axis is now removed from every LIBERO-RIG result. The code-as-policy harness runs the rig at a fixed 20 Hz, so it never applied that fault. Over the ten applied axes, LIBERO→RIG T2 changes from 68.7 to 66.0 ± 0.2, and LIBERO-PRO→RIG from 68.1 to 65.3 ± 1.9. The VLA results already excluded it.

W1 / Q4. Complete resource ledger and release plan. Both are in the revision. Appendix C has the ledger, the totals, and the VLA and hardware cost tables (response to cYKS, Q3). The conclusion states the release: SEAR under the MIT license, including the harness, the judge, LIBERO-RIG, and the reduced SEAR-Lite protocol (response to cYKS, Q1 and Q6).

W1. Physical CaP/AaP comparison. AaP did not draw easier scenes. We agree the comparison cannot support a causal claim, and the revision presents it as two recovery strategies, each from one lineage per model, and the abstract now says so. AaP's five scenes are a subset of CaP's ten, and CaP on those scenes scores slightly lower (Fable 64.0, GPT-6-astra 36.4). On the same scenes, GPT-6-astra's live mode beats its frozen code on 11 of 20 placement pairs and loses on none; Fable's two modes do not differ. Appendix L now states the episode times (median 12.8 vs. 3.8 minutes of robot time) and the Block Stacking stopping rules, which differ between modes.

We hope these changes address the concerns, and we would be glad to clarify anything further.

Thank you for the substantial revisions. Most of my concerns have been addressed.

However, Appendix G acknowledges that the main Lineage-versus-Vanilla results were obtained with access to static asset files during development, whereas the released benchmark will prohibit such access. It remains unclear whether the reported inheritance gains hold under the final protocol. I therefore maintain my rating.

Thank you for the follow-up. Our results do not rest on whether the asset files are readable. SEAR shows how self-evolution and test-time scaling work for LLM agents on robot tasks, and each comparison holds file access fixed across its arms. Below we explain why the comparison stands and why the library gain does not depend on reading these files.

Lineage and Vanilla are compared on common ground. Static scene and asset files were readable during development in every arm, alongside the same tasks, budget, and judge. The judge grades every policy on held-out TEST layouts that no agent read, and no graded RoboLab policy opens a scene or asset file when the judge runs it (0 of 592 graded Opus Lineage and Vanilla policies). Each lineage step is graded exactly like its Vanilla counterpart.

The library gain is not concentrated where the files are used. We scanned the three Opus RoboLab libraries. Some notes tell later tasks to read the scene file first, and a few skills store object poses for scenes that several tasks share; under this protocol, that is inherited experience like any other note. Few graded policies rely on it: 27 of 293 Lineage policies, on 23 of the 105 tasks, take hard-coded poses from a scene file (3 of 299 Vanilla). The Lineage − Vanilla gain is +7.5 points on those 23 tasks and +5.9 on the other 82.

On hardware, where no such files exist, the loop still works. A simulator always differs from the real world, in physics and in how scenes reset, and it always exposes some state a physical robot never has. The only strictly sealed setting is therefore the real one, which SEAR includes: a physical robot has no scene or asset file to read. Three code-as-policy lineages, one per model, start from an empty library, practice five tasks in order for up to 12 hours each, carry their workspace from task to task, and are graded as frozen policies on operator-staged scenes. Fable masters all four tasks that have a mastery rule in 17.3 hours of practice and scores 70.2% on the staged scenes (Section 4.5). Hardware also shows what simulation cannot: mastering a task in practice does not guarantee success on staged scenes, and switching from frozen code to live execution reverses the model ranking. The agent benchmarks closest to ours (MLE-bench, RE-Bench, PaperBench) grade only in software.

The release makes file access an explicit setting. The default, used in the paper, keeps the files readable. A strict setting removes them from the agent's sandbox, as on a physical robot. Every result is labeled with the setting it used.

Reviewer_jKMWOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

SEAR provides a framework for evaluating whether robotic agents improve through interaction, covering adaptation through code, skills, memory, and VLA training. The authors compare inherited-library, no-library, and time-budget-matched best-of-3 conditions across changes in initial states, perturbations, tasks, simulators, and physical hardware. The results suggest that inherited experience helps on some difficult tasks but can also propagate errors. Code policies, VLAs, and agents that remain active during execution exhibit different failure modes. The main contribution is the evaluation framework and its analysis of these failures.

Critical strengths

The paper addresses an important question in robotic self-improvement: does better performance come from accumulated experience, or from more attempts, longer development time, or easier practice conditions? The framework systematically separates these factors and offers a useful starting point for future evaluations.

The analysis of physical interaction is also valuable. Autonomous resets, lost objects, and incorrect success detectors can alter the subsequent practice process, while retained mistakes can repeatedly affect execution. These concrete observations show why practice success alone can be a misleading measure of progress.

Critical weaknesses

The information-isolation issues disclosed in the appendix are my main concern. Some runs accessed hidden goals and propagated them to later stages, and policies in one experimental arm could call a success predicate. The paper reports these cases, but it remains unclear which main-table results include affected runs and whether the conclusions hold after excluding them. For a benchmark that places substantial emphasis on task-blind exploration and protected evaluation, this mapping needs to be explicit.

Several key comparisons also combine runs from before and after the tool-frame correction, with differing rerun rules and task coverage. These differences do not establish that any particular result is inflated, but they make it difficult to determine how much of the reported inheritance benefit or model ranking comes from comparisons under common conditions. The current aggregates need a clearer separation of these settings.

Questions for authors

Which configurations were affected by access to hidden goals or success predicates, and are they included in the main tables? Do the main conclusions hold after excluding those runs or reevaluating them under the corrected isolation protocol?

Can the Lineage, Vanilla, and Best-of-3 results be reported under a common tool-frame version, task set, and rerun policy?

2 comments

Thank you for the positive assessment and the precise questions.

W1 / Q1. Which configurations were affected by hidden goals or the success predicate, and do the main conclusions hold without them? No reported configuration used hidden goals, and every T1 conclusion holds with the predicate blocked.

  • Hidden goals. In the submitted version some Kimi-k3 lineages quoted hidden goals, from a harness bug. We fixed the bug and reran those lineages, and no self-exploration donor library comes from the affected runs. The harness now refuses prompts and sessions that contain held-out task text.
  • Success predicate. All three T1 arms could call it. On RoboLab, which carries the main library result, it answers only after the episode ends, so it cannot change a score. On the LIBERO family and Robosuite it is live; regraded with the call returning False, scores fall by 1.0 point on average, and blocking it never reverses whether the library helps. The response to cYKS (Q1) gives the change per family and arm.
  • Held-out layouts. No agent read a TEST layout (all RoboLab session logs and graded policies checked), and the files are removed from agent containers.

Appendix G adds a table that maps each access path to the conditions it touched and its measured effect.

W2 / Q2. Common tool-frame version, task set, and rerun policy for Lineage, Vanilla, best-of-3? Yes. All three RoboLab arms use the same tool frame and the same 105 tasks, every task graded: Opus 68.0 ± 3.5 vs. 61.9 ± 1.0 vs. 57.1, a matched gain of +6.1 (paired 95% CI [+1.2, +11.3]). The submitted Table 2 footnote and Section 4.4 wrongly said some runs predated the tool-frame fix; what differed was coverage, since the submitted 72.3 was partly estimated before the lineage finished. LIBERO-RIG arms now share the same 18 task–fault cells with the fault hidden, and Robosuite best-of-3 uses one set of attempts. The paper states three exceptions: DeepSeek-v4-flash's RoboLab lineage covers the first 58 tasks (the model was retired), its LIBERO-RIG runs used DeepSeek-v4.1-flash, and Gemini's LIBERO-RIG best-of-3 covers the 9 cells its attempts delivered; every other model's covers all 18. The conclusions of Section 4.1 hold, stated more narrowly: the library helps where no-library runs fail (RoboLab for Opus and Kimi-k3; LIBERO-PRO and LIBERO-RIG for the weaker models), GPT-5.6-sol and Gemini do not gain on RoboLab, and the Opus gain is smaller.

We hope this resolves your concerns. We would welcome further questions.

Thank you. The access-path mapping and the matched RoboLab comparison resolve both of my questions. I will keep my positive score.

Reviewer_cYKSOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SEAR is a framework and benchmark for robot agents that improve through use, either by rewriting code, skills, and memory (code-as-policy, CaP), by staying in the control loop (agent-as-policy, AaP), or by fine-tuning π0.5\pi_{0.5} on rollouts from CaP teachers (auto-research, AR). It targets five evaluation confounds with five mechanisms:

Resampling vs. learning: budget-matched best-of-3 baselines; Practice vs. generalization: a five-tier ladder: new initial states, perturbed conditions (LIBERO-PRO, LIBERO-RIG), unseen tasks, a second simulator (RoboLab on Isaac), and real hardware; Privileged information: a sealed judge that grades frozen artifacts; Luck: three replicated lineages per configuration; Unequal resources: a compute ledger.

The benchmark covers five simulated families and five bimanual physical tasks, with five frontier coding models. Headline findings: an inherited library lifts Opus CaP from 61.9% to 72.3% on RoboLab; a distilled VLA survives camera faults that break its CaP teacher; identical lineages diverge more than models differ; and on hardware, the strongest simulated agent finishes last, while switching from frozen code (CaP) to live execution (AaP) reverses the ranking of the two models tested in both modes.

Critical strengths
  1. Very relevant and timely benchamrks to evaluate growing works on agentic robotics and also controlled study for fair comparisons: Pairing every learning curve with a budget-matched resampling baseline, grading frozen artifacts with a judge the agent cannot see, replicating lineages, and testing transfer up to a second physics engine and real hardware actually helps analyse agentic improvement vs improvements simply from re-sampling. This is now common in LLM-agent evaluation and the paper brings that to robotics. If adopted properly, it could make self-improving robotics systems comparable on a common eval setup and help the community understand and ablate methods and algorithms that actual drive improvements vs gains in score that are artifacts of weak and inconsistent evaluations.

  2. Detailed ablations and findings: I appreciate the authors' effort to investigate failure modes rather than relying only on aggregate performance metrics. Several findings are particularly interesting:

  • Errors introduced by the agents' own measurement tools can persist and propagate through inherited skill libraries.
  • Performance varies considerably across independent lineages, sometimes more than across different models.
  • Achieving a mastery streak during practice does not necessarily translate into good performance on held-out evaluation scenes.
  • The VLA student and its CaP teacher exhibit different failure modes, suggesting complementary strengths and weaknesses.

The analysis of shortcuts discovered by the agents (Appendix G) is also valuable, as it highlights practical challenges in evaluating autonomous agents that can inspect and modify their environments.

Critical weaknesses

1. The sealed-judge boundary is not fully enforced in practice: One of the paper's central evaluation principles is that agents should not have access to privileged information that could artificially inflate their performance. However, Appendix G documents several cases where this boundary was violated:

  • Opus inspected object asset files in 34 runs, obtaining exact object geometry unavailable to real robots.
  • Kimi-k3 accessed hidden goal descriptions, which subsequently propagated through its lineages.
  • In one experimental arm where the success predicate was accessible, 48% of final policies called it directly.

Additionally, Appendix B mentions that RoboLab validation feedback exposes layout descriptions that agents can replay locally, while Section 3.4 allows self-exploration on configurations that may later appear in evaluation.

Although the authors acknowledge these issues and flag affected lineages, it remains unclear how much they influence the reported results. I would like to see a quantitative analysis of the affected runs and a clearer explanation of how these leaks will be prevented. This is especially important given that separating genuine capability from privileged information is one of the benchmark's main goals. This is particularly important because preventing privileged information from being mistaken for genuine capability is one of the benchmark's main stated goals.

2. Several baseline comparisons are not fully matched, and the compute ledger is missing: The paper emphasizes budget-matched evaluation, but several comparisons use inconsistent experimental setups-

  • RoboLab: Vanilla and best-of-3 were run after a tool-frame bug fix, whereas the original Lineage runs used the earlier harness (Table 2). Only lineages that failed to discover the bug were rerun. Additionally, the reported Lineage score is partly reconstructed from Vanilla and a cell-matched per-step gain (Appendix D).
  • LIBERO-RIG: No-library baselines were informed about the injected faults, while library-based agents were not. Best-of-3 TEST results also cover only 12 of 18 cells.
  • Robosuite: Best-of-3 comparisons use different experiment exports, with a single successful attempt accounting for a 15.7-point difference (Appendix D).
  • Compute accounting: Despite being one of the five evaluation controls, a comprehensive compute ledger is missing beyond API costs for one hardware task. Budgets are matched primarily by wall-clock time, making CaP and AR difficult to compare in terms of actual resource usage.

Since evaluation rigor is the paper's primary contribution, I think the main comparisons should use a consistent harness version, identical evaluation cells and information access, and complete resource accounting.

3. Limited statistical power makes several conclusions difficult to support: Most simulation experiments use only three replicates, despite substantial variation across lineages. For instance, three LIBERO-PRO self-exploration lineages achieve 15.5%, 4.6%, and 23.6% success, a spread larger than several reported model differences.

For Opus, the library improvements on Robosuite, LIBERO, and LIBERO-PRO are within approximately one standard deviation of the baselines. On LIBERO, Vanilla even slightly outperforms Lineage (95.2% vs. 94.4%), making strong claims about the benefits of inheritance difficult to justify.

The hardware conclusions are similarly limited. Claims about the strongest simulated agent performing worst on hardware and model rankings reversing under AaP rely on just one lineage per model, with 10 evaluation scenes for CaP and 5 for AaP. Only two models are evaluated under AaP, with no Opus results.

These findings are interesting, but given the small sample sizes and documented lineage variability, I think they should be presented as preliminary case studies rather than general conclusions about model rankings or execution paradigms.

4. Reproducibility and the practical cost of adopting the benchmark: The benchmark relies heavily on proprietary frontier models and custom bimanual hardware, making the T5 evaluation difficult for other research groups to reproduce. The LIBERO-RIG companion report is also withheld during review, limiting the assessment of that component.

The evaluation itself is expensive, requiring up to 12 hours per hardware task per model and 40 hours per LIBERO-PRO axis for AR. These requirements could limit adoption by smaller research groups.

I would like the authors to clarify exactly what will be released and consider providing a smaller, affordable evaluation protocol alongside reference baselines using open-weight models.

Questions for authors
  1. Which reported results include lineages affected by information leakage (asset inspection, hidden goal descriptions, or accessible success predicates)? Could the authors report results excluding these lineages and clarify how these access paths will be blocked in the released benchmark?

  2. Could the authors rerun Lineage, Vanilla, and best-of-3 under the same corrected harness, using identical evaluation cells and fault disclosure? In particular, does the reported RoboLab improvement from 61.9% to 72.3% hold under these matched conditions?

  3. Could the authors provide the promised compute ledger, including token usage, API costs, and GPU-hours for each arm and task family? A cost-normalized comparison between CaP and AR would also be helpful.

  4. Is best-of-3 a fair resampling baseline when shorter attempts often fail to produce a valid policy (e.g., 90% of Kimi-k3's 40-minute RoboLab attempts)? Could the authors evaluate alternative budget splits, such as best-of-2, to assess how sensitive the learning-versus-resampling conclusions are to this choice?

  5. Given that the hardware results are based on a single lineage per model and only 5–10 evaluation scenes per task, could the authors provide additional replicates or uncertainty estimates? Which conclusions about model rankings and CaP versus AaP remain supported under these limitations?

  6. Could the authors clarify which components will be publicly released, including the judge, harness, LIBERO-RIG, policy artifacts, and execution logs? Will there be a reduced-budget protocol with open-weight model baselines to make the benchmark more accessible?

4 comments
AuthorsOpenReview ↗

Rebuttal 1/3

Thank you for the detailed and constructive review.

W1 / Q1. Which results include lineages affected by information leakage, and do results hold without them? How will the release block these paths? None of these access paths changes a reported conclusion. Agents use every opening a harness leaves, so we redid the audit on the final runs; the submitted Appendix G used early pilot counts and wrongly said the predicate was withheld. Path by path:

  • Success predicate: every T1 conclusion holds with it blocked. env.check_success() was callable in all three T1 arms. On RoboLab it answers only after the episode ends, so it cannot change a RoboLab score. On the other families it is live, so we regraded the saved policies with the call returning False and the judge scoring the real final state (table below). You are right that during development it is an unlimited success check on the agent's own rollouts. Every arm had it, it gives the same kind of signal as VAL feedback, and it reveals nothing about TEST states. Our regrade does not measure that use, and Section 3 now says so.
  • Held-out layouts: never read. The session logs of all three RoboLab arms show no read of a TEST layout, and none of the 25,392 graded policies references the TEST layouts or the seed that generates them. The files are deleted from every agent container. VAL layouts are shown to the agent by design; no arm selects on TEST.
  • Asset files: part of the protocol, never opened by a graded policy. Object and scene files are readable during development in every arm, so Lineage and Vanilla are compared on common ground, and agents can read them. No graded policy opens one when the judge runs it, including all 592 graded Opus RoboLab policies. A lineage can carry what it learned from these files to later tasks, as with any other experience. In the Opus RoboLab lineages, 27 of 293 graded policies, on 23 of the 105 tasks, take hard-coded poses from a scene file. The Lineage − Vanilla gain is +7.5 points on those 23 tasks and +5.9 on the other 82 tasks.
  • Hidden goals: no reported run. In the submitted version some Kimi-k3 lineages quoted hidden goals, from a harness bug. We fixed the bug and reran those lineages, and no self-exploration donor library comes from the affected runs. The harness now refuses prompts and sessions that contain held-out task text.
  • Scenes seen during self-exploration: success also rises on unseen ones, from 5.8% to 22.5% on twelve scenes no checkpoint explored (App. H; one lineage).

Predicate blocked inside the policy: change in TEST success (points), mean over the five models.

FamilyLineageVanillaBest-of-3
Robosuite−1.8−0.30.0
LIBERO−0.9−1.0−1.4
LIBERO-PRO−0.8−1.1−1.4
LIBERO-RIG−1.1−1.4−0.7

Blocking it never reverses whether the library helps, for any model or family, and each model's change is below one run-to-run SD in all but three cases. The only library gain that shrinks by more than a point (Gemini on Robosuite) comes from two policies, and no Robosuite library gain is significant either way. Appendix G maps every access path to its measured effect.

Release. The released judge blocks the predicate in the same way. The release also keeps held-out layouts and the TEST seed out of agent containers (the judge receives held-out layouts through one-time grants), closes the network for VLA arms, and refuses prompts with held-out task text.

W2 / Q2. Same harness, cells, and fault disclosure for Lineage, Vanilla, best-of-3; does 61.9 → 72.3 hold? No: under matched conditions the gain is +6.1, not +10.4, and it is significant over tasks. All three arms cover the same 105 tasks, every task graded: 68.0 ± 3.5 (Lineage) vs. 61.9 ± 1.0 (Vanilla) vs. 57.1 (best-of-3), paired 95% CI [+1.2, +11.3]. This interval takes the three lineages as given; across lineages (three per arm) Welch's test gives p ≈ 0.08, and Appendix D states both. What differed was coverage, not the harness: the submitted 72.3 was partly estimated before the lineage finished, and every RoboLab run used the same tool frame (the Table 2 footnote saying otherwise was wrong). On LIBERO-RIG, all arms now use the same 18 task–fault cells with the fault hidden (Opus 97.0 ± 2.7 vs. 96.7 ± 2.2 vs. 30.6).

AuthorsOpenReview ↗

Rebuttal 2/3

W2 / Q3. Compute ledger. Appendix C now has the ledger, and it shows that the library saves time. For every family, arm, and model it gives agent wall clock, tokens, API cost at list price, and GPU-hours. GPU-hours are lower bounds. Tokens and cost marked "e" are partly estimated: Opus's output tokens outside the corrected-rig RoboLab runs, most GPT-5.6-sol counts (extrapolated from its logged RoboLab sessions), and the few sessions whose logs were lost.

  • Time. Over Opus's whole curriculum, including building the library, Lineage used 902 agent-hours and about $22k, against 1,136 h and $29k for Vanilla. On the three LIBERO families the median Lineage attempt is shorter for all five models (Opus on LIBERO-PRO: 28 vs. 58 minutes). On RoboLab both arms use the full two hours, and the library shows up as higher success instead.
  • Totals. The project used about 4.3 agent-years, 3.81 of them measured; the rest is estimated from budgets for sessions whose logs were not kept. The T1 comparisons account for 1.35, all measured (Appendix C).

Code-as-policy (CaP) vs. auto-research (AR): the VLA costs more for equal or lower success, and is more robust. Opus drives both; AR's cost includes its CaP teacher (Appendix C).

FamilyCaP TESTAR TESTAR / CaP API costAR / CaP GPU-hours
LIBERO94.494.04.6×8.8×
LIBERO-PRO91.890.63.6×8.3×
RoboLab68.047.23.1×8.2×

Under LIBERO-RIG camera faults the VLA scores 57–90% against 7–11% for CaP (Section 4.3).

Hardware, at list API prices (Appendix C):

Hardware lineagePractice timeTokens (B)Practice costEvaluation cost
CaP Fable 5.129.5 h0.84$560none (frozen code)
CaP Opus 548.6 h1.66$1,141none
CaP GPT-6-astra60.2 h0.89$1,473none
AaP Fable 5.127.7 h0.98$548$158
AaP GPT-6-astra47.0 h1.38$1,697$615

Q4. Is best-of-3 a fair resampling baseline? Other splits? Yes, and one full session usually beats any split of it. Both kinds of learning are what we study: Lineage − Vanilla is what carries over between tasks through the library, and Vanilla − best-of-3 is what one long session gains over three short ones (Section 4.2). We tested other splits without new runs: we read each full Vanilla run at a k-th of its budget and keep the best of k. Across 20 model–family pairs, best-of-k is highest at k = 1 (Vanilla itself) in 16; the four exceptions win by 1–5 points. Short attempts often deliver nothing: at a third of the budget, only 1% of Kimi-k3's RoboLab attempts reach any VAL success. Appendix D has the comparison. Attempts that never started because of infrastructure failures (376 of 4,971) count as failed; the emulation does not depend on them.

W3 / Q5. Statistical power; hardware uncertainty. Simulation: the main gains are significant over tasks, and where they are not, the text no longer claims them. Paired tests (Appendix D) support the RoboLab gains for Opus and Kimi-k3, the LIBERO-PRO gains for GPT-5.6-sol, Gemini, and DeepSeek, and the LIBERO-RIG gains for Kimi-k3 and Gemini (+22.4, +46.6). For Opus on the other four families the intervals include zero; we agree with your LIBERO reading (94.4 vs. 95.2), and the text claims only a faster solution there. GPT-5.6-sol's RoboLab drop (−4.5) is not significant under the full bootstrap, and the text does not claim the library hurts. These tests take the lineages as given; with three per arm, a test across lineages has little power (Opus on RoboLab: p ≈ 0.08). The drop comes from what GPT-5.6-sol saved, not from inheriting: given the library an Opus lineage held before each task, it scores 60.6 against 46.9 with none (+13.7, [+8.1, +19.3]; Appendix E).

Lineage spread: a diagnosed cause, not noise. In the LIBERO-PRO example (15.5 / 4.6 / 23.6%, Section 4.4), the 4.6% lineage built its own success check; 91% of the successes it reported were failures, and because the check also decided when to stop, 637 rollouts ended without another grasp. The other two lineages had no such check. This is the failure the benchmark is built to expose: an agent's own error carried forward in memory.

AuthorsOpenReview ↗

Rebuttal 3/3

Hardware: three comparisons hold under scene uncertainty, one does not. With one lineage per model, Appendix L adds bootstrap intervals over evaluation scenes (overall score, 95% CI):

Fable 5.1GPT-6-astraOpus 5
CaP70.2 [59.8, 80.6]41.4 [29.2, 53.6]35.6 [26.0, 45.2]
AaP66.4 [55.2, 78.0]88.0 [79.6, 94.4]—
  • Fable leads code-as-policy (+28.8 over GPT-6-astra, CI [+12.6, +44.8]; +34.6 over Opus).
  • GPT-6-astra leads agent-as-policy (+21.6, [+7.6, +34.8]).
  • GPT-6-astra gains from acting live (+46.6, [+32.0, +60.6]).
  • Not supported: GPT-6-astra vs. Opus under CaP (+5.8, [−9.6, +21.6]).

These intervals cover scene sampling, not variation between lineages. We removed the claim that the strongest simulated agent finishes last on hardware (Fable 5.1 and GPT-6-astra were not run in simulation). AaP did not draw easier scenes: its five scenes are a subset of CaP's ten, and CaP on those five scores slightly lower (Fable 64.0, GPT-6-astra 36.4).

W4 / Q6. Release; reduced-budget protocol with open-weight baselines. Release. We will release SEAR soon under the MIT license, as the conclusion now states: the harness, the judge as a container, evaluation seeds (TEST seeds held server-side for a hosted leaderboard, a public set for local use), LIBERO-RIG, all saved policies and libraries, and the physical task specifications. The LIBERO-RIG paper is under double-blind review; Appendix P describes what we used, and we will cite it once that review ends.

Open-weight models. Kimi-k3 and DeepSeek-v4-flash are open-weight, and App. D reports their no-library baselines. Grading calls no model API, so saved policies can be regraded after an API model is retired, with the judge's perception models pinned.

Reduced protocol: one run per arm over the full curriculum reproduces the Lineage ranking. From existing runs, the full RoboLab curriculum with one run per arm (a third of the cost) reproduces the Lineage ranking in every resample and the sign of every significant library gain. Short prefixes of the curriculum do not recover the library effect, so measuring it needs a long curriculum; the revision states this as a finding. We propose SEAR-Lite = LIBERO-RIG (18 cells) + the full RoboLab curriculum, one Lineage and one Vanilla run: about 456 agent-hours per model, or $1.7k–$12.5k of API at our list prices for the models in the paper (Appendix C).

Cost of the expensive tiers. The 12 hours per hardware task were our practice budget; grading a saved policy takes a median of 3.8 minutes per episode, so a group can practice less and still grade on the same scenes. SEAR-Lite covers code-as-policy; the ledger gives the auto-research GPU-hours (about 3,600 on RoboLab vs. 443 for code-as-policy) so a group can decide whether to run that arm.

We hope these answers and the revised PDF address your concerns, and we are glad to provide more detail on any point.

I thank the authors for the effort in answering my questions, addressing the weaknesses. This does improve the state of the paper, and answers my questions. I would raise my score to 3, for the significant effort of the authors.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

We thank the reviewers for their careful reading. All three reviews ask whether the paper applies its own controls consistently. The revised PDF now does, and the main conclusions hold: the library helps where no-library runs fail (RoboLab for Opus and Kimi-k3; LIBERO-PRO and LIBERO-RIG for the weaker models), and inherited errors persist; an audit of everything the agents could reach changes none of this. Two things change: Opus's RoboLab gain is +6.1, not +10.4, and we withdraw the claim that T3/T4 measure inheritance.

1. Every main comparison is now matched. On RoboLab, Opus's three arms cover the same 105 tasks, every task graded: Lineage 68.0 ± 3.5, Vanilla 61.9 ± 1.0, best-of-3 57.1. The submitted 72.3 was partly estimated before the lineage finished. Under the same conditions Kimi-k3 gains +21.8 and GPT-5.6-sol is 4.5 points lower. On LIBERO-RIG, the no-library control is rerun without being told the injected fault, best-of-3 is regraded on all 18 task–fault cells (9 for Gemini), and the control-frequency axis, which the harness never applied, is removed.

2. The gain is significant over tasks. All arms are graded on the same tasks, so we report paired 95% CIs (bootstrap over tasks, then runs):

  • RoboLab: Opus +6.1 [+1.2, +11.3]; Kimi-k3 +21.8 [+14.4, +29.4]. GPT-5.6-sol (−4.5) and Gemini (−1.1) do not gain there; neither change is significant.
  • LIBERO-PRO: GPT-5.6-sol +5.3, Gemini +12.6, DeepSeek +18.0, all excluding zero.
  • On the easier families Opus's gain is not distinguishable from zero, and the text no longer claims one.
  • These intervals take the three lineages as given. A test across lineages, with three per arm, has little power: for Opus on RoboLab, Welch's test gives p ≈ 0.08. Appendix D states both.

3. The access audit changes no conclusion. Agents use every opening a harness leaves (Appendix G), so we audited each one on the final runs. Blocking the success predicate inside the policy lowers scores by 1.0 point on average and never reverses whether the library helps; on RoboLab it cannot change a score at all. No agent read a held-out TEST layout, and the Kimi-k3 lineages that had seen held-out task text were rerun. The release closes every path we found (response to cYKS, Q1).

4. The library saves time. Over Opus's whole curriculum, including the cost of building the library, Lineage used 21% less agent time than Vanilla. Appendix C adds the ledger, the project totals, and the cost of the VLA and hardware runs (response to cYKS, Q3).

5. A cheaper protocol and the release. One run per arm over the full RoboLab curriculum, a third of the cost, reproduces the Lineage ranking; we build SEAR-Lite on it. We will release SEAR soon under the MIT license; both are now in the PDF (response to cYKS, Q6).

6. Claims narrowed.

  • T3/T4. Lineages that start with an empty library score as well as or better than donor-seeded ones, so T3/T4 measure exploration and the rewrite, not inherited experience. We withdraw that claim; the inheritance result we keep is T1 (response to ujzh).
  • Hardware. We add scene-level intervals, keep only the comparisons they support, and the abstract now says one lineage per model.
AuthorsOpenReview ↗

What changed in the revised PDF

Every change is marked in blue in the revised PDF.

New runs, finished after the deadline.

  • RoboLab. The matched Lineage, Vanilla, and best-of-3 runs over all 105 tasks. Opus Lineage is now 68.0 ± 3.5 (submitted: 72.3); the abstract, introduction, conclusion, figures, and tables use them.
  • LIBERO-RIG. The control without the fault disclosed, the best-of-3 regrade, and the regrade of the wrist-camera-mount fault.

Corrections.

  • The LIBERO-RIG control-frequency axis is removed.
  • The abstract and Section 3 say the judge is sealed for scoring and that the success predicate was callable during development. The abstract's ladder now lists all five levels.
  • Section 4.1: Opus matches its no-library control within noise on the easier families (it said "slightly improve"), and the LIBERO-RIG time-to-ceiling claim is removed, since the two arms were graded at different intervals.
  • Section 4.5: Fable's four masteries took 17.3 hours, not 29.5 (its total practice).
  • Figure 1 was drawn before the matched runs; its caption points to the updated endpoints.
  • The Table 2 footnote and Section 4.4 no longer say that some RoboLab runs predate the tool-frame fix; none does.
  • Section 4.1 no longer says that T3/T4 scores track the donor library.
  • The abstract no longer says that the strongest simulated agent finishes last on hardware.
  • The shortcuts appendix, the sentence on Lin et al. (2026), one Robosuite VLA entry, and two swapped SDs are fixed.

Added. Appendix A: the empty-library control. Appendix C: the compute and cost ledger, project totals, the VLA and hardware cost tables, and SEAR-Lite. Appendix D: paired tests and other budget splits. Appendix E: a RoboLab library swap. Appendix G: the access-path table, the predicate regrade, and the hidden-goal reruns. Appendix L: hardware intervals, the matched-scene comparison, episode times, and stopping rules.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.