acceptodds
Under review as a conference paper at ICLR 2027

In-MANU: Probing the Capabilities and Self-Improvement of Frontier Agents in Robotic Manipulation

Abstract

Frontier multimodal models can act as robot policies without robot-specific training: they interpret camera images and task instructions and issue end-effector commands directly. The extent of this capability relative to post-trained embodied foundation models, and the extent to which such an agent can improve it from its own experience, have not been characterized. We present In-MANU, a study of frontier agents in robotic manipulation that evaluates GPT-6-Astra on the 26 test tasks of the EBench mobile manipulation benchmark and compares it with seven post-trained vision–language–action and world-action policies. With a single annotated demonstration per task, the agent ranks second among the eight systems. It attains the highest success rate on short-horizon tasks and exhibits exploration, recovery from disrupted subgoals and correction after failed attempts within an episode, but it ranks below most post-trained policies on long-horizon and tabletop dexterous-and-precise tasks, which require precise alignment, physical contact and bimanual coordination. We further examine two means of reducing this gap. Access to a post-trained policy as a callable tool increases success only on tasks that the policy solves on its own, and the agent's delegation does not correspond to the competence of the policy. A self-improvement loop, in which a critic instantiated from the same model revises the agent's playbook and harness on the basis of execution feedback, alters the agent's strategy without a statistically significant improvement in test success, both for GPT-6-Astra and, in a smaller replication, for Claude Opus 5.5; the critic's episode-level motion-error reports coincide with measured deviations, but its aggregate account attributes all failure modes to execution.These findings suggest that physical execution reliability remains a major bottleneck for frontier agents under the evaluated control interface.

Primary area:
applications to robotics, autonomy, planning
Keywords:
embodied agents, robotic manipulation, agent self-improvement

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nsmAOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper do a systematic evaluation of GPT-6-Astra as a frozen, direct-action policy for robotic manipulation, comparing it with seven post-trained VLA and WAM policies across 26 EBench tasks. The authors also investigate policy delegation and non-parametric self-improvement to address its limitations. The results show competitive performance on short-horizon tasks but weaknesses in long-horizon and contact-rich manipulation, while neither improvement strategy yields significant gains. The study provides valuable insights into multimodal agents for robotic control, although some concerns remain regarding novelty, experimental fairness, and statistical significance.

Critical strengths
  • The paper provides a comprehensive and fine-grained empirical analysis of frontier agents in robotic manipulation.
Critical weaknesses
  • The novelty of the main findings is somewhat limited, as many of the observations are already well known in the research community.
  • The evaluation is limited to a single simulation benchmark (EBench) and one robot configuration, without real-world experiments. Although EBench covers diverse manipulation tasks, it remains unclear whether the findings generalize to different robotic platforms and more complex real-world scenarios.
Questions for authors
  • What is the expected mechanism through which self-improvement can enhance precise manipulation when the model weights and low-level controllers remain frozen? Is the proposed framework fundamentally limited to improving high-level strategies rather than acquiring new manipulation capabilities?
  • Could the authors evaluate joint-space control instead of end-effector-space control? The choice of action space may also significantly affect performance.
  • Although EBench contains diverse tasks, could the authors clarify which conclusions are expected to generalize beyond this specific simulator and robot configuration?
  • More models should be evaluated. Models such as GPT-6 Astra and Claude Opus 5.5 may have been trained with robotics-related data, which could affect the results and may not fully support the paper's claims.
2 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank the reviewer for the precise questions about novelty, mechanism and scope. In response, the revision adds one page on what the evidence implies beyond the expected contrast between task reasoning and contact precision: Section 5.6 locates the capability boundary, and three highlighted insights state the implications for delegation and self-improvement. SR is averaged within each task and then equally across tasks. Statistical comparisons pair outcomes by scene.

W1: The novelty of the main findings is somewhat limited, as many of the observations are already well known in the research community.

We agree that a frontier agent reasoning better about tasks than it handles precise contact is expected. Prior studies of frontier models as robot policies mainly establish control interfaces, whereas this study measures the agent against seven post-trained policies on the same scenes. The revision states three findings from this comparison that go beyond the expected contrast.

  • The deficit is located. In 33 of 35 unsuccessful nut and gear episodes, the end effector holding the part reaches its commanded seating height before release. A second model under the same interface, Qwen3.8-Flash, misses its targets about as often as GPT-6-Astra, on 6.6% and 5.2% of arm waypoints, but completes no subgoal in 35 of its 47 failures, against 7 of 27. Task progress therefore depends on the model, whereas final alignment fails after an accurate reach, which calls for perception and primitives that close the loop at contact.
  • Learned execution helps by stage, not uniformly. π0.5 as a tool raises SR significantly from 46.7% to 52.4% on all 510 scenes. Yet the agent also misses 59 scenes that direct control solves, and on the test tasks its delegation shows no correlation with the policy's standalone success, at r = 0.07. A policy offered as a tool should therefore report its competence and the outcome of each segment.
  • Self-improvement is bounded by its records. With frozen weights, it revises what its records make visible but not contact precision, as our answer to Q1 details.

Q1: What is the expected mechanism through which self-improvement can enhance precise manipulation when the model weights and low-level controllers remain frozen? Is the proposed framework fundamentally limited to improving high-level strategies rather than acquiring new manipulation capabilities?

The framework is not limited in principle, but the five rounds improved strategies only, because the execution records contained no signal of contact precision. With frozen weights and controllers, harness revisions can still improve precision in two ways: by adding executable primitives, such as a closed-loop insertion skill or a learned policy segment, and by adding feedback channels, such as contact or clearance estimates from the cameras. Over the five rounds, 20 of 22 accepted revisions were playbook rules, and they changed decisions. After a rule on completing enclosures, the agent plans to close the dishwasher in all five dishwasher test episodes instead of one. The final version also ends 3 of 90 test episodes early instead of 10. No accepted revision specified a tolerance or added a contact primitive, because millimeter misalignment raises no error in the records. A new channel must also be acted upon: with an added waypoint residual, the agent corrected 1 of 30 large misses, against 5 of 21 without it. Section 6.2 now states this bound as Insight 3, and the conclusion names contact perception and validated closed-loop tools as the route to new manipulation capabilities.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

Q2: Could the authors evaluate joint-space control instead of end-effector-space control? The choice of action space may also significantly affect performance.

We did not evaluate a frontier agent that issues joint targets, and we agree that the action space can matter. The diagnostics suggest it is not the main factor in the high-precision failures, because these occur after the commanded pose is reached: on the nut and gear tasks, 33 of the 35 failures release the part with the end effector within 4 mm of its commanded seating height. A different parameterization of the reach is therefore unlikely to change this final alignment unless it also supplies contact feedback. The π0.5 tool outputs joint targets from fresh observations and raises SR on long-horizon and tabletop tasks, but not on the high-precision tasks, where SR is 11.3% without it and 7.5% with it. Because it changes the executor as well as the action space, this result does not isolate joint-space control. Appendix D lists a joint-command ablation as unevaluated, and the paper characterizes each model together with its position-waypoint interface.

W2: The evaluation is limited to a single simulation benchmark (EBench) and one robot configuration, without real-world experiments. Although EBench covers diverse manipulation tasks, it remains unclear whether the findings generalize to different robotic platforms and more complex real-world scenarios.

Q3: Although EBench contains diverse tasks, could the authors clarify which conclusions are expected to generalize beyond this specific simulator and robot configuration?

We expect the mechanisms to generalize, not the success rates or rankings. Three findings concern the coupling of a reasoning model with its executors rather than EBench itself. First, an agent that observes only between open-loop motions fails where contact needs feedback within a motion. Second, a learned policy helps only on the stages the agent assigns it well, which calls for tool interfaces that report the policy's competence and results. Third, a self-improvement loop learns what its records measure. The revision states both points: the conclusion presents these mechanisms as testable hypotheses for other embodiments, and the scope paragraph states that the success rates and rankings hold only for this simulator and robot.

Q4: More models should be evaluated. Models such as GPT-6 Astra and Claude Opus 5.5 may have been trained with robotics-related data, which could affect the results and may not fully support the paper's claims.

The revision adds Qwen3.8-Flash with the same interface on two scenes of each of the 26 tasks. It completes 5 of 52 scenes against 25 for GPT-6-Astra. Claude Opus 5.5 is evaluated in the self-improvement study, with π0.5 available as a tool, on 90 paired episodes. The models differ in task progress rather than reach accuracy, so the model's contribution, including any robotics-related priors, appears in how far a task progresses. We cannot rule out robotics-related pretraining data for proprietary models, so the paper defines "no robot-specific training" operationally: we perform no robot-specific parameter updates. The claims therefore concern the deployed models with their existing priors and the stated inference inputs.

Reviewer_FPNyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies whether a frontier multimodal model can act directly as a robot manipulation policy and improve from its own experience. GPT-6-Astra controls a simulated two-arm mobile robot through end-effector waypoints chosen from camera images and robot state, with one annotated demonstration per task, and is compared with seven post-trained vision-language-action and world-action policies on the 26 test tasks of EBench. It ranks second overall (46.7% success) and first on short-horizon tasks, but falls behind on long-horizon and precise tabletop tasks, which the authors trace, from recorded episodes, to unreliable physical execution. Two ways to narrow this gap, calling π0.5 as a tool and a self-improvement loop in which a critic from the same model revises the agent’s playbook and harness, give no significant gain, including in a smaller replication of the loop with Claude Opus 5.5.

Critical strengths
  • Timely and carefully executed. Putting a frontier model used directly as a policy next to post-trained VLAs and WAMs on a shared benchmark, split by horizon, precision and mobility, is useful, and the call-by-call case analyses (App. D, F.6) make the failure modes concrete.
  • Transparent about the setup: prompts and tool schemas (App. A), protocol changes and author interventions during training (App. E.2), token budgets, and a promise to release logs, prompts and every playbook-harness version.
Critical weaknesses
  • Limited scope and novelty. Only one model, GPT-6-Astra, is tested in the main evaluation; it is available only through OpenAI’s service, and the paper gives neither the exact model version nor the dates of the runs, so the results may not be reproducible. All experiments use one simulator and one robot, and the loop combines existing frameworks (RPent, ENPIRE). The main finding, good task interpretation but weak precise contact, is not very surprising, so the contribution rests mostly on the analysis.
  • Model limits and interface limits are not separated. Actions run open loop between observations, 84% of commands are a single waypoint (l. 252), and 3.1% of waypoints in the Section 6 training runs miss by more than 5 cm without any error (Table 15). The paper concludes that the agent’s “deficits lie in execution” (l. 1059), but without an interface ablation (e.g., reachability feedback or a closed-loop grasp or insertion primitive) it is unclear how much belongs to the model and how much to this harness.
  • The comparison is not matched in conditions or cost. The agent has uncapped tokens, tool calls and thinking time while the simulation waits (l. 195-196), gets hand-written manipulation guidance in its prompt (l. 826-830) and a demonstration with recorded states and actions, and uses about 1.7-1.9M tokens per Section 6 episode (App. E.2). No wall-clock time or cost is reported for the evaluated models on leaderboard.
Questions for authors
  • Sharing videos of randomly chosen episodes, not only the selected keyframes would be helpful.
  • What were the wall-clock time and API cost per episode, and on which tasks was the actor prompt developed?
  • Why was Claude Opus 5.5 not run in the main 26-task evaluation, and do the capability findings hold for it?
2 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank the reviewer for the detailed review and for noting the transparency of the setup. The central concern, that model limits and interface limits are not separated, shaped the revision: Section 5.6 now locates the capability boundary with a same-interface comparison across models, and the paper characterizes each model together with its interface. SR is averaged within each task and then equally across tasks. Statistical comparisons pair outcomes by scene.

W2: Model limits and interface limits are not separated. Actions run open loop between observations, 84% of commands are a single waypoint (l. 252), and 3.1% of waypoints in the Section 6 training runs miss by more than 5 cm without any error (Table 15). The paper concludes that the agent’s “deficits lie in execution” (l. 1059), but without an interface ablation (e.g., reachability feedback or a closed-loop grasp or insertion primitive) it is unclear how much belongs to the model and how much to this harness.

We agree, and the revision no longer attributes the execution failures to the model alone. Both suggested ablations are partly available, and a second model separates the model's share.

  • Reachability feedback. From round 1 of the loop, the harness reports the residual of every waypoint, yet the final version corrects 1 of 30 large misses on the test tasks, against 5 of 21 at the starting point without this report. The final version also differs in its playbook, so this comparison does not isolate the report.
  • A closed-loop learned primitive. With π0.5 as a tool on all 510 scenes, SR rises significantly from 46.7% to 52.4%, mainly on long-horizon and tabletop tasks, but high-precision SR is 11.3% without and 7.5% with it.
  • A second model. Qwen3.8-Flash with the same interface misses its targets about as often as GPT-6-Astra, on 6.6% and 5.2% of arm waypoints, but completes no subgoal in 35 of its 47 failures, against 7 of 27.

The diagnostics then locate where GPT-6-Astra fails: only 4 of 51 unsuccessful peg, nut and gear episodes contain a waypoint miss above 5 cm, and in 33 of 35 unsuccessful nut and gear episodes the end effector holding the part reaches its commanded seating height before release. Task progress therefore depends on the model, whereas final alignment fails after an accurate reach, under an interface that returns feedback only after each motion. Section 5.6 states this decomposition, and Appendix D names a purpose-built contact primitive as the ablation that remains open.

W3: The comparison is not matched in conditions or cost. The agent has uncapped tokens, tool calls and thinking time while the simulation waits (l. 195-196), gets hand-written manipulation guidance in its prompt (l. 826-830) and a demonstration with recorded states and actions, and uses about 1.7-1.9M tokens per Section 6 episode (App. E.2). No wall-clock time or cost is reported for the evaluated models on leaderboard.

Q2: What were the wall-clock time and API cost per episode, and on which tasks was the actor prompt developed?

We agree that the comparison is not cost-matched, and the revision now reports time and cost in the setup and in Appendix E. A main-run episode takes a median 38.0 minutes, including networking, simulation and recording, and uses 3.77M tokens on average, 91.5% of them cached, an estimated $7.8 at list price and $3,988 for all 510 episodes. Cost concentrates where execution fails: unsuccessful episodes use about three times the tokens of successful ones, so a success costs $5.11 on short-horizon and $41.64 on long-horizon tasks. When π0.5 serves as a tool, the median episode shortens to 25.8 minutes, which suggests that more reliable execution also lowers cost. The task-independent guidance was developed on a separate desk setup before the main evaluation. Pipeline, budget and demonstration pilots touched 32 test scenes, and excluding them gives 46.2% SR against 46.7%. The policies, in turn, train on 6,600 episodes of the same tasks, whereas the agent receives one annotated demonstration per task.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

W1: Limited scope and novelty. Only one model, GPT-6-Astra, is tested in the main evaluation; it is available only through OpenAI’s service, and the paper gives neither the exact model version nor the dates of the runs, so the results may not be reproducible. All experiments use one simulator and one robot, and the loop combines existing frameworks (RPent, ENPIRE). The main finding, good task interpretation but weak precise contact, is not very surprising, so the contribution rests mostly on the analysis.

We agree that good task interpretation combined with weak precise contact is not surprising, and that the contribution therefore rests on the analysis. The revision states three findings of the analysis that this result alone does not predict.

  • The deficit is located. Insertion fails after an accurate reach, and a second model under the same interface reaches as accurately but progresses far less. The harness therefore needs perception and primitives that close the loop at contact.
  • Learned execution helps by stage, not uniformly. With π0.5 as a tool, the agent solves 88 scenes that direct control misses but misses 59 that direct control solves, and on the test tasks its delegation shows no correlation with the policy's standalone success, at r = 0.07. A policy offered as a tool should therefore report its competence and the outcome of each segment.
  • Self-improvement is bounded by its records. It revises what they make visible, such as task stages, stopping and delegation, but not contact precision, which raises no error signal. Corrections that pass held-out validation should therefore be retained as closed-loop tools rather than playbook rules.

Section 5.6 and three highlighted insights now state these findings, and the conclusion turns each into a design direction for embodied agents, together with a safety layer below the agent. For reproducibility, the recorded identifier is gpt-6-astra at high reasoning effort. The main run spans 14 to 17 September 2026, the test evaluations of the loop versions ran on 24 and 25 September, and the remaining 420 episodes with the π0.5 tool on 8 October. Because this service alias does not pin the weights, we will release the logs, prompts and every playbook and harness version. The loop reuses RPent and ENPIRE components as an instrument for the paired evaluation.

Q3: Why was Claude Opus 5.5 not run in the main 26-task evaluation, and do the capability findings hold for it?

A full 26-task evaluation of Opus is not needed for the paper's claims, because a paired comparison on shared scenes already shows that its capability profile matches that of GPT-6-Astra. The 26-task evaluation profiles GPT-6-Astra against the post-trained policies, and Opus tests whether this profile depends on the model. On the 90 test episodes that the two models share, both starting with π0.5 available as a tool, Opus reaches a similar SR, 40.0% against 44.4%, and the same task-group profile: 62.9% on short-horizon, 25.5% on long-horizon and 15.0% on high-precision tasks, against 71.4%, 27.3% and 10.0%. The two models differ in delegation: across tasks, how much Opus delegates follows how often π0.5 alone succeeds, at r = 0.60, against 0.07 for GPT-6-Astra. The strength on short-horizon tasks and the weakness at high precision are therefore not specific to GPT-6-Astra, whereas the use of a learned policy depends on the model.

Q1: Sharing videos of randomly chosen episodes, not only the selected keyframes would be helpful.

We agree that selected keyframes can misrepresent typical behavior. The revised supplementary material contains the videos of 52 randomly selected episodes, two per task, with a manifest of their outcomes. As Appendix G reports, these outcomes are close to those of the full run: 24 successes, 18 partial completions and 10 zero Scores, against 237, 188 and 85 of the 510 episodes.

Reviewer_fMEuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In-MANU evaluates whether a frontier multimodal agent can control a robot without robot-specific training, how its capabilities compare with post-trained vision-language-action (VLA) and world-action models, and whether delegation or self-improvement can close the gap. On 26 EBench tasks, GPT-6-Astra achieves a 46.7% success rate that ranks second among eight evaluated systems and performs best on short-horizon tasks. Its performance is weaker on long-horizon, high-precision, and tabletop manipulation. The paper evaluates access to a pretrained robot policy as a callable tool and a self-improvement loop that revises the agent’s playbook and harness based on execution feedback. Neither approach produces a statistically significant overall improvement on test tasks.

Critical strengths
  1. The paper investigates an important gap between general-purpose multimodal reasoning and reliable physical control.
  2. Evaluating the agent against seven post-trained policies on a common benchmark helps identify where general-purpose agents are competitive and where specialized policies retain an advantage.
  3. The analysis of partial completion, recovery behavior, delegation decisions, and execution errors provides useful insight into why tasks fail.
  4. The lack of significant improvement from self-improvement is informative, particularly because the authors analyze how the agent’s strategy changes without corresponding gains in success.
Critical weaknesses
  1. The improvement loop uses one run per model and relatively few episodes per task. Consequently, the negative result is informative but does not establish that execution-feedback-based self-improvement is generally ineffective.
  2. The post-trained policies’ results are taken from the EBench leaderboard rather than a fully controlled rerun of all systems. Differences in evaluation conditions may affect the ranking.
  3. The experiments use one simulator and one robot. The findings may not transfer to other embodiments or real-world manipulation.
  4. The recorded trajectories support the proposed execution bottlenecks, but the experiments do not fully isolate the causal contributions of planning, perception, and low-level control.
Questions for authors
  1. Given the small number of episodes and a single run per model, how confident can we be that the self-improvement loop produces no meaningful test-time gains?
  2. Why does the agent struggle to improve the harness meaningfully? The paper acknowledges limitations in the self-improvement approach, but it would be useful to discuss what changes might help and what directions should be explored next.
  3. How comparable are the evaluation protocols for GPT-6-Astra and the leaderboard policies, and could these differences affect the reported ranking?
  4. Which findings are expected to generalize beyond the evaluated simulator and robot?
3 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank the reviewer for the careful reading and for finding the failure analysis and the self-improvement result informative. The revision adds one page of analysis: Section 5.6 locates the capability boundary, and three highlighted insights state what the evidence implies for delegation and self-improvement. SR is averaged within each task and then equally across tasks. Statistical comparisons pair outcomes by scene.

W1: The improvement loop uses one run per model and relatively few episodes per task. Consequently, the negative result is informative but does not establish that execution-feedback-based self-improvement is generally ineffective.

Q1: Given the small number of episodes and a single run per model, how confident can we be that the self-improvement loop produces no meaningful test-time gains?

We agree that the evaluation cannot rule out a meaningful gain, and the revision no longer reads it as evidence that execution-feedback self-improvement is ineffective. Specifically, each model went through five rounds of improvement instead of a single round, and its final version was tested on 90 paired episodes, five scenes from each of 18 tasks. The observed changes are smaller than this sample can resolve: GPT-6-Astra moves from 44.4% to 46.7% SR, so a moderate gain or loss remains possible, and Opus 5.5 moves from 40.0% to 31.1%. What the five rounds do change is the agent's behavior. The final GPT-6-Astra version ends 3 of 90 test episodes early, compared with 10 at the starting point. After it learns a rule to add a closing stage once an appliance is loaded, it plans to close the dishwasher in all five dishwasher episodes instead of one, and succeeds in four instead of two. Section 6 now states which differences this sample can detect and reports the strategy changes separately from differences in success. The conclusion names tests on more scenes per task as the way to establish whether such changes translate into reliable gains.

W2: The post-trained policies’ results are taken from the EBench leaderboard rather than a fully controlled rerun of all systems. Differences in evaluation conditions may affect the ranking.

Q3: How comparable are the evaluation protocols for GPT-6-Astra and the leaderboard policies, and could these differences affect the reported ranking?

The evaluation protocols are aligned, so the success rates of the eight systems are directly comparable. All eight are evaluated on the same 510 test scenes with the benchmark's scorer and observation protocol, and the leaderboard entry of GPT-6-Astra reproduces the local run episode by episode, which allows comparisons paired by scene. The ranking therefore reflects the systems rather than their evaluation conditions.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

W4: The recorded trajectories support the proposed execution bottlenecks, but the experiments do not fully isolate the causal contributions of planning, perception, and low-level control.

We agree that the causal contributions are not fully isolated, and the revision adds three partial separations.

  • Low-level control. Only 4 of 51 unsuccessful peg, nut and gear episodes contain a waypoint miss above 5 cm, and in 33 of 35 unsuccessful nut and gear episodes the end effector holding the part reaches its commanded seating height within 4 mm before release.
  • Perception. True initial object poses yield 5 of 20 successes against 2 of 20, too few pairs to establish or exclude a localization effect.
  • Model versus interface. Qwen3.8-Flash with the same interface misses its targets about as often as GPT-6-Astra, on 6.6% and 5.2% of arm waypoints, but completes no subgoal in 35 of its 47 failures, against 7 of 27.

The new Section 5.6 therefore places the capability boundary between task reasoning, which differs across models, and final alignment at contact, where the interface returns feedback only after each motion.

Q2: Why does the agent struggle to improve the harness meaningfully? The paper acknowledges limitations in the self-improvement approach, but it would be useful to discuss what changes might help and what directions should be explored next.

The loop revises what its execution records make visible, and contact misalignment is not visible in them. Of 22 accepted revisions, 20 are playbook rules and two enable feedback fields. Strategy errors such as leaving the dishwasher open or ending an episode early appear in the agent's stated reasons and in the Score, and the accepted rules correct them. Millimeter misalignment raises no error, because the harness reports no contact and the scorer's tolerances are withheld. The critic assigns all eight recurring failure modes to execution, yet no accepted revision specifies a tolerance or adds a contact primitive. A new signal must also be acted upon: with the added waypoint residual, the agent corrects 1 of 30 large misses on the test tasks, against 5 of 21 without it. Section 6.2 now states this bound as Insight 3, and the conclusion derives its directions from it and from Insight 2: contact perception in the harness, failure and revision pairs turned into validated closed-loop tools, and policies trained as tools that report their competence and the outcome of each segment.

W3: The experiments use one simulator and one robot. The findings may not transfer to other embodiments or real-world manipulation.

Q4: Which findings are expected to generalize beyond the evaluated simulator and robot?

We expect the mechanisms to transfer, not the success rates or rankings. Three findings concern the coupling of a reasoning model with its executors rather than EBench itself. First, an agent that observes only between open-loop motions fails where contact needs feedback within a motion. Second, a learned policy helps only on the stages the agent assigns it well: with π0.5 as a tool, the agent solves 88 scenes that direct control misses but misses 59 that direct control solves. Third, a self-improvement loop learns what its records measure. The revision states both points: the conclusion presents these mechanisms as testable hypotheses for other embodiments, and the scope paragraph states that the success rates and rankings hold only for this simulator and robot.

Thank you for the clarifications and additional analyses. The shared evaluation protocol addresses my comparability concern, and the expanded evaluation strengthens the delegation result. I appreciate the acknowledgment that the self-improvement results remain inconclusive. The failure analysis is useful, though it does not fully separate perception, planning, and control errors. Overall, my main concerns are sufficiently addressed. I maintain my rating of 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.