- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper do a systematic evaluation of GPT-6-Astra as a frozen, direct-action policy for robotic manipulation, comparing it with seven post-trained VLA and WAM policies across 26 EBench tasks. The authors also investigate policy delegation and non-parametric self-improvement to address its limitations. The results show competitive performance on short-horizon tasks but weaknesses in long-horizon and contact-rich manipulation, while neither improvement strategy yields significant gains. The study provides valuable insights into multimodal agents for robotic control, although some concerns remain regarding novelty, experimental fairness, and statistical significance.
Critical strengths
- The paper provides a comprehensive and fine-grained empirical analysis of frontier agents in robotic manipulation.
Critical weaknesses
- The novelty of the main findings is somewhat limited, as many of the observations are already well known in the research community.
- The evaluation is limited to a single simulation benchmark (EBench) and one robot configuration, without real-world experiments. Although EBench covers diverse manipulation tasks, it remains unclear whether the findings generalize to different robotic platforms and more complex real-world scenarios.
Questions for authors
- What is the expected mechanism through which self-improvement can enhance precise manipulation when the model weights and low-level controllers remain frozen? Is the proposed framework fundamentally limited to improving high-level strategies rather than acquiring new manipulation capabilities?
- Could the authors evaluate joint-space control instead of end-effector-space control? The choice of action space may also significantly affect performance.
- Although EBench contains diverse tasks, could the authors clarify which conclusions are expected to generalize beyond this specific simulator and robot configuration?
- More models should be evaluated. Models such as GPT-6 Astra and Claude Opus 5.5 may have been trained with robotics-related data, which could affect the results and may not fully support the paper's claims.
2 comments
Official Comment by Authors (2/2)
Q2: Could the authors evaluate joint-space control instead of end-effector-space control? The choice of action space may also significantly affect performance.
We did not evaluate a frontier agent that issues joint targets, and we agree that the action space can matter. The diagnostics suggest it is not the main factor in the high-precision failures, because these occur after the commanded pose is reached: on the nut and gear tasks, 33 of the 35 failures release the part with the end effector within 4 mm of its commanded seating height. A different parameterization of the reach is therefore unlikely to change this final alignment unless it also supplies contact feedback. The π0.5 tool outputs joint targets from fresh observations and raises SR on long-horizon and tabletop tasks, but not on the high-precision tasks, where SR is 11.3% without it and 7.5% with it. Because it changes the executor as well as the action space, this result does not isolate joint-space control. Appendix D lists a joint-command ablation as unevaluated, and the paper characterizes each model together with its position-waypoint interface.
W2: The evaluation is limited to a single simulation benchmark (EBench) and one robot configuration, without real-world experiments. Although EBench covers diverse manipulation tasks, it remains unclear whether the findings generalize to different robotic platforms and more complex real-world scenarios.
Q3: Although EBench contains diverse tasks, could the authors clarify which conclusions are expected to generalize beyond this specific simulator and robot configuration?
We expect the mechanisms to generalize, not the success rates or rankings. Three findings concern the coupling of a reasoning model with its executors rather than EBench itself. First, an agent that observes only between open-loop motions fails where contact needs feedback within a motion. Second, a learned policy helps only on the stages the agent assigns it well, which calls for tool interfaces that report the policy's competence and results. Third, a self-improvement loop learns what its records measure. The revision states both points: the conclusion presents these mechanisms as testable hypotheses for other embodiments, and the scope paragraph states that the success rates and rankings hold only for this simulator and robot.
Q4: More models should be evaluated. Models such as GPT-6 Astra and Claude Opus 5.5 may have been trained with robotics-related data, which could affect the results and may not fully support the paper's claims.
The revision adds Qwen3.8-Flash with the same interface on two scenes of each of the 26 tasks. It completes 5 of 52 scenes against 25 for GPT-6-Astra. Claude Opus 5.5 is evaluated in the self-improvement study, with π0.5 available as a tool, on 90 paired episodes. The models differ in task progress rather than reach accuracy, so the model's contribution, including any robotics-related priors, appears in how far a task progresses. We cannot rule out robotics-related pretraining data for proprietary models, so the paper defines "no robot-specific training" operationally: we perform no robot-specific parameter updates. The claims therefore concern the deployed models with their existing priors and the stated inference inputs.
Official Comment by Authors (1/2)
We thank the reviewer for the precise questions about novelty, mechanism and scope. In response, the revision adds one page on what the evidence implies beyond the expected contrast between task reasoning and contact precision: Section 5.6 locates the capability boundary, and three highlighted insights state the implications for delegation and self-improvement. SR is averaged within each task and then equally across tasks. Statistical comparisons pair outcomes by scene.
We agree that a frontier agent reasoning better about tasks than it handles precise contact is expected. Prior studies of frontier models as robot policies mainly establish control interfaces, whereas this study measures the agent against seven post-trained policies on the same scenes. The revision states three findings from this comparison that go beyond the expected contrast.
The framework is not limited in principle, but the five rounds improved strategies only, because the execution records contained no signal of contact precision. With frozen weights and controllers, harness revisions can still improve precision in two ways: by adding executable primitives, such as a closed-loop insertion skill or a learned policy segment, and by adding feedback channels, such as contact or clearance estimates from the cameras. Over the five rounds, 20 of 22 accepted revisions were playbook rules, and they changed decisions. After a rule on completing enclosures, the agent plans to close the dishwasher in all five dishwasher test episodes instead of one. The final version also ends 3 of 90 test episodes early instead of 10. No accepted revision specified a tolerance or added a contact primitive, because millimeter misalignment raises no error in the records. A new channel must also be acted upon: with an added waypoint residual, the agent corrected 1 of 30 large misses, against 5 of 21 without it. Section 6.2 now states this bound as Insight 3, and the conclusion names contact perception and validated closed-loop tools as the route to new manipulation capabilities.