- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents a simulation study of feedback-driven code revision for robotic manipulation. Using a shared execution framework and skill library, it examines model choice, skill priors, temporal and spatial feedback, and a privileged-to-perception curriculum across five VABAR tasks and four MimicGen tasks. The model comparison evaluates initial policies and two feedback rounds; the curriculum experiments provide longer trajectories with more iterations.
Critical strengths
-
The study evaluates policy improvement beyond initial generation and reveals meaningful differences in learning dynamics. In Tab.1, Opus 5 xhigh, Fable 5.1, GPT-6 Astra, and Sonnet 5 begin at 43–45% overall success but reach 74%, 64%, 48%, and 46% after two feedback rounds. This shows why initial success alone is an incomplete evaluation of an agent that can revise its policies. Fig.10 adds useful distinctions between partial progress, delayed improvement, and stagnation. For example, perception-only Threading improves its stage reward while task success remains near zero. These observations provide a valuable basis for evaluating iterative robot programming.
-
The feedback and curriculum comparisons offer concrete design insights, including important negative results. Tab.2 and 3 report overall success of 25/36/52% for episode/motion/skill feedback and 30/40/52% for world-frame/object-frame/part-annotated feedback. The earlier perception error behind Tool Hanging's final reaching failure makes the value of intermediate feedback tangible. The curriculum also has sharply different effects across tasks: Tab.4 reports 86% versus 2% on Threading, but 16% versus 63% on Tool Hanging, for curriculum versus perception-only learning.
Critical weaknesses
-
The source and robustness of the iterative gains remain insufficiently characterized. Moving from @0 to @1 and @2 adds model inference, execution, candidate policies, and potentially validation-based selection. The main paper does not report a comparison with independent restarts or refinement without execution feedback under matched inference and execution budgets. The feedback ablations compare report formats but do not establish how much of the overall gain comes from using interaction evidence rather than additional attempts. The trajectory evidence also warrants a narrower interpretation: nine of ten model settings have smaller second-round gains, yet Opus 5 high still improves by 19 points and GPT-6 Sol improves more in the second round than the first. Sol's entire overall gain comes from USB-C, with the other four tasks unchanged. The longer curriculum curves include plateaus, delayed gains, and reversals under changing observation conditions. These results establish useful differences within the tested budgets, but leave unclear how sensitive the model comparisons and bottleneck interpretations are to the selected refinement budget.
-
The central interpretation about information use is stronger than the feedback controls establish. In Tab.2, skill-level feedback includes non-motion operations such as perception that motion-level feedback omits, so information coverage and semantic organization change together. In Tab.3, part annotations add functional geometry; their benefit cannot be attributed solely to reorganizing information already available. The world-frame/object-frame comparison is more directly relevant, but the complete information equivalence needs to be established. These results support useful feedback interfaces more directly than the central claim that interpreting information is often more limiting than obtaining sufficient information.
Questions for authors
-
Does Table 1's @k evaluate the policy produced at iteration k, or the policy selected by validation from iterations 0 through k? How does this relate to Fig.10's curves and Tab.4's last-three-iteration averages? Are training layouts reused across rounds, resampled, or accumulated, and which execution outcomes are visible to the agent?
-
Under the same total inference and execution budgets and validation rule, how does the feedback loop compare with independent policy generation/restarts or refinement without execution outcomes? A comparison on representative tasks would help establish the contribution of feedback use.
-
For representative settings that improve rapidly, stall, or improve late, what happens when refinement continues under unchanged observations and feedback? Please distinguish current-policy performance from the performance of the policy selected by validation, and report variability across runs for these curves and Tables 1–3. This would help assess whether the reported differences are robust to the stopping budget.
-
Does the world-frame condition contain the full poses and object geometry needed to derive all quantities in the object-frame report? Can the temporal comparison preserve coverage of perception and other non-motion steps while changing only report organization? This would separate representation effects from added information.
-
Is there a within-model comparison of skill availability that supports the boundary/scaffold interpretation? The initial unmatched-line fraction alone does not directly measure useful adaptation during later rounds. Do code changes across feedback rounds, or changes in policy graphs across curriculum transfer, support the proposed mechanisms beyond initial line matching and total skill counts?