- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how can frontier models facilitate robotic manipulation tasks. It contrasts execution-time decision making, where the model repeatedly selects the next operation, with executable-policy synthesis, where the model writes a persistent robot program and revises it from execution feedback. The paper instantiates the latter with GPT-6 Astra in a Codex session and evaluate on LIBERO, LIBERO-Pro, and the RoboCasa365 Target50 subset. Reported results include 100% success on LIBERO, 95.0% on LIBERO-Pro, and 88.0% on RoboCasa365 Target50, including 75.0% on unseen composite tasks. A refinement-budget study on LIBERO-Pro shows success increasing from 45% with one turn to 94% with eight turns.
Critical strengths
-
The paper studies an important research question. Rather than presenting another robotic policy, it examines how model capability is exposed through different interaction paradigms. The framing is likely to matter beyond the particular systems studied here.
-
The refinement-budget experiment provides direct evidence to support the claim that execution feedback remains useful even for strong frontier models. Additionally, reporting the refinement turns, tokens, and wall-clock time gives a more complete picture of what the performance improvement costs.
-
The Appendix provides implementation details, which provides a reproducible foundation for future research.
Critical weaknesses
-
The comparison between interaction paradigms is not controlled. HarnessVLA and the proposed Codex refinement loop appear to differ in interface, available information, budget, and model access. The results therefore cannot isolate whether executable-policy synthesis itself is responsible for the observed advantage.
-
The conclusion about scaffolding appears to be broader than the empircal evidence. As recognized by the paper, the RATs comparison is explicitly partial and descriptive since the task coverage and number of completed trials differ, which weakens the performance comparison.
-
The experimental evaluation relies primarily on GPT-6 Astra or GPT-series models, whereas assessments of other models (e.g., Claude and Gemini) are incomplete or inconsistent, limiting the generalizability of conclusions across different frontier models.
Questions for authors
-
Can the paper compare execution-time decision making and executable-policy synthesis under the same interface, available information, budget, and model access?
-
Which forms of scaffolding were actually tested, and which remain untested? Is the negative conclusion specific to RATs or intended to cover structured policy systems more broadly?
-
Does the refinement-budget trend persist when token and wall-clock budgets are matched across variants?
-
In long-horizon tasks, how is the iterative process prevented from becoming trapped in local optima or overfitting to specific scenario configurations?
4 comments
Individual Response [2/ 3]: W3 & Q3
W3: Extending conclusions beyond GPT-6
To ensure our findings generalize, we replicated the experiments from Table 4 using Claude Opus 5.5 (via Claude Code on high effort). As shown in Table R4, we ran 160 total sessions across the 40 seed=1 tasks in all four LIBERO-Pro suites, keeping the prompts, RGB-D observations, and rules completely identical to our original setup.
| Model Family (Seed 1, 40 Tasks) | 1-Turn Success | 2-Turn Success | 4-Turn Success | 8-Turn Success |
|---|---|---|---|---|
| GPT-6 Astra | 16/40 (40.0%) | 30/40 (75.0%) | 36/40 (90.0%) | 38/40 (95.0%) |
| Claude Opus 5.5 | 28/40 (70.0%) | 34/40 (85.0%) | 39/40 (97.5%) | 40/40 (100.0%) |
The results in Table R4 clearly show that the success of our finding is not limited to a single model. When tested with Claude Opus 5.5 under the same minimal interface, we see the exact same pattern: giving the model more chances to interact and correct itself drives a massive performance boost, going from 70.0% success at 1-Turn all the way to 100.0% at 8-Turns. This proves that relying on minimal scaffolding and increased trial-and-error feedback is a universally effective strategy for maximizing frontier model performance.
Q3: Performance with token-budget cap and time-budget cap
In addition to the 480 uncapped sessions in Table 4, we evaluated 160 capped sessions across all four LIBERO-Pro suites (40 tasks, seed=1). Here, the model is explicitly informed of its attempt, time, and token budgets via the prompt, while the harness enforces a hard cumulative cap (set at the 90th percentile of one-shot N=1 runs per suite: 5–8 minutes or 45K–54K uncached tokens across all turns, terminating immediately once reached):
| Budget Regime (Seed 1, 40 Tasks) | N = 1 Success | N = 8 Success |
|---|---|---|
| Uncapped | 16/40 (40.0%) | 38/40 (95.0%) |
| Time-Capped | 16/40 (40.0%) | 34/40 (85.0%) |
| Token-Capped | 15/40 (37.5%) | 39/40 (97.5%) |
| Budget (N = k) | LIBERO-10-Swap (Indep.) pass@k | LIBERO-10-Swap (Feedback) | Goal-Swap (Indep.) pass@k | Goal-Swap (Feedback) |
|---|---|---|---|---|
| 1 Turn | 21.0% | 20.0% | 48.0% | 45.0% |
| 2 Turns | 30.0% | 60.0% | 63.0% | 60.0% |
| 4 Turns | 35.0% | 80.0% | 76.0% | 80.0% |
| 8 Turns | 40.0% | 95.0% | 90.0% | 90.0% |
| Budget (N = 8) | LIBERO-10-Swap (Indep.) | LIBERO-10-Swap (Feedback) | Goal-Swap (Indep.) | Goal-Swap (Feedback) |
|---|---|---|---|---|
| Success | 40% | 95% | 90% | 90% |
| Wall-clock per task (min) | 29.5 (4.4×) | 6.7 | 14.4 (2.7×) | 5.3 |
| Tokens per task (K) | 214 (4.3×) | 50 | 95 (2.5×) | 38 |
(i) Gains persist under fixed resource caps. Allowing up to eight attempts improves success from 40% to 85% under the same time cap and from 37.5% to 97.5% under the same token cap (table above). (ii) The gains go beyond additional independent attempts. On LIBERO-10-Swap, feedback-guided refinement achieves 95% success versus 40% for eight independent one-shot attempts under the same attempt limit and reset rules.
Individual Response [3/ 3]: Q4
Q4: Preventing local optima and scenario overfitting in long-horizon tasks
Local Optima: We did not introduce any explicit mechanism to prevent local optima on long-horizon tasks. However, based on our observations, three inherent properties of the loop prevent stagnation in practice: (1) each retry resets cleanly to the initial state, which prevents physical execution errors from compounding across attempts or trapping the robot in unrecoverable states; (2) diagnostic feedback triggers structural code changes rather than mere parameter tuning—across 54 long-horizon RoboCasa runs, ~50% of revisions alter 10–50% of the policy across both solved and failed episodes, allowing the agent to abandon flawed control logic and explore new execution strategies; and (3) the agent locks in working stages to build multi-stage policies incrementally, preserving early-stage progress so later revisions can focus exclusively on unresolved subtasks without breaking working code. Together, these properties allow 8 feedback turns to solve 95% of long-horizon LIBERO-Pro tasks compared to 40% for 8 independent one-shot retries, with the few remaining failures largely tied to budget limits or unobservable state thresholds.
Trajectory Overfitting: To test for trajectory overfitting, we ran two evaluations: repeating the refinement loop across three independent random seeds on the full 40-task benchmark, and executing each of the 113 successful policies unchanged on the other two initial states of the same task, yielding 226 cross-seed replays without further refinement. Because the generated code queries live RGB-D and wrist-camera streams to compute target poses online rather than memorizing fixed waypoints, policies adapt naturally to layout variations. Consequently, we observe stable cross-seed success (38/40, 38/40, and 37/40) and 74.3% zero-shot transfer to unseen states (168/226 overall, and 83.3% across the Object, Spatial, and Goal suites). Transfer initially drops to 44% on LIBERO-Long when the agent takes shortcuts by hard-coding episode coordinates, a byproduct of our default instance-specific prompt rather than the loop itself. In a controlled ablation on LIBERO-Long, simply adding one prompt sentence stating that the policy will be tested on unseen layouts forces the agent to rely on perception queries, boosting zero-shot transfer from 39% to 78% while preserving in-session success (39/40 vs. 38/40) at ~27% higher token cost.
Inquiry Regarding Review Status
Dear Reviewer kG9J,
Thank you again for your valuable feedback and time dedicated to reviewing our manuscript.
We have submitted our rebuttal and revised manuscript addressing the reviewers' concerns. As we look forward to any further feedback, we would like to gently check whether you have any additional questions or comments, or require further clarification regarding our response.
We are more than happy to provide additional experiments, analyses, or explanations if anything remains unclear.
Best regards,
Submission 50586 Authors
Individual Response [1/ 3]: W1 (Q1) & W2 ( Q2 )
We thank Reviewer kG9J for the valuable feedback and constructive suggestions, which have helped us substantially strengthen our empirical analysis. Below, we address your comments point by point and have integrated all results into the revised manuscript.
W1 & Q1: Controlled comparison between interaction paradigms
The original HarnessVLA setup uses a prior bootstrapping phase to build task-specific memory and disables resets during evaluation. For a fair comparison, we implemented a matched execution-time decision baseline following HarnessVLA's core design, evaluated with GPT-6 Astra on the exact same task instances, RGB-D inputs, and time caps as our executable-policy synthesis method. Under this setup, the model selects action primitives step-by-step during execution, can reset and retry after failures, and starts from scratch without prior memory.
Under the same reset rules and budgets, writing an executable policy reaches higher overall success than step-by-step decision-making (88.0% vs. 78.0%, and 87.5% vs. 68.8% on Composite-Seen) while using roughly half the total tokens (3.75M vs. 7.52M) in about the same time.
W2 & Q2: Scope of the scaffolding conclusion and tested variants
To analyze how different scaffolding structures impact frontier model performance on robotics tasks, we evaluated five setups across 120 LIBERO-Pro swap instances (4 suites × 10 tasks × 3 seeds) using GPT-6 Astra-High with identical resets and RGB-D inputs (Table R2). We report 1-try (N=1) and 8-try (N=8) success rates, plus the median token cost of the first attempt. The first four setups operate under the same codex-cli settings with increasing structure, compared alongside the official RATs harness:
According to the results above, we can conclude: