- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LOGIC, a framework for generating low-level robot control programs without predefined skill APIs. The method explicitly identifies geometric ambiguities, generates alternative hypotheses, and uses anticipatory planning to identify physical parameters for optimization. Candidate policies are optimized using TPE and refined through execution feedback. Experiments on RLBench, Robosuite, and LIBERO show substantial improvements over existing code-generation and evolutionary baselines.
Critical strengths
- The distinction between geometric semantic ambiguity and physical parameter ambiguity is well motivated. Exploring alternative geometric interpretations rather than committing to a single assumption seems particularly useful for low-level robotic control, where incorrect functional directions or interaction points can lead to systematic failures.
- LOGIC achieves 71% average success across 18 RLBench tasks, compared with 34% for the strongest baseline. Notably, LOGIC without evolution already achieves 65%, suggesting that the structured generation and optimization process contributes substantially. The RGB-D results are also encouraging, with 86% average success across 20 tasks.
Critical weaknesses
-
LOGIC generates multiple programs for each geometric hypothesis and performs extensive parameter optimization. While the baselines also use TPE, it is unclear whether the overall search budgets are comparable. Could the authors compare against a baseline that generates a similar number of diverse programs under the same total LLM and environment-interaction budget, but without explicit geometric hypothesis branching? This would help clarify the contribution of the proposed search structure.
-
While the overall framework is understandable, the paper does not clearly explain how execution feedback leads to changes in geometric hypotheses versus numerical parameters or control logic. The approach also depends on the LLM generating useful initial hypotheses and parameter ranges, but its robustness to incorrect or incomplete proposals is unclear. More concrete examples would make the method and its limitations easier to assess.
Questions for authors
Q1. Could the authors compare against a baseline that generates a similar number of diverse policies under matched LLM and environment-interaction budgets, but without explicit hypothesis branching? This would help isolate the contribution of the proposed geometric structure. Q2. The experiments use gpt-5-mini. Would the advantage of explicit hypothesis branching remain as large with a stronger reasoning model, such as GPT-6? Comparing LOGIC and the baselines under the same stronger backbone would help determine whether the framework addresses limitations that persist as model capabilities improve. Q3. When a policy fails, how does LOGIC distinguish between an incorrect geometric hypothesis, poor parameter values, and flawed control logic? Is this decision made entirely by the LLM? If a hypothesis changes, are the planning and parameter optimization stages repeated? Q4. The results contain many success-rate comparisons, but relatively little explanation of what actually changes during optimization and evolution. A step-by- step example showing an initial hypothesis, execution failure, and subsequent revision would be valuable. For tasks with lower success rates, such as PutUmbrellaInUmbrellaStand and OpenWashingMachine, what are the main failure causes? Q5. If the correct geometric interpretation is missing from the initial hypothesis space, or useful parameter values fall outside the LLM-proposed ranges, can the evolution process recover? This seems important for generalization to unfamiliar objects and more complex tasks.
6 comments
Rebuttal by Authors (2/5)
R2. The experiments use gpt-5-mini. Would the advantage of explicit hypothesis branching remain as large with a stronger reasoning model, such as GPT-6? (Q2)
We studied the effect of model capability in Appendix D.5 (Table 6), where LOGIC is evaluated with gpt-5-mini, gpt-5, gpt-5.4-mini, and gpt-5.4. LOGIC scales with model capability (average success 0.80 with gpt-5-mini, 0.83 with gpt-5, and 0.82 with gpt-5.4), and the small gap between the standard and mini variants shows that the structured search lowers the dependence on raw model scale.
We agree that GPT-6 Astra, as the latest and strongest reasoning model, deserves a dedicated study, and that the baselines should be given the same backbone. We therefore ran LOGIC and all four baselines with GPT-6 Astra on six tasks. The GPT-6 Astra results are reported as mean ± std over 3 seeds, each evaluated on 50 held-out episodes. The LOGIC (GPT-5-mini) column gives the results reported in the paper (5 seeds for the simulator-state tasks in Table 1, and 3 seeds for the RGB-D tasks in Table 2).
| Type | Task | LOGIC (GPT-5-mini) | LOGIC (GPT-6 Astra) | CaP (GPT-6 Astra) | AlphaEvolve (GPT-6 Astra) | EOH (GPT-6 Astra) | MLES (GPT-6 Astra) |
|---|---|---|---|---|---|---|---|
| Oracle State | TakeMoneyOutSafe | 0.67 ± 0.35 | 0.70 ± 0.11 | 0.37 ± 0.08 | 0.40 ± 0.28 | 0.53 ± 0.06 | 0.22 ± 0.17 |
| Oracle State | PutItemInDrawer | 0.65 ± 0.23 | 0.72 ± 0.03 | 0.44 ± 0.10 | 0.56 ± 0.14 | 0.52 ± 0.04 | 0.66 ± 0.18 |
| Oracle State | ScoopWithSpatula | 0.83 ± 0.09 | 0.88 ± 0.08 | 0.32 ± 0.12 | 0.45 ± 0.15 | 0.64 ± 0.06 | 0.42 ± 0.28 |
| Perception | open the microwave | 0.86 ± 0.03 | 0.91 ± 0.08 | 0.82 ± 0.07 | 0.64 ± 0.09 | 0.70 ± 0.06 | 0.85 ± 0.14 |
| Perception | stack the cube | 1.00 ± 0.00 | 1.00 ± 0.00 | 0.98 ± 0.03 | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 |
| Perception | turn on the stove | 0.93 ± 0.01 | 0.97 ± 0.04 | 0.63 ± 0.02 | 0.92 ± 0.09 | 0.68 ± 0.11 | 0.64 ± 0.22 |
| Average | 0.82 | 0.86 | 0.59 | 0.66 | 0.68 | 0.63 |
Three findings follow.
- The advantage persists under the same stronger backbone. With GPT-6 Astra, LOGIC achieves the best result on every task, with an average of 0.86 compared with 0.59 to 0.68 for the baselines.
- LOGIC with gpt-5-mini (0.82) still outperforms every baseline equipped with GPT-6 Astra (at most 0.68). The structure of the search contributes more than the upgrade of the backbone.
- LOGIC itself continues to benefit from the stronger model (0.82 to 0.86), consistent with the trend in Table 6.
The reason is that the limitation LOGIC addresses is a lack of information, which stronger reasoning alone does not supply. Which local axis points toward a tool head, or on which side a hinge lies, is not determined by the task description and sparse observations, so even a strong model must commit to a guess unless alternatives are explored and tested by execution. This is consistent with Appendix F.2, where a strong VLM (GPT-5.4) asked to select a single interpretation from code, success rates, and three-camera renderings chose the correct one in only about one third of the cases (Table 10). The GPT-6 Astra results are now reported in Appendix D.5 (Table 7).
Rebuttal by Authors (3/5)
R3. When a policy fails, how does LOGIC distinguish between an incorrect geometric hypothesis, poor parameter values, and flawed control logic? Is this decision made entirely by the LLM? If a hypothesis changes, are the planning and parameter optimization stages repeated? (Weakness 2, Q3)
We thank the reviewer for the opportunity to clarify this. LOGIC addresses the three failure sources through complementary mechanisms. The LLM proposes diagnoses and revisions, while execution determines candidate selection.
- Poor parameter values are addressed by TPE before evolution. Each candidate’s parameters are optimized under its current geometric hypothesis and program structure (Section 4.4). Failures that persist after this search motivate examining the hypothesis, control logic, missing parameters, and search ranges.
- Incorrect geometric hypotheses are addressed through branching and execution-based selection. Candidate hypotheses are instantiated as separate branches and ranked by measured task success (Eq. 4). This deprioritizes poorly performing branches without requiring the LLM to determine which geometric interpretation is correct.
- Residual failures are diagnosed in Stage 5. The LLM receives the policy code, the original prompt specifying the hypothesis, anticipatory plan, and parameter instructions, the measured success rate, and the final frame of a failed execution (Section 4.5, Appendix I.5). These explicit assumptions and control steps provide a basis for identifying a likely cause, such as an incorrect opening arc (Appendix E.2) or a collision with the stick head before grasping in Hockey (Section 4.5).
Each proposed revision undergoes parameter re-optimization with the full TPE budget and joins the candidate pool. The elite set is then re-selected from parents and revised candidates by measured success rate (Algorithm 1). Revisions are therefore evaluated by their execution performance rather than accepted solely on the LLM’s diagnosis.
When a hypothesis changes, the LLM jointly revises the affected plan steps, waypoints, parameters, and search ranges within a single evolution step, followed by full parameter re-optimization. If the interpretation was absent from the initial hypotheses, the revised program enters the pool as a new branch. We describe this procedure in Section 4.5 and give two complete revision examples in Appendix F.6 of the revised paper.
Rebuttal by Authors (1/5)
We appreciate your insightful feedback and constructive comments! We address your comments point by point below (R1 to R6) and have revised the paper accordingly.
R1. Are the overall search budgets comparable? Could the authors compare against a baseline that generates a similar number of diverse programs under the same total LLM and environment-interaction budget, but without explicit geometric hypothesis branching? (Weakness 1, Q1)
Thank you for this suggestion. The search budgets are matched in the main comparison: every baseline in Table 1 receives the same total number of generated programs (86) and environment rollouts (172,000) as the full LOGIC pipeline (Appendix F.4). Both budgets the reviewer mentions, the LLM budget (LLM calls and LLM-generated programs) and the environment-interaction budget, are therefore matched between the full LOGIC pipeline and the baselines. Moreover, LOGIC without the evolution stage uses substantially fewer LLM calls and environment interactions than the baselines, and it still outperforms all of them (Table 1). In addition, the ablation "w/o Geometric Branching" in Appendix D.3 is the control you describe. It repeatedly samples diverse policies with anticipatory planning and explicit parameter generation, without assigning them to geometric branches, under the same overall search budget and the same per-policy TPE settings as the full method. In Table 3, this reduces success from 0.90 ± 0.03 to 0.53 ± 0.02 on OpenDrawer, from 0.74 ± 0.06 to 0.33 ± 0.06 on BlockPyramid, and from 0.76 ± 0.19 to 0.36 ± 0.07 on Hockey.
As this question is central to our contribution, we extended the component ablations to six more tasks, three with simulator-provided states and three under RGB-D perception, so that they now cover nine tasks. The first three rows are from the original submission (mean ± std over 5 seeds); the newly run ablations in the other six rows report mean ± std over 3 seeds, each evaluated on 50 held-out episodes. In these six rows, the LOGIC (Full) results, and the w/o Evolution results of the three simulator-state tasks, are those reported in the paper (5 seeds for the simulator-state tasks in Table 1, and 3 seeds for the RGB-D tasks in Table 2).
With the number of diverse programs and the interaction budget held fixed, removing explicit hypothesis branching lowers the average success rate from 0.82 to 0.46, the largest drop among all components. This shows that explicit hypothesis branching is the most important component of LOGIC, and that its advantage comes from the search structure and not from a larger budget. The nine-task ablation is now in the revised paper (Table 3).