Code as Policies via Geometry-Conditioned Search

Synthesizing robot-control programs from language requires geometric information that object poses alone do not provide, including functional directions and interaction locations. We introduce LOGIC, a geometry-conditioned policy synthesis framework that maps task instructions to a search space structured around geometric hypotheses and task-relevant control parameters. For each hypothesis, an LLM constructs an execution plan, identifies the parameters needed to carry it out, and generates parameterized control programs. Environment rollouts guide both parameter optimization and selection among hypothesis-conditioned programs. Execution feedback also refines the search space itself: failures guide revisions to the geometric hypotheses and task-relevant parameterizations that define subsequent policy candidates. New programs are then generated and optimized within this revised space. This combination of search-space construction, execution-based optimization, and feedback-driven refinement enables the synthesis of policies that compute end-effector poses and gripper commands without predefined task-level skill APIs. Experiments across manipulation and tool-use tasks, using both simulator-provided geometry and geometry estimated from RGB-D observations, show substantial improvements over code-as-policies and evolutionary baselines.