acceptodds
Withdrawn from ICLR 2027

Write Once, Plan Many: Building Reusable Visual Planning Systems from One Scene

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) can reason about visual planning tasks, but querying them for every instance incurs repeated inference cost and latency. Generating reusable planning programs offers an alternative, but constructing reliable planning systems for new problem domains without extensive supervision remains challenging. We present a training-free framework that constructs reusable perception and planning programs from a domain specification and a single practice scene without a paired solution. The central challenge is obtaining useful feedback from the limited input. Our framework addresses it by leveraging VLM's ability to provide a renderer and a dynamic model based on its scene understanding. Rendering detected states enables comparison with the practice scene, while checking simulated action outcomes against the domain specification helps expose errors in rule implementation. Together, these checks guide refinement of the grounding program and planner without requiring solution labels. After refinement and component selection, the programs are frozen and solve new instances without further VLM calls. The same framework constructs systems across domains without manual pipeline redesign. Across fifteen domain variants spanning 2D and 3D scenes with discrete and continuous state, our systems achieve 99.3% average planning success, compared with 80.9% for the strongest domain-level program-generation baseline and 90.7% for the strongest instance-level baseline. Cost analyses show a favorable success–cost trade-off, and ablations confirm the contributions of the executable domain model, execution feedback, and component selection.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision Language Model, Planning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KwMaOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce RVPA, a framework that uses VLMs to encode scenes and environments into a different, cheaper representation that can subsequently be queried for in-domain queries without needing to re-use the VLM.

Critical strengths

Tradeoff: I appreciate the framing of Figure 1: trading off per-domain construction with per-inference cost. With more clarity in the text, I can appreciate the potential of exploring the pareto frontier in this regard. This allows downstream users to "tune" how important each criteria is: if you have many future domains you may want cheap domain construction, but if you are constrained to one domain with many inferences you will prefer the opposite.

Critical weaknesses

Method Clarity: The current text in S4 is very hard to follow. Even after multiple reads I am still unsure what each component looks like. Almost all relevant entities are under-described. For example: what is the state s, how is it represented? How is the renderer instantiated? What is the reader G? What do "plans" look like? How is "solvable" determined by the Candidate Planner? All of these entities are described at a high level and I still do not understand what they encode and how they encode it. Similarly, Figure 2 does little to help: the authors should replace the box that says "State" with an actual visualization of what the "state" is, likewise for the reader G and Dynamics. I understand the flow of the Figure, but there is not sufficient information to understand each component.

As the method section is severely underspecified, I find it difficult to contextualize the experiments. What is "more reusable code" (RQ1)? Does this refer to the Readers? Likewise there are no qualitative figures to help explain where the differences stem from in comparing the methods in experiments. In the papers' current state, I cannot evaluate the results in the experiments and do not understand why the proposed method outperforms the baselines.

Questions for authors

Please see the weakness. The paper is highly underspecified and describes things only at a high-level, this severely impedes the ability to assess it's soundness and contributions.

Reviewer_4q45OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes RVPA, a training-free framework in which a VLM writes a reusable visual-planning system for a new domain. The inputs are a natural-language domain specification, prescribed interfaces, and a single unlabeled practice scene. The VLM first builds an executable domain model: a state representation, a renderer (appearance model), and a dynamics simulator.

Grounding programs are then refined with a render-and-compare loop. Each candidate reader’s recovered state is rendered and compared to the observation by a separate VLM judge, and the final reader is chosen by pairwise agreement among K independently authored candidates. The planner is refined with a simulate-and-check loop against the authored dynamics model, on the practice state plus synthetic variants that model generates. After selection, all programs are frozen and solve new instances locally with no further VLM calls.

Experiments cover 15 synthetic domain variants (2D/3D, discrete/continuous), with 20 held-out instances each. RVPA reports 99.3% mean success, against 80.9% for the strongest domain-level baseline (direct code writing with a stronger backbone) and 90.7% for the strongest per-instance baseline (Astra instance code). Amortized cost is about 2 cents per instance, roughly an order of magnitude cheaper than strong per-instance methods. Ablations removing the renderer, grounding refinement, agreement-based selection, or planner refinement reduce success.

Critical strengths
  1. Amortizing VLM reasoning into frozen, locally executable perception and planning programs is relevant: it addresses inference cost, latency, and reproducibility at deployment.
  2. The domain model works as a “twin” in both directions. Rendering gives analysis-by-synthesis feedback for grounding, and simulation gives outcome-level feedback for planning. This is a sensible way to obtain correctness signals without reference solutions.
  3. RVPA is best or tied-best on all 15 variants. Its zero variance across five construction runs on 14/15 tasks contrasts sharply with the high variance of baselines (std up to 0.55).
  4. In continuous-state and 3D domains (tangram, sheet cutting, box packing, shelf filling), the gains are largest, which supports the value of explicit state and simulation.
  5. Domain-level code generation (several backbones, BofN, Self-Refine, Parsel, CodeTree, OpenHands) and per-instance methods are both covered, with explicit cost accounting.
  6. Full prompts are given, and the case study honestly reports failure modes of the feedback signals.
Critical weaknesses
  1. The supervision claims are inconsistent with the appendix. Section 3.1 says the VLM declares the state representation. Appendix C.1 instead says the state fields, coordinate conventions, action format, and goal condition are supplied inputs, and that “no separate model call … invents the state schema.” A human-designed state schema and interface is substantial domain engineering, and it weakens the “without manual pipeline redesign” claim.
  2. There is evidence of reference-state and test-pool leakage. Appendix C.1.1 states that renderer smoke checks used a known practice state. Appendix C.2.6 states that reader selection uses the unlabeled observation pool, so the evaluation scenes are not a separate split. This contradicts Eq. 4, which is defined only on o₀, and the “single practice scene” framing. The selection is then transductive, and its effect on the headline numbers is unquantified.
  3. The feedback signals are shown to be unreliable, yet they are credited with the gains. In the Blocks case study, the renderer uses 10 cm cubes against 5 cm in the scene and crops the top block, yet the perception judge returns “match.” The dynamics judge also wrongly penalized a correct rule and was “fixed” by changing the demonstration rather than the rule. This suggests success may come largely from strong prompting, supplied interfaces, and permitted tools rather than from the claimed closed-loop feedback. Per-domain diagnostics of judge accuracy are missing.
  4. The ablations do not match the claims. The abstract, introduction, and Section 4.4 claim to ablate the executable dynamics model, but Table 3 contains no “w/o dynamics” row; “w/o planner refinement” appears instead. Removing the renderer has no effect on 2D discrete domains. It is also unclear whether the ablations average multiple runs.
  5. Baseline parity is unclear. It is not stated whether domain-level baselines received the same state schema, prescribed interfaces, renderer/dynamics interfaces, packages (SAM, GroundingDINO, OWL-ViT, OR-Tools), 20 s solver budget, and synthetic practice variants. If not, the comparison partly measures scaffolding rather than method. Per-instance Luna methods are compared against far stronger direct baselines, and their low scores (≤46.7%) may reflect weak backbones rather than the per-instance paradigm.
  6. Reported numbers seems inconsistent. Table 5 covers 16 tasks for baselines but 15 for RVPA, the 16th task is unidentified, and the caption mentions “reference-planner use.” Baseline numbers also differ across tables: one-shot Astra is 0.77 in Table 5 vs. 0.81 in Table 1, and OpenHands is 0.58 vs. 0.60. Appendix C.2.6 notes that some recorded call counts exclude renderer and perceiver construction, which raises doubts about whether construction cost is complete.
  7. The evaluation scope is limited. All scenes are procedurally rendered and noiseless (depth is explicitly noise-free), with 20 test instances per domain drawn from the same generator as the practice scene. There is no test under distribution shift (more objects, different cameras or lighting, real images), which is the setting where a single-scene construction is most likely to fail.
  8. Positioning relative to prior work is thin. The paper does not discuss analysis-by-synthesis and inverse-graphics work, LLM-built world models and PDDL generation (e.g., Guan et al. 2023; WorldCoder is cited only in passing), or visual-programming systems (VisProg, ViperGPT, Code-as-Policies). It is also unclear how RVPA differs technically from Hao et al. 2026 (Simulation to Rules).
  9. The bibliography has 29 entries, and 13 of them (about 45%) are cited only as arXiv preprints. For an ICLR submission, this is a concern for three reasons. a) Some preprints have peer-reviewed versions. To my knowledge, several of the works cited as arXiv preprints have been published, and the published versions should be cited: b) There are duplicate entries, and this suggests the bibliography was not carefully curated. c) Key claims rest on unreviewed, very recent preprints.
  10. Several statements in the appendix conflict with the “single unlabeled practice scene, no labels” framing: a human-supplied state schema, reference-state access during renderer smoke tests, and reader selection over the evaluation observation pool. The evaluation is also limited to small, synthetic, noiseless benchmarks.
Questions for authors

Q0) Answer to the main criticisms discussed above.

Q1) The supplied state schema. Which parts of the state schema and interfaces are human-written per domain, and how many lines or hours of effort did this require for each of the 15 variants? Can you show results where the VLM truly proposes the schema?

Q2) Selection without the evaluation pool. What is the success rate when reader selection uses only o₀, or a disjoint unlabeled pool, rather than the evaluation pool? For which domains did renderer construction access a reference state?

Q3) How often did the perception and dynamics judges give incorrect verdicts across domains, measured against ground-truth states that are available offline? How sensitive are the final results to judge errors?

Q4) Agreement-based selection. How is Agree(sₖ, sⱼ) defined for continuous states (tolerances, matching of object identities)? What is the tie-break rule, and how often were ties or full disagreement observed?

Q5) What are the values of K, the numbers of dynamics and planner authors, the revision budgets, and the planner’s partial-credit and acceptance criteria? How do success and cost scale with K?

Q6) How many construction runs failed or exhausted the budget, and was any domain-specific prompt tuning done during development using test-set feedback?

Q7) How does a frozen system behave with sensor noise, occlusion, unseen object counts or sizes, or a camera pose different from the practice scene? Can it detect when its own reading is unreliable at deployment?

Q8) For combinatorial domains (Rubik’s cube, polyomino, pyramid), are the generated planners general algorithms (e.g., IDA*, CP-SAT encodings), and what are their worst-case runtimes on the test instances?

Q9) Does the framework retain its advantage with an open-weight backbone, and how does performance change with the reasoning-effort and token limits?

Q10) The 16th task and “reference-planner use.” What is the 16th task in Table 5, and what does “reference-planner use” refer to in its caption?

Reviewer_GmsuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a framework that uses a VLM to construct reusable perception and planning programs from a task specification and a practice scene. It uses reconstruction and simulation feedback to refine the programs without ground-truth solutions. The generated programs achieve 99.3% average success across 15 task variants.

Critical strengths
  • Reusing generated programs without further VLM calls is a useful direction, particularly for repeated planning tasks.
  • The results are strong across several types of structured planning problems, with substantial improvements over program-generation baselines.
Critical weaknesses
  • The generalization claim is not sufficiently supported. All evaluated tasks have explicit rules and controlled environments. It remains unclear whether the approach works with more realistic visual observations or changing task conditions.
  • The evaluation uses only 20 instances per task variant. Moreover, Appendix C.2.6 indicates that reader selection uses additional unlabeled observations, raising questions about the strict single-scene setting and evaluation independence.
Questions for authors
  • Can the authors evaluate on a strictly held-out set that is never used for program construction or selection?
  • How robust are the generated programs to visual perturbations, observation noise, or changes in scene complexity?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.