acceptodds
Under review as a conference paper at ICLR 2027

Say Where: Language and Goal Sets for Frozen Visual Planning

Abstract

A task often accepts many outcomes, while a goal image depicts only one. We study an interface that preserves these alternatives for planning with a frozen visual world model. Task specifications define sets of acceptable states; finite clouds of their visual representations supply the planner’s targets. This supports joint constraints, alternative destinations, precision, and factors left free by the task. Across four simulated environments, a shared-start comparison with the same nearest-goal cost increases average success from 31.1% with one accepted target to 52.3% with a large cloud. Targeted studies show that jointly valid targets improve simultaneous constraint satisfaction, and retaining alternatives improves attainment over committing to the initially nearest branch. We then learn to construct goal clouds from language through retrieval or generation. On the full endpoint suite, the retrieval interface achieves 46.3% success versus 31.6% for one accepted image, and is especially effective on region and property instructions. Set-agreement measurements distinguish successful specification from remaining grounding errors. These results connect richer task specifications to useful planning behavior without retraining the predictive dynamics, while identifying precise constraints and selection among valid alternatives as continuing challenges.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, joint-embedding predictive architecture (JEPA), reward-free planning, goal-conditioned planning latent-space planning, language-specified goals, goal specification, model predictive control, linear probes, grounding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_F9p4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an interface called Say Where, that aims to improve visual world-model planning by representing task instructions as sets of acceptable latent targets instead of single goal images. The framework supports logical compositions such as intersections (joint constraints) and disjunctions (alternative destinations) without retraining the underlying dynamics of the visual world model. They implement both rule-based groundings and learned text-to-cloud mechanisms. The paper shows empirical gains obtained by the proposed method on four simulated environments.

Critical strengths
  • The motivating problem is intuitive and conceptually clear. The fact that since goal images can over-specify tasks by fixing angle, position, and other incidental factors, despite the fact that real instructions have many free factors is understandable and intuitive. The proposed solution that treats a task as an acceptable set with standard set operations such as intersection, union, and unconstrained factors is also a meaningful solution.

  • The fact that the proposed method requires no additional retraining and can be integrated directly with a frozen visual world model and its cross entropy action search makes its application easy and efficient. However, I have other efficiency concerns that I mention in the weaknesses.

  • The diagnostic decomposition provided by measuring set agreement (IoU) separately from downstream control helps separate "the target set was wrong" from "the planner failed to reach a valid target." and reflects the paper’s proper experimental design.

Critical weaknesses
  • The scope of empirical experiments is rather limited. All experiments are limited to rather low-complexity, 2d or pseudo 3d simulated environments and only study a single pertained model architecture as the backbone, which is the LeWorldModel family. Since the modular design of the proposed method that can be combined with different world-model backbones and does not require additional retraining is a main strength of the method, but the experiments are just limited to one model. I believe evaluating the method with other popular world-model paradigms such as JEPA variants and diffusion-based world models would be very beneficial.

  • The paper’s novelty seems a bit limited to me. Goal sets, goal-conditioned planning to sets and language-to-latent-goal prediction (LAGO) are existing ideas that the paper also cites. I think the technical contribution is mainly applying a k-th-nearest-neighbor cost over a cloud of latents and training standard components. However, there is no empirical comparison to LAGE or any other language-conditioned latent-goal planner.

  • The Goalbank requires an offline database of 100000 embedded states per environment to sample from that may be computational intensive and impractical for real-world and lifelong robotic applications.

  • The composition studies are small and seem to be in the proposed method’s interest. The intersection study uses the easier PushT positional conjunctions, with eight of sixteen formulas selected by historical successs. The union study uses four fixed disjunctions, again in PushT only. The used baselines seem unnatural to me. Taking the cost-sum and max objectives aren’t natural ways to express a conjunction in a latent space. A stronger baseline could be. Learned or classifier-based joint goal predictor.

Questions for authors

Please find my concerns in the weaknesses. In addition to those, as the paper says, the instructions are template-derived and paraphrases came from Qwen with automatic acceptance and the IoU falls considerably under rephrasing. Have you considered evaluating on human-authored instructions, or at least instructions authored by different LLMs that include instructions with vocabulary or compositional structure unseen in training?

4 comments
AuthorsOpenReview ↗

Response to F9p4 (1/2): contribution and composition

Thank you. The expanded composition studies address formula selection and stronger objectives, while clarifying the contribution and its limits.

1. Contribution and LAGO. A goal cloud can leave incidental final-state details free while reusing an existing visual planner. Our empirical contribution is to measure when this choice helps under matched budgets, evaluating goal-set fidelity separately from physical success. LAGO trains a shared predictor for dynamics and language-conditioned intermediate subgoals; we study the interface with the visual predictor held fixed. We have not run a matched LAGO comparison.

2. Conjunctions without outcome-based formula selection. We evaluated all 32 prespecified formulas: 16 PushT formulas, including position/angle constraints, and 16 Reacher formulas combining fingertip goals with elbow posture. Each has 64 paired starts. Success requires satisfying both conditions at the same post-action simulator step.

Planning objectiveSuccess (%)
Known-valid joint goal cloud54.4
GoalBank joint goal cloud52.9
Same GoalBank reader used directly as an acceptance classifier40.2
Llama + probe: physical constraint cost61.2
Supplied rule + probe: physical constraint cost72.2

Using the same learned scorer to construct a cloud instead of directly scoring predicted states improves success by 12.7 points (95% CI [7.56, 17.67]). This controls for scorer quality; it does not test every possible classifier architecture.

The physical-cost baselines decode physical quantities from predicted latents and penalize constraint violations. They achieve higher rates: one receives the correct rule, while Llama infers it from text; invalid parses count as failures. The classifier result establishes a benefit for that controlled comparison, not a universally best objective. These formulas were not systematically held out from reader training.

3. Disjunctions. On 32 formulas across four environments, with 32 paired starts each, union reaches 75.1%, random commitment 70.2%, and planning-based commitment 76.5%. All use 64 final targets. Union-minus-random is +4.9 points (95% CI [0.06, 9.45]); there is no demonstrated advantage over planning-based commitment.

All comparisons use 50 simulator steps and 18,000 candidate-plan evaluations, including target selection. CIs resample formulas and starting configurations/episodes, with trained models fixed.

AuthorsOpenReview ↗

Response to F9p4 (2/2): language, bank cost, and backbones

4. Language beyond the original generator. Llama-3-8B rewrote 80 instructions across four environments in conversational and vocabulary-focused styles, replacing the earlier Qwen generator. We tested each original and rewrite on 20 paired starts with both interfaces. Both interfaces have a 128-target cap, rank-one cost, paired planning randomness, and budgets of 50 simulator steps and 18,000 candidate-plan evaluations. Success is any post-action satisfaction of the original intended goal; invalid parses and empty clouds remain failures.

Qwen2.5-7B judged 86/160 rewrites meaning-preserving. Each cell shows original → rewritten success (%), averaging environments equally. Original rates differ because the checked subsets contain different instructions; n counts rewritten instructions.

Rewrite population (n)GoalBankLlama + probe
Checked conversational (39)35.3 → 29.035.7 → 32.5
Checked vocabulary-focused (47)40.0 → 29.242.7 → 33.0
All conversational (80)43.5 → 33.942.1 → 27.5
All vocabulary-focused (80)43.5 → 29.942.1 → 26.4

On checked rewrites, the point estimates fall for both methods and favor Llama. GoalBank's checked vocabulary-focused change is -10.8 points (95% CI [-18.76, -3.55]). The higher GoalBank rates over all outputs do not establish better language understanding: unchecked rewrites can change the intended meaning.

The semantic judge is fallible, and using Llama for generation and parsing may favor that baseline. There was no human annotation; we did not enforce that words or compositions were unseen in training. The evidence therefore concerns these synthetic instructions. The original membership-IoU diagnostic measures the separate question of goal-set fidelity.

5. Bank cost. On 320 paired cases, reducing the inference bank from 100,000 to 10,000 states changes success from 45.0% to 45.9% (+0.9 points; 95% CI [-5.70, 8.01]). A 1,000-state bank gives 40.6%. The reader and 128-target cap are unchanged; similar rates do not establish equivalence.

Median uncached goal construction, including text encoding, scoring, and selection, falls from 31.4 to 7.5 ms on an RTX 5090; float32 latent storage falls from 73.2 to 7.3 MiB. Offline encoding/training is excluded. This tests inference-bank requirements, not reduced training data or lifelong scalability.

6. Other backbones. A frozen DINO-model PushT pilot, also using proprioception, reaches 28.5% with one Oracle target and 33.5% with ten, across 200 paired cases. The gain's 95% CI is [-4.19, 15.60]. This is feasibility evidence only; broader backbone transfer, including diffusion models, remains untested.

CIs resample tasks and starting configurations/episodes, with trained models fixed.

Thank you for the additional results and clarifications in the rebuttal. My main remaining concern is incremental novelty over LAGO and the lack of comparison against LAGO-style methods under matched compute/data budgets. I find the main contribution as an empirical analysis and system interface rather than a completely new algorithm, which could be totally fine, but only if proper empirical comparison shows whether keeping the world model frozen and using terminal goal clouds outperforms or is competitive with LAGO-style methods. I also appreciate the authors’ additional experiments, particularly for shrinking the offline goal bank, but as they mentioned, this only tests inference time storage and retrieval. While I agree efficiency is a bigger concern at the inference time, training the goal bank reader still requires a massive offline dataset paired with ground-truth simulator predicates which weakens its practicality.

Thank you for distinguishing reader-training requirements from inference storage. We have now completed a separate retraining study, reported in Section 5.4, Table 5, and Appendix A.5 of the revised manuscript.

We trained three fresh readers per condition and environment, including fresh full-data controls: 24 fits, each with 20,000 optimization steps. Evaluation uses all 20 existing instructions per environment and the first 20 starts from each fixed 50-start holdout manifest, without inspecting outcomes. These are the established PushT configuration and Reacher episode holdouts. The smaller start population and fresh fits distinguish these rates from our earlier 50-start reader study (Table 4). Rank-one cost, a 128-target cap, 18,000 candidate-plan evaluations, and 50 simulator steps are shared. Means average environments, instructions, and the three reader seeds. The primary comparisons keep the full training-only retrieval bank fixed:

Reader-training conditionPushT (%)Reacher (%)Mean (%)
Full data38.965.952.4
10k states, full instruction pool requested36.365.050.7
10k states, 10% of instructions requested35.162.148.6
1k states, 1% of instructions requested28.850.239.5

Full training uses 100,000/79,846 states and 30,599/28,298 instructions in PushT/Reacher. The 10k/10% condition retains 3,008/2,809 training instructions, plus 99 validation instructions each. For this reduced condition, labels and latent normalization use only the selected 10k states. The approximately 3,000 instructions produce 1.40M/1.20M distinct automatically labeled instruction–state pairs; 3,000 is the number of instructions, not labels. Unsupported training instructions are counted and omitted; every evaluation instruction remains.

The primary 10k/10%-minus-full difference is −3.8 percentage points (95% CI [−7.23, −0.62]). Thus, reducing training data has a measurable cost. Reducing only states gives −1.8 points [−4.74, 1.23]; the 1k/1% condition loses 13.0 points [−17.96, −8.09]. Intervals resample instructions and starting configurations/episodes with trained readers and data subsets fixed.

On the same evaluation cases, the 10k/10% reader remains 11.2 points above one accepted Oracle image (95% CI [3.21, 19.12]). Its +1.7-point difference from Llama plus probe is unresolved ([−5.94, 9.99]). Thus reduced training retains a benefit over the single-image reference on this subset, without establishing an advantage over the parser interface.

In a separate training-and-storage reduction arm, using those same 10k states as the 10k/10% reader's retrieval bank gives 50.4% mean success. This additional observed rate does not establish equivalence to full data. These experiments retain simulator-derived predicates, inherited physical calibration, and the existing frozen backbones. The study quantifies reader-data sensitivity; it does not establish reduced end-to-end supervision requirements or lifelong practicality.

Regarding LAGO, we agree that these controls cannot establish competitiveness with LAGO-style methods under matched compute/data budgets. That comparison remains missing, and Sections 6–7 now state this explicitly. Our supported contribution is the controlled empirical characterization of acceptable-outcome representations within a frozen visual planner, including their limitations, with specification fidelity evaluated separately from physical success.

Reviewer_ftmbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that a single goal image over-specifies a task when many final states would satisfy the instruction. Say Where represents acceptable outcomes as a set and approximates it with a finite cloud of latent visual goals for a frozen LeWorldModel planner. The formal-rule experiments use true state factors to select accepted bank states; the language interface either retrieves them from a learned bank or generates latents with a conditional flow model. Evaluation spans PushT, Reacher, TwoRoom, and OGBench-Cube. Targeted studies test conjunctions and disjunctions. Set-membership agreement is evaluated separately from control success, exposing a substantial weakness on precise point and angle instructions.

Critical strengths
  1. The formal goal-set experiments isolate a meaningful interface change. The main size study holds starts, instructions, interaction budget, and nearest-member cost fixed while varying the number of accepted targets. This supports the claim that preserving more valid outcomes can help the existing planner without retraining its dynamics. The selected conjunction experiment also tests simultaneous satisfaction. These results show that set operations have behavioral consequences rather than merely changing the representation.
  2. The paper measures both specification fidelity and control. GoalBank constructs usable target clouds from text and does particularly well on region/property instructions. Reporting IoU alongside physical success helps distinguish a language-grounding error from planner failure. The paper also reports less favorable cases and notes that a planning-based branch selector exceeds the union objective. This gives a more informative picture than an aggregate success number alone.
Critical weaknesses
  1. The language comparison changes more than the source of the goals. The one- and ten-image references use nearest-member cost, while large-cloud interfaces use fourth-nearest cost in the full endpoint suite. The size study controls this rank for the known-membership reference, but there is no analogous matched-rank comparison for the learned language interface. Further, GoalBank's 46.3% mean is close to ten accepted images at 45.6% and to the Llama parser plus probe at 46.5%. Therefore the evidence supports a useful language interface, but not a clear control-performance advantage over other ways of supplying several goals.
  2. Generalization is limited by the training/evaluation design. Banks come from the same source datasets as evaluation starts, the current episode is excluded from retrieval but may have appeared during interface training. On a configuration-disjoint PushT check, GoalBank obtains 35.0% versus 27.2% for one image, but the paired advantage is unresolved; paraphrased instructions also reduce set IoU from 90.6% to 70.7%. Together with the four simulated environments and template-derived instructions, these facts leave open how reliably the method handles new states and ordinary language.
Questions for authors
  1. On a representative matched subset, how do GoalBank, ten accepted images, and the strongest parser-plus-probe baseline compare with the same goal-cost rank, planning draws, and action budget? This would determine whether the language construction itself explains any control difference.
  2. What fraction of evaluation configurations or episodes appeared in interface training, as distinct from retrieval-bank membership? Can the authors clarify which disjointness is guaranteed in the main endpoint suite and which is guaranteed only in the PushT check?
  3. Is the GoalBank improvement over one image consistent across independent reader-training seeds on the main suite or a smaller, fixed subset? The current uncertainty intervals resample physical starts, not model training.
1 comment
AuthorsOpenReview ↗

Response to ftmb: fair comparisons, training exposure, and seeds

Thank you. The new comparisons address target count, cost rank, training exposure, and independent reader-training seeds separately.

1. Matched counts and cost rank. We reran a matched subset: 80 instructions across four environments, with 20 paired starts each. Every method uses distance to the nearest goal (rank one), paired planning randomness, 18,000 candidate-plan evaluations, and 50 simulator steps.

Goal source10 targets128 targets
Known-valid images (Oracle)43.6%48.1%
GoalBank37.9%43.5%
Llama + probe38.4%42.1%

GoalBank-minus-Llama at 128 targets is +1.4 points (95% CI [-4.27, 7.43]). GoalBank's 43.5% is also close to the ten-image reference's 43.6%. Changing only GoalBank's rank from one to four gives 43.7%. These controls do not establish superior planning from the learned construction.

2. Original training exposure. Excluding the current episode from retrieval did not make the original evaluation training-disjoint. Among unique evaluation episodes, the proportions represented in the reader's labeled training pool were 9.3% in PushT, 14.5% in Reacher, 20.2% in TwoRoom, and 14.5% in Cube. PushT configuration overlap was much higher: 165/167 (98.8%). These are pool memberships; individual training-batch visits were not logged.

3. Held-out data and independent training seeds. PushT uses 148 training and 37 held-out configurations. Reacher uses 8,000 training and 2,000 held-out episodes, with its bank, probe, and labels rebuilt from training episodes. Each has 20 instructions and 50 starts per instruction. TwoRoom/Cube use 20 starts per instruction from the main suite, without a new disjointness guarantee.

Each row reports three independently trained readers, including the existing seed zero where applicable. All entries are success percentages.

EvaluationOne Oracle targetLlama, 128 targetsGoalBank seeds 0 / 1 / 2
PushT: held-out configurations28.731.742.4 / 39.5 / 37.9
Reacher: held-out episodes43.762.271.3 / 66.4 / 69.3
TwoRoom: main subset30.838.030.3 / 29.8 / 33.0
Cube: main subset19.839.035.3 / 40.8 / 38.5

Gains over one image appear across all three seeds in PushT, Reacher, and Cube, but not TwoRoom. Three seeds do not precisely estimate training variability. The PushT/Reacher holdouts concern interface training; backbone-pretraining exposure remains unresolved. The rewording study also identifies a separate language limitation.

Success means reaching the condition at any post-action step within the 50-step budget; invalid parses remain failures. The CI resamples tasks and starting configurations/episodes, with trained models fixed.

Reviewer_Smg5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a framework for goal specification in frozen visual planning. Instead of a goal image which does not capture the variety of acceptable states, the authors propose using task specifications to define a "goal cloud": a set of satisfactory states in the goal latent space. This can then be used for planning while leaving the encoder and dynamics as before. They first demonstrate this framework with GoalOracle which knows the precise target and can generate acceptable images for the goal cloud. Here, they show that increasing the size of the goal cloud increases success rates across 4 task environments. They also extend this to study the composition (union or intersection) of goal clouds.

The paper then offers an approach for learning to build clouds from natural language. This is accomplished through on of two methods: GoalBank (identifying acceptable states from a given set) or GoalGen (generating acceptable states via flow matching). The authors compare these methods to using a single goal image, using ten goal images, and several existing methods for parsing the language instruction into a rule.

Critical strengths
  1. The core idea is nice: replacing a goal image with a goal cloud makes sense and the authors provide methods which do no require significant retraining.
  2. In section 5, measuring IoU and success rate separately is a clean way of separating the ability to capture goals from task performance. It also allows the authors to distinguish between types of instructions where their method performs well (region, property, and proximity) and those for which it struggles (point and angle).
Critical weaknesses
  1. GoalBank and GoalGen don't seem to add much when compared to other methods like the Llama parse + probe. The latter matches or beats GoalBank and GoalGen in the paired tests (table S21), has equal or better specification fidelity (table S23), and is more robust to rephrasing (table S18). Thus, the more complete analysis in the supplemental casts doubt on the practicality of the methods.
  2. In several places, the comparisons between the authors' methods and alternatives seem unfair. For example:
  • The comparison to goal images seems lop-sided when the other methods can have up to 1,024 targets rather than one or ten. This seems like a significant confound for some of the headline results
  • In the composition results, union is primarily compared to nearest-branch commitment, but when compared to random commitment the 95% interval contains 0 (table S22).
Questions for authors
  1. What is the benefit of GoalBank over Llama parser + probe? Is it enough to offset the need to train GoalBank for each task? Do your results point to real settings in which GoalBank would be preferable?
  2. For the single goal image comparison, rather than using a random image or the nearest initial image, could you compare to using the single accepted image with lowest predicted planning cost (something like your "planning-based selector")? That would allow us to distinguish the benefit of a goal cloud from the benefit of a well-chosen single target.
1 comment
AuthorsOpenReview ↗

Response to Smg5: single-target selection and matched controls

Thank you. We added the planning-selected single-image comparison you requested, together with matched-size language controls.

1. A goal cloud versus a well-chosen single target. On 80 instructions across four environments, with 20 paired starts each, all methods use the same nearest-goal cost, 50 simulator steps, and 18,000 candidate-plan evaluations. The selectors access the same 128 known-valid targets as the cloud. An initial 1,200-evaluation search selects one target; this search counts toward the common budget. It approximates target selection rather than exhaustively optimizing every target.

Target choiceSuccess (%)
Planning-selected single target, kept throughout42.1
Planning-selected single target, reconsidered at each replan45.2
128-target cloud48.1

The cloud gains 6.0 percentage points over keeping one target (95% CI [1.84, 10.12]). Allowing reselection reduces the gap to 2.9 points ([-1.10, 6.97]); that comparison remains uncertain.

2. Matched target counts. Under that same protocol:

Goal source10 targets128 targets
Known-valid images (Oracle)43.6%48.1%
GoalBank37.9%43.5%
Llama + probe38.4%42.1%

The observed GoalBank-minus-Llama difference at 128 targets is +1.4 points (95% CI [-4.27, 7.43]), without an established advantage. The rewording test likewise does not establish better robustness. Accordingly, we present GoalBank as a compact text-to-goal reader, without claiming superior language robustness. GoalGen remains exploratory.

3. Practical benefit and training cost. GoalBank trains one reader per environment across many instructions, not one per instruction. Its 4.56M-parameter reader avoids an LLM parser and physical-state decoder at inference. The new PushT/Reacher fits took 3.7–4.2 minutes each on an RTX 5090, excluding bank/label preparation. The intended use is repeated planning in a fixed environment with simulator-derived labels: the trained reader and stored bank are reused across instructions. An end-to-end speed advantage over Llama remains unmeasured.

4. Union versus random commitment. Across 32 disjunctions in four environments, union reaches 75.1%, random commitment 70.2%, and planning-based commitment 76.5%, with equal target counts and planning budgets. Union-minus-random is +4.9 points (95% CI [0.06, 9.45]), with a lower bound close to zero. This suggests a modest benefit over random commitment on this suite, without a demonstrated advantage over planning-based commitment.

Success means reaching the requested condition at any post-action step within the 50-step budget; invalid parses remain failures. CIs resample tasks and starting configurations/episodes, with trained models fixed.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of completed controls and scope of the contribution

We thank the reviewers for identifying the controls needed to separate the benefit of goal sets from the quality of a particular language interface. Our central question is how the representation of acceptable outcomes affects planning when the visual predictor is frozen.

In the new matched evaluation, the 128-target Oracle reaches 48.1% success, versus 42.1% for a planning-selected single target kept throughout. Reselecting that target raises the baseline to 45.2%, and the remaining difference is uncertain. The additional training-seed, disjoint-data, and composition studies further characterize where the interface helps and where it fails.

These results support a focused empirical contribution: evaluating goal representation under controlled planning budgets, while measuring goal-set fidelity separately from physical success. They do not establish a general advantage of GoalBank over parser-based interfaces or structured physical costs. The replies below report the completed controls, training exposure, language degradation, and remaining limitations.

AuthorsOpenReview ↗

Manuscript revised to incorporate the completed controls

We have revised the manuscript to incorporate the completed controls. Table 1 presents matched planning-selected targets and language interfaces; Table 2 covers all 32 conjunctions with classifier and physical-cost baselines. Tables 4–5 add independent readers, interface-training holdouts, and the new reader-training-data ablation. Appendix A gives the protocols, actual supervision counts, additional controls, and uncertainty conventions. The abstract and Sections 6–7 now reflect the supported contribution and remaining limitations, including the missing matched LAGO comparison. Figure 3's caption specifies its 100/200/50-step qualitative recording budgets and excludes these examples from quantitative benchmark rates. Historical and new evaluations are identified separately throughout.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.