- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes an interface called Say Where, that aims to improve visual world-model planning by representing task instructions as sets of acceptable latent targets instead of single goal images. The framework supports logical compositions such as intersections (joint constraints) and disjunctions (alternative destinations) without retraining the underlying dynamics of the visual world model. They implement both rule-based groundings and learned text-to-cloud mechanisms. The paper shows empirical gains obtained by the proposed method on four simulated environments.
Critical strengths
-
The motivating problem is intuitive and conceptually clear. The fact that since goal images can over-specify tasks by fixing angle, position, and other incidental factors, despite the fact that real instructions have many free factors is understandable and intuitive. The proposed solution that treats a task as an acceptable set with standard set operations such as intersection, union, and unconstrained factors is also a meaningful solution.
-
The fact that the proposed method requires no additional retraining and can be integrated directly with a frozen visual world model and its cross entropy action search makes its application easy and efficient. However, I have other efficiency concerns that I mention in the weaknesses.
-
The diagnostic decomposition provided by measuring set agreement (IoU) separately from downstream control helps separate "the target set was wrong" from "the planner failed to reach a valid target." and reflects the paper’s proper experimental design.
Critical weaknesses
-
The scope of empirical experiments is rather limited. All experiments are limited to rather low-complexity, 2d or pseudo 3d simulated environments and only study a single pertained model architecture as the backbone, which is the LeWorldModel family. Since the modular design of the proposed method that can be combined with different world-model backbones and does not require additional retraining is a main strength of the method, but the experiments are just limited to one model. I believe evaluating the method with other popular world-model paradigms such as JEPA variants and diffusion-based world models would be very beneficial.
-
The paper’s novelty seems a bit limited to me. Goal sets, goal-conditioned planning to sets and language-to-latent-goal prediction (LAGO) are existing ideas that the paper also cites. I think the technical contribution is mainly applying a k-th-nearest-neighbor cost over a cloud of latents and training standard components. However, there is no empirical comparison to LAGE or any other language-conditioned latent-goal planner.
-
The Goalbank requires an offline database of 100000 embedded states per environment to sample from that may be computational intensive and impractical for real-world and lifelong robotic applications.
-
The composition studies are small and seem to be in the proposed method’s interest. The intersection study uses the easier PushT positional conjunctions, with eight of sixteen formulas selected by historical successs. The union study uses four fixed disjunctions, again in PushT only. The used baselines seem unnatural to me. Taking the cost-sum and max objectives aren’t natural ways to express a conjunction in a latent space. A stronger baseline could be. Learned or classifier-based joint goal predictor.
Questions for authors
Please find my concerns in the weaknesses. In addition to those, as the paper says, the instructions are template-derived and paraphrases came from Qwen with automatic acceptance and the IoU falls considerably under rephrasing. Have you considered evaluating on human-authored instructions, or at least instructions authored by different LLMs that include instructions with vocabulary or compositional structure unseen in training?
4 comments
Response to F9p4 (2/2): language, bank cost, and backbones
4. Language beyond the original generator. Llama-3-8B rewrote 80 instructions across four environments in conversational and vocabulary-focused styles, replacing the earlier Qwen generator. We tested each original and rewrite on 20 paired starts with both interfaces. Both interfaces have a 128-target cap, rank-one cost, paired planning randomness, and budgets of 50 simulator steps and 18,000 candidate-plan evaluations. Success is any post-action satisfaction of the original intended goal; invalid parses and empty clouds remain failures.
Qwen2.5-7B judged 86/160 rewrites meaning-preserving. Each cell shows original → rewritten success (%), averaging environments equally. Original rates differ because the checked subsets contain different instructions; n counts rewritten instructions.
| Rewrite population (n) | GoalBank | Llama + probe |
|---|---|---|
| Checked conversational (39) | 35.3 → 29.0 | 35.7 → 32.5 |
| Checked vocabulary-focused (47) | 40.0 → 29.2 | 42.7 → 33.0 |
| All conversational (80) | 43.5 → 33.9 | 42.1 → 27.5 |
| All vocabulary-focused (80) | 43.5 → 29.9 | 42.1 → 26.4 |
On checked rewrites, the point estimates fall for both methods and favor Llama. GoalBank's checked vocabulary-focused change is -10.8 points (95% CI [-18.76, -3.55]). The higher GoalBank rates over all outputs do not establish better language understanding: unchecked rewrites can change the intended meaning.
The semantic judge is fallible, and using Llama for generation and parsing may favor that baseline. There was no human annotation; we did not enforce that words or compositions were unseen in training. The evidence therefore concerns these synthetic instructions. The original membership-IoU diagnostic measures the separate question of goal-set fidelity.
5. Bank cost. On 320 paired cases, reducing the inference bank from 100,000 to 10,000 states changes success from 45.0% to 45.9% (+0.9 points; 95% CI [-5.70, 8.01]). A 1,000-state bank gives 40.6%. The reader and 128-target cap are unchanged; similar rates do not establish equivalence.
Median uncached goal construction, including text encoding, scoring, and selection, falls from 31.4 to 7.5 ms on an RTX 5090; float32 latent storage falls from 73.2 to 7.3 MiB. Offline encoding/training is excluded. This tests inference-bank requirements, not reduced training data or lifelong scalability.
6. Other backbones. A frozen DINO-model PushT pilot, also using proprioception, reaches 28.5% with one Oracle target and 33.5% with ten, across 200 paired cases. The gain's 95% CI is [-4.19, 15.60]. This is feasibility evidence only; broader backbone transfer, including diffusion models, remains untested.
CIs resample tasks and starting configurations/episodes, with trained models fixed.
Thank you for the additional results and clarifications in the rebuttal. My main remaining concern is incremental novelty over LAGO and the lack of comparison against LAGO-style methods under matched compute/data budgets. I find the main contribution as an empirical analysis and system interface rather than a completely new algorithm, which could be totally fine, but only if proper empirical comparison shows whether keeping the world model frozen and using terminal goal clouds outperforms or is competitive with LAGO-style methods. I also appreciate the authors’ additional experiments, particularly for shrinking the offline goal bank, but as they mentioned, this only tests inference time storage and retrieval. While I agree efficiency is a bigger concern at the inference time, training the goal bank reader still requires a massive offline dataset paired with ground-truth simulator predicates which weakens its practicality.
Thank you for distinguishing reader-training requirements from inference storage. We have now completed a separate retraining study, reported in Section 5.4, Table 5, and Appendix A.5 of the revised manuscript.
We trained three fresh readers per condition and environment, including fresh full-data controls: 24 fits, each with 20,000 optimization steps. Evaluation uses all 20 existing instructions per environment and the first 20 starts from each fixed 50-start holdout manifest, without inspecting outcomes. These are the established PushT configuration and Reacher episode holdouts. The smaller start population and fresh fits distinguish these rates from our earlier 50-start reader study (Table 4). Rank-one cost, a 128-target cap, 18,000 candidate-plan evaluations, and 50 simulator steps are shared. Means average environments, instructions, and the three reader seeds. The primary comparisons keep the full training-only retrieval bank fixed:
| Reader-training condition | PushT (%) | Reacher (%) | Mean (%) |
|---|---|---|---|
| Full data | 38.9 | 65.9 | 52.4 |
| 10k states, full instruction pool requested | 36.3 | 65.0 | 50.7 |
| 10k states, 10% of instructions requested | 35.1 | 62.1 | 48.6 |
| 1k states, 1% of instructions requested | 28.8 | 50.2 | 39.5 |
Full training uses 100,000/79,846 states and 30,599/28,298 instructions in PushT/Reacher. The 10k/10% condition retains 3,008/2,809 training instructions, plus 99 validation instructions each. For this reduced condition, labels and latent normalization use only the selected 10k states. The approximately 3,000 instructions produce 1.40M/1.20M distinct automatically labeled instruction–state pairs; 3,000 is the number of instructions, not labels. Unsupported training instructions are counted and omitted; every evaluation instruction remains.
The primary 10k/10%-minus-full difference is −3.8 percentage points (95% CI [−7.23, −0.62]). Thus, reducing training data has a measurable cost. Reducing only states gives −1.8 points [−4.74, 1.23]; the 1k/1% condition loses 13.0 points [−17.96, −8.09]. Intervals resample instructions and starting configurations/episodes with trained readers and data subsets fixed.
On the same evaluation cases, the 10k/10% reader remains 11.2 points above one accepted Oracle image (95% CI [3.21, 19.12]). Its +1.7-point difference from Llama plus probe is unresolved ([−5.94, 9.99]). Thus reduced training retains a benefit over the single-image reference on this subset, without establishing an advantage over the parser interface.
In a separate training-and-storage reduction arm, using those same 10k states as the 10k/10% reader's retrieval bank gives 50.4% mean success. This additional observed rate does not establish equivalence to full data. These experiments retain simulator-derived predicates, inherited physical calibration, and the existing frozen backbones. The study quantifies reader-data sensitivity; it does not establish reduced end-to-end supervision requirements or lifelong practicality.
Regarding LAGO, we agree that these controls cannot establish competitiveness with LAGO-style methods under matched compute/data budgets. That comparison remains missing, and Sections 6–7 now state this explicitly. Our supported contribution is the controlled empirical characterization of acceptable-outcome representations within a frozen visual planner, including their limitations, with specification fidelity evaluated separately from physical success.
Response to F9p4 (1/2): contribution and composition
Thank you. The expanded composition studies address formula selection and stronger objectives, while clarifying the contribution and its limits.
1. Contribution and LAGO. A goal cloud can leave incidental final-state details free while reusing an existing visual planner. Our empirical contribution is to measure when this choice helps under matched budgets, evaluating goal-set fidelity separately from physical success. LAGO trains a shared predictor for dynamics and language-conditioned intermediate subgoals; we study the interface with the visual predictor held fixed. We have not run a matched LAGO comparison.
2. Conjunctions without outcome-based formula selection. We evaluated all 32 prespecified formulas: 16 PushT formulas, including position/angle constraints, and 16 Reacher formulas combining fingertip goals with elbow posture. Each has 64 paired starts. Success requires satisfying both conditions at the same post-action simulator step.
Using the same learned scorer to construct a cloud instead of directly scoring predicted states improves success by 12.7 points (95% CI [7.56, 17.67]). This controls for scorer quality; it does not test every possible classifier architecture.
The physical-cost baselines decode physical quantities from predicted latents and penalize constraint violations. They achieve higher rates: one receives the correct rule, while Llama infers it from text; invalid parses count as failures. The classifier result establishes a benefit for that controlled comparison, not a universally best objective. These formulas were not systematically held out from reader training.
3. Disjunctions. On 32 formulas across four environments, with 32 paired starts each, union reaches 75.1%, random commitment 70.2%, and planning-based commitment 76.5%. All use 64 final targets. Union-minus-random is +4.9 points (95% CI [0.06, 9.45]); there is no demonstrated advantage over planning-based commitment.
All comparisons use 50 simulator steps and 18,000 candidate-plan evaluations, including target selection. CIs resample formulas and starting configurations/episodes, with trained models fixed.