- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces LCWM-Bench, a unified benchmark for language-conditioned world modeling across navigation, robotics, gaming, and driving, integrating 15 datasets with standardized action representations and evaluation metrics. It also proposes Tiresias, a selective imagination framework that generates candidate trajectories and ranks them using trajectory-level scores and world-model-generated visual rollouts. Experiments demonstrate improved trajectory prediction and video generation performance over existing baselines, with ablations highlighting the benefits of combining trajectory and visual ranking.
Critical strengths
- The benchmark covers diverse datasets across four domains with a standardized evaluation protocol.
- The proposed selective imagination framework efficiently combines trajectory ranking and visual verification.
- The paper provides extensive experiments and ablations demonstrating consistent performance improvements.
Critical weaknesses
W1. Insufficient Justification and Evidence for Unified Evaluation
The paper does not clearly establish why a unified cross-domain benchmark is necessary or how its evaluation goes beyond aggregating domain-specific trajectory metrics.
W2. Misalignment Between Evaluation Metrics and Task Success
The benchmark primarily measures similarity to a single reference trajectory, which does not necessarily reflect instruction fulfillment or actual task success.
W3. Unclear Necessity and Contribution of World-Model-Based Imagination
Although visual ranking improves trajectory selection, the experiments do not establish whether explicit future generation provides benefits beyond simpler scene-conditioned trajectory evaluation.
Questions for authors
Regarding W1 — Insufficient Justification for Unified Evaluation
Q1. What scientific insights does LCWM-Bench enable that cannot be obtained through separate domain-specific evaluations, particularly when the current unified score simply aggregates trajectory-matching metrics across heterogeneous domains?
Q2. Beyond reporting an aggregated score, can the authors demonstrate how the proposed unified evaluation reveals meaningful cross-domain capabilities, generalization patterns, or failure modes?
Regarding W2 — Misalignment Between Evaluation Metrics and Task Success
Q3. Why does LCWM-Bench rely on endpoint proximity and trajectory similarity rather than directly evaluating task success, particularly when multiple valid trajectories may satisfy the same instruction?
Regarding W3 — Unclear Novelty and Necessity of World-Model-Based Imagination
Q4. Given prior work on language-conditioned world modeling and video-based planning, such as UniPi [1], RoboDreamer [3], and LCVN [2], what are the key methodological advances of Tiresias beyond combining existing trajectory generation and ranking components?
Q5. Can the authors compare Tiresias against a direct scene-conditioned trajectory ranker that evaluates the initial observation, instruction, and candidate actions without generating future videos, to establish whether explicit imagination provides additional decision-making value?
[1] Du et al., Learning Universal Policies via Text-Guided Video Generation, NeurIPS 2023
[2] Dong et al., Language-Conditioned World Modeling for Visual Navigation, 2026
[3] Zhou et al., RoboDreamer: Learning Compositional World Models for Robot Imagination, ICML 2024.
W1 & Q1 & Q2:
Language-conditioned world modeling has so far been studied in individual domains with different action spaces, datasets, and metrics, so it has not been possible to test whether a method works across embodiments. LCWM-Bench provides one task definition, one action interface, and one evaluation protocol, and all methods are retrained under it. The overall score is only a summary. Table 1 reports SR, nDTW, ATE, and RPE for each domain, and the SR threshold and nDTW normalizer are defined by each domain's average step size so that the numbers are comparable across domains with very different motion scales (Appendix C.1).
The unified evaluation leads to several observations that are hard to obtain from separate domain-specific benchmarks. First, better prediction does not imply better planning: LCVN-WM has better FVD and rollout SSIM than LCVN-Uni on both splits (Table 2), but they have close planning scores on val seen (54 vs. 53), and LCVN-Uni is better on val unseen (42 vs. 38). Second, method rankings depend on the domain. Diamond is the weakest method in navigation, robotics, and driving, but in 3D gaming it is better than both policy baselines (SR 0.31 vs. 0.25 and 0.27 on val seen). Third, the generalization gap varies across domains. For Tiresias, SR drops by 0.09 from seen to unseen in gaming (held-out maps), compared with 0.19 in navigation and 0.22 in driving (held-out sources), and NWM, NWM (lang), LCVN-WM, and LCVN-Uni show the same pattern (Table 1). Fourth, on the language side, language-only conditioning works best for proposals (Table 4), and landmark-based instructions give higher SR than concise and intricate ones (0.65 vs. 0.59 and 0.56, Fig. 5). We also discuss the common failure cases in Sec. 5.3, including late-episode drift when the instruction does not specify motion magnitude, and imagined futures that deviate from the real scene beyond what the initial observation shows.
W2 & Q3:
SR in LCWM-Bench measures task success. It is a binary criterion on goal completion: an episode is successful only when the agent stops within one average step of the goal position, and for robotics the terminal gripper state must also be correct, because reaching the right position but releasing at the wrong time does not complete the manipulation (Sec. 3.2, Appendix C.1). This follows the common definition of success in instruction following (stopping within a threshold of the goal) and is the same as in LCVN.
SR does not depend on the path, so a trajectory that takes a different route but ends at the goal is still counted as a success. Path fidelity is measured by nDTW, which was proposed for instruction-conditioned navigation (Ilharco et al., 2019). Since its alignment is monotonic but not one-to-one, it allows different speeds and step counts along the same route and only penalizes a wrong route (Appendix C.1). Our instructions describe the route as well as the destination (for example, landmark-based instructions specify what to pass and where to turn), so following the described path is also part of fulfilling the instruction. The overall score is computed from SR and nDTW, and ATE and RPE are reported as additional references.
W3 & Q4 & Q5:
In Tiresias, the world model is used to evaluate candidates: we sample many action candidates and imagine the outcome of each one in the current scene, so that they can be compared under the instruction. This involves three designs that we verify in the ablations. Proposals are generated from language only and the scene-dependent selection is left to the second stage, which works better than conditioning the policy on the observation (Table 4). The two-stage funnel makes selection by imagination affordable, and imagining 8 ranked candidates is better than imagining 16 random ones (SR 0.602 vs. 0.460, Table A4). The rankers do not depend on the proposal source and improve MPC and NoMaD (lang) without fine-tuning (Table 7). Tiresias is also evaluated on four domains, while [1] [2] [3] focus on manipulation or navigation.
For the benefit of explicit imagination, Table 5 lists the selectors that do not use imagination (consensus 0.24, cluster-vote 0.18, Tr-Ranker 0.30), and all of them are below the fused ranker (0.39) on the same candidates. With the full funnel, adding the imagined evidence improves SR from 0.530 to 0.602 (Table A4), and we observe the same gain with MPC (0.49 → 0.56) and NoMaD (lang) (0.42 → 0.47) proposals (Table 7). We also find that using the scene directly without imagination is less effective. Adding the initial observation to the trajectory module decreases performance (0.56 vs. 0.60, Table 4), and GNM (lang) and NoMaD (lang), which map the observation and instruction directly to actions, obtain 36 and 40, compared with 63 for Tiresias (Table 1).