acceptodds
Withdrawn from ICLR 2027

Tiresias: Towards Language-Conditioned World Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

World models enable planning through internal simulation, while natural language provides intuitive task specification. Yet, language-conditioned world modeling remains fragmented across domains with disparate action spaces, datasets, and evaluation protocols. We introduce LCWM, a shared task that maps an initial agent-view observation and a language instruction to an action sequence that fulfills the instruction. Its accompanying benchmark, LCWM-Bench, integrates 15 datasets across navigation, robotics, 3D gaming, and driving under a shared action interface and evaluation protocol, pairing 142,987 episodes with 428,961 instructions in three styles. We further propose Tiresias, an agent that plans via selective imagination: a language-conditioned diffusion policy generates candidate trajectories, a trajectory ranker shortlists them, and a video-pretrained world model imagines their visual consequences. A visual ranker evaluates these rollouts, complementing trajectory scores to select the final plan. Tiresias achieves competitive planning and imagination performance on LCWM-Bench. Ablations demonstrate the complementary benefits of trajectory ranking and visual verification, while experiments with alternative proposal methods show that the trained rankers remain effective beyond our diffusion policy. Together, LCWM-Bench and Tiresias provide a unified evaluation setting and an effective starting point for studying language-guided planning through world models.

Primary area:
generative models
Keywords:
World Models, Language Conditioning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SWgTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LCWM-Bench, a unified benchmark for language-conditioned world modeling across navigation, robotics, gaming, and driving, integrating 15 datasets with standardized action representations and evaluation metrics. It also proposes Tiresias, a selective imagination framework that generates candidate trajectories and ranks them using trajectory-level scores and world-model-generated visual rollouts. Experiments demonstrate improved trajectory prediction and video generation performance over existing baselines, with ablations highlighting the benefits of combining trajectory and visual ranking.

Critical strengths
  1. The benchmark covers diverse datasets across four domains with a standardized evaluation protocol.
  2. The proposed selective imagination framework efficiently combines trajectory ranking and visual verification.
  3. The paper provides extensive experiments and ablations demonstrating consistent performance improvements.
Critical weaknesses

W1. Insufficient Justification and Evidence for Unified Evaluation

The paper does not clearly establish why a unified cross-domain benchmark is necessary or how its evaluation goes beyond aggregating domain-specific trajectory metrics.

W2. Misalignment Between Evaluation Metrics and Task Success

The benchmark primarily measures similarity to a single reference trajectory, which does not necessarily reflect instruction fulfillment or actual task success.

W3. Unclear Necessity and Contribution of World-Model-Based Imagination

Although visual ranking improves trajectory selection, the experiments do not establish whether explicit future generation provides benefits beyond simpler scene-conditioned trajectory evaluation.

Questions for authors

Regarding W1 — Insufficient Justification for Unified Evaluation

Q1. What scientific insights does LCWM-Bench enable that cannot be obtained through separate domain-specific evaluations, particularly when the current unified score simply aggregates trajectory-matching metrics across heterogeneous domains?

Q2. Beyond reporting an aggregated score, can the authors demonstrate how the proposed unified evaluation reveals meaningful cross-domain capabilities, generalization patterns, or failure modes?

Regarding W2 — Misalignment Between Evaluation Metrics and Task Success

Q3. Why does LCWM-Bench rely on endpoint proximity and trajectory similarity rather than directly evaluating task success, particularly when multiple valid trajectories may satisfy the same instruction?

Regarding W3 — Unclear Novelty and Necessity of World-Model-Based Imagination

Q4. Given prior work on language-conditioned world modeling and video-based planning, such as UniPi [1], RoboDreamer [3], and LCVN [2], what are the key methodological advances of Tiresias beyond combining existing trajectory generation and ranking components?

Q5. Can the authors compare Tiresias against a direct scene-conditioned trajectory ranker that evaluates the initial observation, instruction, and candidate actions without generating future videos, to establish whether explicit imagination provides additional decision-making value?

[1] Du et al., Learning Universal Policies via Text-Guided Video Generation, NeurIPS 2023

[2] Dong et al., Language-Conditioned World Modeling for Visual Navigation, 2026

[3] Zhou et al., RoboDreamer: Learning Compositional World Models for Robot Imagination, ICML 2024.

1 comment

W1 & Q1 & Q2:

Language-conditioned world modeling has so far been studied in individual domains with different action spaces, datasets, and metrics, so it has not been possible to test whether a method works across embodiments. LCWM-Bench provides one task definition, one action interface, and one evaluation protocol, and all methods are retrained under it. The overall score is only a summary. Table 1 reports SR, nDTW, ATE, and RPE for each domain, and the SR threshold and nDTW normalizer are defined by each domain's average step size so that the numbers are comparable across domains with very different motion scales (Appendix C.1).

The unified evaluation leads to several observations that are hard to obtain from separate domain-specific benchmarks. First, better prediction does not imply better planning: LCVN-WM has better FVD and rollout SSIM than LCVN-Uni on both splits (Table 2), but they have close planning scores on val seen (54 vs. 53), and LCVN-Uni is better on val unseen (42 vs. 38). Second, method rankings depend on the domain. Diamond is the weakest method in navigation, robotics, and driving, but in 3D gaming it is better than both policy baselines (SR 0.31 vs. 0.25 and 0.27 on val seen). Third, the generalization gap varies across domains. For Tiresias, SR drops by 0.09 from seen to unseen in gaming (held-out maps), compared with 0.19 in navigation and 0.22 in driving (held-out sources), and NWM, NWM (lang), LCVN-WM, and LCVN-Uni show the same pattern (Table 1). Fourth, on the language side, language-only conditioning works best for proposals (Table 4), and landmark-based instructions give higher SR than concise and intricate ones (0.65 vs. 0.59 and 0.56, Fig. 5). We also discuss the common failure cases in Sec. 5.3, including late-episode drift when the instruction does not specify motion magnitude, and imagined futures that deviate from the real scene beyond what the initial observation shows.

W2 & Q3:

SR in LCWM-Bench measures task success. It is a binary criterion on goal completion: an episode is successful only when the agent stops within one average step of the goal position, and for robotics the terminal gripper state must also be correct, because reaching the right position but releasing at the wrong time does not complete the manipulation (Sec. 3.2, Appendix C.1). This follows the common definition of success in instruction following (stopping within a threshold of the goal) and is the same as in LCVN.

SR does not depend on the path, so a trajectory that takes a different route but ends at the goal is still counted as a success. Path fidelity is measured by nDTW, which was proposed for instruction-conditioned navigation (Ilharco et al., 2019). Since its alignment is monotonic but not one-to-one, it allows different speeds and step counts along the same route and only penalizes a wrong route (Appendix C.1). Our instructions describe the route as well as the destination (for example, landmark-based instructions specify what to pass and where to turn), so following the described path is also part of fulfilling the instruction. The overall score is computed from SR and nDTW, and ATE and RPE are reported as additional references.

W3 & Q4 & Q5:

In Tiresias, the world model is used to evaluate candidates: we sample many action candidates and imagine the outcome of each one in the current scene, so that they can be compared under the instruction. This involves three designs that we verify in the ablations. Proposals are generated from language only and the scene-dependent selection is left to the second stage, which works better than conditioning the policy on the observation (Table 4). The two-stage funnel makes selection by imagination affordable, and imagining 8 ranked candidates is better than imagining 16 random ones (SR 0.602 vs. 0.460, Table A4). The rankers do not depend on the proposal source and improve MPC and NoMaD (lang) without fine-tuning (Table 7). Tiresias is also evaluated on four domains, while [1] [2] [3] focus on manipulation or navigation.

For the benefit of explicit imagination, Table 5 lists the selectors that do not use imagination (consensus 0.24, cluster-vote 0.18, Tr-Ranker 0.30), and all of them are below the fused ranker (0.39) on the same candidates. With the full funnel, adding the imagined evidence improves SR from 0.530 to 0.602 (Table A4), and we observe the same gain with MPC (0.49 → 0.56) and NoMaD (lang) (0.42 → 0.47) proposals (Table 7). We also find that using the scene directly without imagination is less effective. Adding the initial observation to the trajectory module decreases performance (0.56 vs. 0.60, Table 4), and GNM (lang) and NoMaD (lang), which map the observation and instruction directly to actions, obtain 36 and 40, compared with 63 for Tiresias (Table 1).

Reviewer_PM5yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose LCWM-Bench, which integrates 15 existing datasets from four domains: navigation, robot manipulation, 3D games, and autonomous driving. It constructs a cross-domain benchmark through unified action representation, language instruction annotation, and evaluation metrics. Furthermore, it proposes a selective mechanism to filter trajectories, uses a world model to generate corresponding future visual information, and employs a visual ranker.

Critical strengths
  1. Unlike previous single-domain benchmarks, this benchmark integrates datasets from different domains, aiming to explore the general capabilities of Language-Conditioned World Modeling.
  2. The experimental and ablation experiments were quite thorough. The paper compared the planning methods with various existing planning methods and world models, and evaluated the planning performance and future video generation quality respectively. 3.The two-stage selection mechanism helps improve inference efficiency. The paper also provides a complete analysis of the impact of candidate quantity on performance.
  3. The trained Ranker can be applied to other candidate generation methods. This indicates that the proposed candidate selection mechanism does not completely depend on a specific trajectory generation model and has certain modularity and reusability potential.
Critical weaknesses
  1. The construction of benchmarks involves the integration of existing datasets, which limits both innovation and the amount of engineering work required.
  2. The Visual Ranker is a core component of the paper's method, but the authors do not adequately explain how the supervision labels for these scores are constructed, or how the model learns and distinguishes these semantic attributes. In particular, whether Motion Validity truly reflects physical feasibility and whether Semantic Consistency truly reflects scene consistency lacks independent quantitative verification. Existing ablation experiments only demonstrate that the Visual Ranker improves trajectory selection performance, but do not prove that each scorehead possesses its claimed semantic meaning. This affects the method's interpretability, reproducibility, and technical credibility.
  3. The Trajectory Ranker scores candidates solely based on language commands and candidate trajectories, completely disregarding information such as road structure within the current scene. Since the feasibility of the same language command and trajectory can vary drastically across different scenarios, this scene-independent filtering mechanism may fail. Although the authors have demonstrated that this mechanism outperforms a random ranker, a comparison with scene-aware ranker methods is still lacking.
  4. The paper mainly uses open-loop evaluation metrics and cannot verify the model's performance in closed-loop operation.
Questions for authors
  1. How to ensure that the visual ranker is a meaningful ranking.
  2. The trajectory ranker should be compared with the scene-aware ranker.
  3. It is recommended to add training on three domain datasets and testing on a fourth, unseen domain dataset.
1 comment

W1:

The 15 sources cannot be used for this task in their released form. They differ in control signals (wheel odometry, absolute end-effector poses, camera extrinsics, vehicle-pose matrices, quaternion keyframes), frame rates, resolutions, and episode structure, and none of them comes with instructions in the three styles. To build LCWM-Bench (Sec. 3.2, Appendix A), we converted every episode into per-step increments in the local frame, deriving them from consecutive pose differences when explicit actions were not available, and normalized them by the per-source average step size, with extra channels for elevation, velocity, and 6-DoF gripper control (3–7 dimensions). We then filtered backward, near-static, and overly short clips and segmented the rest into semantically coherent episodes, which gives 142,987 episodes (≈4.4M frames). For the language side, we wrote 150 seed triplets by hand, generated 428,961 instructions in three styles with domain-specific prompts, and had human annotators verify and correct each of them against the video. A human study on 1,200 episodes with three ratings per item gives 5.37/6. Finally, we defined source-level seen/unseen splits (TartanDrive, LIBERO, nuScenes, and held-out game maps) and metrics normalized per domain, and retrained seven baselines under this protocol.

The table below compares LCWM-Bench with related benchmarks.

BenchmarkDomain(s)Language instruction → action sequenceWorld-model imaginationShared cross-domain action interface
LoopNav (Lian et al., 2025)Gaming (Minecraft)✗✓✗
EWMBench (Yue et al., 2025)Manipulation✗✓✗
WorldLens (Liang et al., 2025)Driving✗✓✗
LCVN (Dong et al., 2026)Navigation✓✓✗
LCWM-Bench (ours)Navigation, robotics, 3D gaming, driving✓✓✓

LCVN is the closest prior benchmark. It covers navigation only, with 39,016 trajectories and 117,048 instructions. LCWM-Bench is 3.7× larger and adds three domains with 4–7-dimensional action spaces, so that the same method can be evaluated on language-guided planning and imagination across different embodiments.

W2 & Q1:

All four V-Ranker heads are trained with trajectory-derived targets, which are computed offline from the candidate being scored and its ground-truth trajectory in the same way as for the Tr-Ranker (Appendix B.4). The fused score is trained with listwise and pairwise margin-ranking losses, and MSE losses are applied to the fused and auxiliary predictions. Each head is thus regressed onto a measurable target. The three auxiliary heads have smaller weights (0.5, 0.5, 0.25) than the overall head and act as auxiliary supervision (Appendix B.5).

As for whether the ranking is meaningful, the V-Ranker sees neither the candidate actions nor the ground truth at test time, so this can be measured directly from the selection results. With the same eight random candidates, selecting by the V-Ranker alone improves SR from 0.13 (random) to 0.28 and reduces ATE from 2.35 to 1.20 (Table 5). It is also better than the Qwen3.6 verifier with MVoT-inspired reasoning on the same rollouts (SR 0.28 vs. 0.26, nDTW 0.51 vs. 0.49). Fusing it with the Tr-Ranker improves over the Tr-Ranker alone, from 0.30 to 0.39 on the same candidates (Table 5) and from 0.530 to 0.602 with the full 128 → 8 funnel (Table A4). The same V-Ranker also improves MPC (0.49 → 0.56) and NoMaD (lang) (0.42 → 0.47) proposals without fine-tuning (Table 7).

W3 & Q2:

Table 5 includes scene-aware selectors. The Qwen3.6 verifier and the V-Ranker both score candidates based on rollouts imagined from the current observation. On the same eight candidates, the Tr-Ranker (SR 0.30, nDTW 0.53, ATE 1.14, RPE 0.17) is better than the Qwen3.6 verifier (0.26 / 0.49 / 1.28 / 0.19) and the V-Ranker (0.28 / 0.51 / 1.20 / 0.19) on all four metrics, and it needs no imagination.

Scene information is used in the second stage. The Tr-Ranker only serves as a coarse filter over the 128 candidates, and the final selection among the eight shortlisted ones relies on the world model, which is conditioned on the initial frame through its VAE latent and DINOv3 features, together with the V-Ranker (Sec. 4.4). In practice the first-stage filter keeps good candidates: the top-8 shortlist retains 85% of the full-pool oracle SR (Table 6), and shortlisting 8 candidates with the Tr-Ranker (SR 0.602) is better than randomly shortlisting 16 (0.460) (Table A4). We also tried adding the observation at the proposal stage, and an LDP conditioned on both language and observation is worse than the language-only one, both without ranking (SR 0.11 vs. 0.13) and with the full pipeline (0.56 vs. 0.60) (Table 4).

Reviewer_kibBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces LCWM, the task of predicting an action sequence from an initial observation and a language instruction. The accompanying benchmark, LCWM-Bench, integrates 15 datasets from navigation, robotic manipulation, 3D gaming, and driving through a shared action interface with domain-specific control channels, and pairs each episode with instructions in three styles. Tiresias generates candidate trajectories with a language-only diffusion policy (LDP), shortlists them with a Tr-Ranker that uses no visual input, and imagines the shortlisted futures with a Wan2.2-based world model. Trajectory and V-Ranker scores are then fused to select the final trajectory. In offline trajectory prediction, Tiresias attains the highest overall score among the compared methods on both the seen and unseen splits. The paper also evaluates imagination quality, candidate budgets, and the applicability of the rankers to alternative proposal generators.

Critical strengths
  1. Coverage and construction of the benchmark. Integrating 15 sources with heterogeneous action definitions, together with instructions in three styles that the authors report as human-verified, represents a substantial effort. The shared interface and evaluation protocol provide a useful basis for cross-domain comparison. If the data and split files are released as planned, the benchmark would be a reusable resource.
  2. Systematic analysis of candidate filtering. The paper varies pool size, shortlist size, and fusion weights, and compares learned and random filtering at matched imagination budgets. Imagining eight ranked candidates outperforms imagining sixteen random candidates, which supports the selective-imagination design. Fusing trajectory and visual scores also improves results across different proposal generators.
Critical weaknesses
  1. The contribution of scene-dependent information is not adequately isolated. The non-visual LDP + Tr-Ranker already reaches an SR of about 0.53 on val seen, higher than every external baseline in the main comparison (the strongest, LCVN-WM, has a macro-averaged SR of about 0.48 across the four domains). Adding the visual branch raises SR to about 0.60, and using both instruction and DINOv3 cross-attention conditions in the world model outperforms retaining either alone, which indicates that the visual branch provides a benefit. Nevertheless, the comparison with external baselines admits at least two explanations: these baselines may be disadvantaged under this protocol, or a substantial portion of the benchmark may be solvable from language and learned trajectory priors alone. Because annotation uses trajectory metadata, reporting the non-visual baseline by domain and instruction style would be particularly informative. In addition, the V-Ranker fuses statistical descriptors of imagined latents with instruction, scene, and domain features, and the current controls do not distinguish scene-dependent imagined evidence from motion cues directly available in the candidate actions. Furthermore, SR assesses endpoint error and, for robotics, the terminal gripper state; it does not directly establish object-interaction or obstacle-avoidance success. A matched ranker that replaces imagined descriptors with candidate-action features, while retaining the same context and supervision, would help clarify the mechanism. Otherwise, the interpretation of visual verification should be narrowed.
  2. Training details and reproducibility materials are incomplete. The paper describes the V-Ranker's output heads but not its supervision labels or training losses. The proposal sampling and optimization procedure of the MPC variant are likewise unspecified. The Wan2.2 checkpoint and parameter count are not reported, whereas NWM and LCVN-WM use backbones of approximately 1B parameters trained from scratch on this benchmark; gains over these baselines therefore remain confounded with differences in pretraining and model scale. The candidate-funnel ablations report standard deviations, but the main baseline comparisons provide no corresponding uncertainty estimates. According to the manuscript, the supplementary sample only illustrates the data format, and the full benchmark and split files are planned for release upon acceptance; the sample is therefore insufficient to reproduce the quantitative results.
Questions for authors
  1. (Additional analysis) Could you report LDP + Tr-Ranker results by domain and instruction style, including the unseen split, to clarify the conditions under which the visual branch provides a benefit?
  2. (Clarification and additional analysis) What supervision labels and losses are used to train each V-Ranker head? How does a matched ranker that replaces imagined descriptors with candidate-action features perform under the same candidate pool, context, and supervision?
  3. (Clarification) Please specify the Wan2.2 checkpoint and parameter count, the MPC proposal and optimization procedure, and the number of runs and uncertainty estimates for the main results. In addition, are the LDP and Tr-Ranker shared across domains or trained separately, and how are different action dimensions handled?
1 comment

W1 & Q1:

All baselines were run in the setting they were designed for. LCVN-WM and LCVN-Uni (Dong et al., 2026) take an initial egocentric observation and a language instruction and output an action sequence, with no goal image, which is the same input and output as LCWM. Diamond, NWM, and NWM (lang) are paired with LCVN-AC to produce actions, as in LCVN (Sec. 5.1). We used the official implementations and best reported configurations for the world-model baselines (Appendix C.2) and retrained every baseline on the LCWM-Bench train split under the shared action interface. The trends reported for these models also hold on our benchmark: LCVN-WM gives more coherent rollouts (Table 2), and LCVN-Uni transfers better to unseen sources (42 vs. 38 on val unseen, Table 1).

On the annotation, trajectory metadata is only used to help the annotation model tell the direction of motion. Our prompts state that the trajectory map is "for judging direction changes only" (Listings A1, A3, A5) and that coordinates or metadata values must not be written into the instruction (Listing A4). Each generated instruction was then checked and corrected by human annotators against the video (Appendix A.2), and the human study on 1,200 episodes gives an average rating of 5.37/6 (Appendix A.3). The examples in Table A1 show what the released instructions look like: they describe visible scene content and qualitative maneuvers, and contain no displacements, angles, step counts, or coordinates. Describing the route in language is how the task is defined, as in other instruction-following benchmarks.

Regarding the V-Ranker, it does not take the candidate action sequence as input. The only candidate-specific input is the imagined rollout, and the instruction, scene, and domain features are the same for all candidates within an episode (Appendix B.5). A ranker that scores candidate-action features against the instruction with trajectory-derived supervision is what the Tr-Ranker does, so the comparison the reviewer asks for is in Table 5. On the same eight candidates, the Tr-Ranker gets SR 0.30, the V-Ranker 0.28, and their fusion 0.39. If the V-Ranker were only picking up motion cues already present in the actions, fusing it with the Tr-Ranker would not bring a 9-point gain. We see similar gains with all three proposal sources in Table 7 (LDP 0.53 → 0.60, MPC 0.49 → 0.56, NoMaD (lang) 0.42 → 0.47).

W2 & Q2 & Q3:

The V-Ranker is described in Appendix B.5, including the rollout descriptor (per-channel temporal mean, standard deviation, and mean absolute frame-to-frame difference of the imagined latents, excluding the anchor frame), the context features, the fusion MLP, the four heads, and the 1 : 0.5 : 0.5 : 0.25 weighting. Its training targets are trajectory-derived, as for the Tr-Ranker (Appendix B.4): they are computed offline from each training candidate and its ground-truth trajectory. We train the fused score with listwise and pairwise margin-ranking losses and apply MSE losses to the fused and auxiliary predictions.

The world model is initialized from Wan2.2-TI2V-5B (5B parameters), and using a video-pretrained world model is part of our method design (Sec. 4.2). Among the baselines, LCVN-Uni is fine-tuned from the pretrained Anole-7B (Appendix C.2), which is larger than our backbone, and Tiresias is 10 and 9 points higher on val seen and val unseen (Table 1) with lower latency (36 s vs. 85 s, Table A5). In addition, the planning gain does not rely on the video backbone. As the reviewer points out in W1, the LDP (trained from scratch) with the Tr-Ranker alone already reaches SR 0.53 (128 → 1 in Table A4) without any video model, which is higher than all external baselines. We also replaced all umT5 text encoders with CLIP, and the overall score changes from 63 to 62 (Appendix C.3).

For uncertainty, inference uses a fixed set of noise seeds (Sec. 5.1), and Table A4 gives the standard deviation over runs for the default configuration used in Table 1 (SR 0.602 ± 0.013, nDTW 0.658 ± 0.004). The SR gap to the strongest baseline (0.60 vs. 0.48, macro-averaged) is about nine times this standard deviation.

The LDP and Tr-Ranker are shared across the four domains. They work on the shared action interface in Sec. 3.1, where all domains use the base parameterization (ut,φt)(u_t, φ_t) and domain-specific channels are appended (3–7 dimensions, Table A2). The denoising loss is averaged over valid steps and valid control dimensions, excluding padding, and the robotics gripper is predicted by a separate head trained with binary cross-entropy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.