acceptodds
Withdrawn from ICLR 2027

Let Relations Shape the World: Learning Instruction-Conditioned World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Relational instructions can express the same task across different participants, yet demonstrations provide only scene-specific visual outcomes. We introduce RWM, a relation-shaped world model that learns language-grounded object interactions to predict desired futures. Its relational transformer, RELATRON, grounds language in ordered object pairs through a read–write operation, with separate branches determining which interactions contribute and what information they convey. Both branches are trained end-to-end with the future predictor using demonstrated visual changes, without explicit object-pair relation annotations. Shared pairwise operators provide an inductive bias for reuse across participants, while their spatial context augments rather than replaces dense visual features. The predictor generates short-term and terminal visual goals supervised by future observations rather than symbolic goal-state targets. These goals guide action proposals, whose predicted consequences are compared with the short-term goal in a shared visual representation space. Controlled evaluations on our TELETRAAN I benchmark show higher future-goal agreement under simultaneous target–reference color shifts, improved instructed-object grasping under joint target–distractor changes, and higher mean pickup success on colors excluded from task training. Ablations show contributions from both interaction selection and language-conditioned messages. On LIBERO, consequence-based selection improves mean success over single-proposal execution. These results support learning relational conditioning from demonstrations to apply familiar task requirements under changes in participant appearance and scene context.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Models, World Action Models, Instruction Conditioning, Relational Inductive Biases, Latent Future Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fZRmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RWM uses language-conditioned interactions between object pairs to predict short-term and terminal visual goals. These goals condition an action policy, while a separate consequence model scores candidate action chunks. Experiments cover goal retrieval, simulated manipulation, LIBERO, and real-robot execution.

Critical strengths

The ablations show that both learned interaction selection and language-conditioned messages improve goal prediction. The pickup results on unseen colors improve over the reported baselines, and the paper includes real-robot experiments.

Critical weaknesses

The goal interface is built around object-pair relations and short-term/terminal visual futures. Its applicability to contact-rich or long-horizon manipulation is unclear. Force, compliance, and constraints on intermediate action sequences may not be captured by the desired visual endpoints, even when the language instruction is detailed. More complex instructions also require the future generator to resolve temporal and compositional structure. The scope of the approach beyond visually specified object rearrangement needs to be made explicit.

The desired-future generator predicts visual goals without explicitly checking their physical reachability. Consequence-based selection chooses the candidate whose predicted endpoint is closest to the goal, but this relative ranking does not establish that any candidate can achieve it. The paper does not explain how unreachable goals or uniformly poor candidates are detected and handled.

The terminal goal is sampled once and retained throughout the episode, while short-term goals are repeatedly regenerated. The formulation does not explicitly enforce consistency between these goals. For tasks with multiple valid solutions, short-term predictions may correspond to a different solution from the sampled terminal goal. The paper should clarify how this consistency is maintained and how an incorrect or subsequently infeasible terminal goal is revised.

Questions for authors

See weaknesses

1 comment

Thank you for the thoughtful questions regarding the scope and assumptions of RWM.

RWM is designed to address instruction-conditioned visual goal construction. RELATRON grounds language in ordered object interactions, and the shared future predictor generates two complementary predictions: a short-term future corresponding to the next action chunk and a terminal future representing task completion. These predictions condition the action expert together with current-scene features. Our experiments investigate this formulation primarily in visually specified manipulation, particularly under changes in object appearance and scene context.

Regarding physical reachability, the optional consequence predictor estimates the visual outcome of candidate action chunks and compares these predictions against the desired short-term future. Its purpose is to rank candidate actions according to their predicted agreement with the goal. It is therefore a relative outcome-ranking mechanism, rather than an explicit physical feasibility checker. In particular, it does not guarantee that the selected candidate can fully realize the desired goal.

Regarding consistency between short-term and terminal predictions, both are generated by the same Transformer with shared parameters and relational conditioning, while horizon embeddings distinguish their respective prediction targets. During execution, the short-term future is refreshed at each replanning step, whereas the terminal future is retained throughout the episode. This provides a shared prediction framework across the two horizons, although it does not explicitly impose a cross-horizon consistency constraint or revise the terminal goal when it becomes infeasible.

We have clarified the prediction mechanism and the intended scope of the framework in the revised manuscript. Extending this formulation with explicit physical feasibility estimation and stronger temporal consistency constraints would be valuable directions for future work.

Reviewer_qmioOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes RWM, an instruction-conditioned world model for relational manipulation tasks. Its main component, RELATRON, uses language-conditioned read/write operations over ordered object pairs, with separate branches for relation selection and message generation. The model predicts short-term and terminal visual goals, which are used to condition an action policy. A separate consequence predictor evaluates candidate actions against the generated goals. The paper evaluates the approach on the proposed TELETRAAN I benchmark, LIBERO, and robot manipulation tasks.

Critical strengths
  • The problem of reusable relational grounding is relevant. Learning to apply relational instructions across changes in object appearance and scene context is an important problem for robotic manipulation. Separating goal prediction from action execution is also a reasonable direction to explore.
Critical weaknesses

In its current form, the work does not seem sufficiently developed to support its claimed contribution. While the direction is interesting, I have many concerns regarding its presentation and results. The paper needs clearer design motivation, more transparent experimental reporting, and stronger evidence that the added complexity leads to meaningful improvements.

  • Unnecessarily complex method with insufficient motivation. The paper relies heavily on elaborate terminology and equations, but provides little intuition for its key design choices. Why are two independent branches needed? Why is the read assignment reused for writing? Why are messages aggregated at the first endpoint? More importantly, what does this conditioning mechanism offer over simpler alternatives? The method section focuses too much on describing operations rather than explaining why they are necessary. The current ablations do not adequately justify these choices, making it difficult to identify the actual contribution.

  • Experimental evidence is not convincing enough. Despite the emphasis on reusable relational grounding, Bowl pair-transfer execution achieves only 7.29% success in Appendix B.1. The benefit of action-consequence selection is also inconsistent. Section 4.4 reports an improvement over uniform selection, while Appendix C.4 shows identical success rates (83.3%) in a separate matched cohort. Table 2 also reports substantially different single-proposal results (90.6% vs. 79.4%) without clearly explaining the difference. Moreover, RWM underperforms π0.5 and LingBot-VA on LIBERO despite its additional complexity.

Questions for authors
  1. Architecture necessity: Could the authors explain why the specific RELATRON design is necessary? While the component ablations show that relation selection and language-conditioned messages are useful, they do not establish the need for independent branches, reusing read assignments for writing, or aggregating messages at the first endpoint. A comparison with a simpler language-conditioned graph attention or message-passing module under matched training conditions would help clarify the contribution of these design choices.

  2. Bowl pair-transfer execution: Appendix B.1 reports only 7.29% success for Bowl pair-transfer execution, despite the stronger results in future-goal retrieval. What is the main source of failure in this setting? In particular, could the authors distinguish errors in goal generation, action proposals, consequence prediction, and action selection? This would help clarify how much the improved relational grounding translates into successful execution.

  3. Evaluation consistency and action selection: Could the authors clarify why “RWM (single proposal)” and “Single proposal” in Table 2 obtain different average success rates (90.6% vs. 79.4%), and what differs between their evaluation protocols? Also, while the reported LIBERO results show that RWM trails π\pi 0.5 and LingBot-VA, which settings most clearly justify its additional architectural complexity? Finally, Appendix C.4 reports identical success rates for AC and uniform selection in the matched timing cohort. How consistently does consequence-based scoring improve performance when the candidate set and computational budget are matched?

1 comment

Thank you for the detailed questions regarding RELATRON and the experimental evaluation.

Our central design choice in RELATRON is to separate two complementary functions: identifying which object interactions are relevant to an instruction, and determining what instruction-dependent information should be conveyed through those interactions. We evaluated these functions through component ablations. Removing relation selection reduces joint-shift future-goal accuracy from 60.42% to 46.88%, while removing language-conditioned messages reduces it to 22.92%. These results demonstrate that both interaction selection and language-conditioned message construction contribute to relational future prediction.

The read–write operation uses the same attention assignments for retrieving and writing relational information, so that language information is returned to the object pairs from which it was retrieved. First-endpoint aggregation follows the directed structure of ordered object pairs: each pair contributes a message to its designated first participant. Together, these operations provide a structured way to route instruction-grounded information back into the visual scene.

Regarding Bowl pair-transfer execution, the 7.29% result reported in the earlier version measures end-to-end task completion, whereas future-goal retrieval evaluates whether the predicted visual outcome matches the instructed relation. Successful execution additionally depends on action generation and physical realization. The two metrics therefore capture different stages of the task, and improved future-goal prediction does not by itself guarantee successful execution.

For LIBERO, we extended action-policy training and evaluated a later checkpoint at 35,000 action-training steps, achieving 94.75% mean success. This improvement reflects more complete policy optimization. Table 4 further separates the effects of future pretraining and explicit object-pair conditioning, using the same task and initial-state pairs across the variants.

Regarding the comparison with π0.5 and LingBot-VA, our objective is not to establish a higher overall manipulation score than these larger general-purpose policies. Rather, we investigate how relational structure can provide reusable instruction conditioning for visual future prediction and complement an existing pretrained action policy. The participant-shift evaluations on TELETRAAN I, together with the component ablations, provide the most direct evidence for this contribution.

Regarding action selection, the earlier matched timing cohort reported 83.3% for both intended-goal and uniform selection. We do not rely on this comparison to claim a consistent execution advantage from consequence-based scoring. The central contribution of RWM is the relational conditioning mechanism and its role in instruction-conditioned future prediction.

We have updated the experimental presentation and clarified the scope of the contributions accordingly.

Reviewer_8437OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RWM, an instruction-conditioned visual world model for language-guided manipulation. Its core is RELATRON, a relational conditioner in which language reads from and writes to ordered object-pair features. One branch decides which pairs matter and another decides what message each pair carries. No relation labels are used. The resulting spatial context conditions a flow-matching generator that predicts short-term and terminal visual goals in V-JEPA 2 feature space. A π0.5-initialised action expert proposes candidate action chunks, and an action-conditioned predictor picks the chunk whose predicted outcome best matches the short-term goal. On the new TELETRAAN I benchmark, RWM improves future-goal agreement under joint color shifts (60.42 vs 55.21), instructed-object grasping (85.42 vs 46.88), and pickup on held-out colors (57.50 vs 39.17). It also reports results on LIBERO (90.6% average) and two real-robot tasks.

Critical strengths
  1. A principled, well-motivated conditioning design, supported by component ablations. Separating interaction selection from message content is a clean inductive bias. So are the ordered pairs, the equivariance to object re-indexing, and using relational context to augment dense visual features rather than replace them. All of it is learned end-to-end without relation annotations. On Bowl Placement both branches clearly matter (Fig. 7):

    • removing language-guided selection drops in-distribution agreement from 99.96 to 50.08 and joint-shift agreement from 60.42 to 46.88;
    • removing language from the messages drops them to 33.58 and 22.92.

    The selection heatmaps (Fig. 6) also look qualitatively plausible.

  2. Clear gains on attribute generalisation, under a carefully controlled protocol. On instructed-object grasping and held-out-color pickup (Table 1), RWM's margins are large: 85.42 vs 46.88 under joint shift, and 57.50 vs 39.17 OOD on average. The protocol is matched: identical initial states and instructions, paraphrases checked against training captions, fixed assessors, and retrieval-ceiling calibration reported (90.0/93.3%).

Critical weaknesses
  1. The evidence that RELATRON beats simpler conditioners is thin, partly contradicted in the appendix, and plausibly driven by under-tuned baselines.
    • Contradicting diagnostic. In the appendix's instruction-following diagnostic (Table 5), RELATRON is worse than both Object Transformer and Naive Relation on joint retrieval (72.7 vs 84.8 and 77.7) and on paired accuracy (63.0 vs 79.8).
    • Failures even in-distribution. Several baselines fail completely where the task is in-distribution. Object Transformer reaches 54.17% in-distribution grasping against RWM's 100%. Language Cross-Attention scores 0.00% in-distribution pickup. Fine-tuned π0.5 scores 0% on every red-block prompt, and LingBot-VA 0% on all Bowl Placement prompts and all physical tasks (Table 4, App. B.1). Gaps this large on familiar scenes point to a training or adaptation problem in the baselines, not a benefit of relational conditioning, and so they weaken the execution-level comparisons.
    • Narrow scope. TELETRAAN I is a small synthetic benchmark (colored blocks, bowls and cylinders; three relations), and "generalisation" means color swaps only. The real-robot results (100% / 85%) have no working baseline and no reported trial count.

Please report: properly tuned baselines (at least matching RWM in-distribution), and harder relational or compositional tests beyond appearance shifts.

  1. The second contribution, a visual-goal interface from language to control, is not shown to improve control, and its reported numbers are inconsistent.
    • LIBERO. RWM (90.6) is below the π0.5 model it is initialised from (94.3) and far below LingBot-VA (98.5). Even with action-consequence selection it reaches 91.8, still under plain π0.5. The "matched" single-proposal baseline scores 79.4, while "RWM (single proposal)" in the same table scores 90.6. This is unexplained.
    • Bowl Placement. §4.4 reports 91.66% for intended-goal selection vs 83.33% for uniform selection. Yet App. C.4 states that in the matched timing cohort, uniform and AC selection both obtain 83.3%, with AC winning two cases and losing two. App. C.4 also reports a separate 58.3 → 87.5 comparison "on their evaluation cohorts", i.e., not matched.
    • Cost. AC selection adds decision latency (Fig. 9).

Overall, there is no clean, matched evidence that predicted futures and consequence-based selection improve task success over the base policy. The abstract's LIBERO claim therefore rests on a comparison against a weakened single-proposal variant. The paper should reconcile these numbers and report AC against the strongest matched single-proposal policy.

Questions for authors

See weaknesses.

1 comment

Thank you for the detailed review.

We designed TELETRAAN I to study whether language-conditioned object interactions help preserve task intent when the appearance and context of participating objects change. Under joint target–distractor shifts, RWM achieves 85.42% instructed-object grasping success, compared with 46.88% for Object Transformer. On colors excluded from task training, it achieves 57.50% mean pickup success, compared with 39.17% for the strongest evaluated baseline. For future-goal prediction, RWM also achieves 60.42% under joint target–reference color shifts, compared with 55.21% for Object Transformer. These experiments provide a controlled evaluation of relational grounding under participant changes.

Regarding the baseline comparisons, our goal is to examine alternative conditioning mechanisms under the designed evaluation settings, rather than establish a universal ranking of different model families. We have clarified this distinction and specified the scope of the generalization claims.

For LIBERO, we extended action-policy training and evaluated a later checkpoint at 35,000 action-training steps. This increased mean success from the previously reported 90.6% to 94.75%, reflecting more complete policy optimization. The revised manuscript clarifies the training schedule and checkpoint selection.

We would also like to clarify the positioning of RWM relative to π0.5 and LingBot-VA. These models target broad manipulation capabilities, whereas our work investigates a complementary question: how relational structure can improve instruction-conditioned visual future prediction and be integrated into an existing pretrained action policy. LIBERO evaluates overall policy performance, while TELETRAAN I is designed to isolate relational grounding under controlled participant shifts. We view these as complementary evaluation settings rather than competing objectives.

Finally, the real-robot Approach and Relative Pick-and-place experiments demonstrate that the predicted instruction-conditioned futures can be connected to physical execution, complementing the controlled simulation evaluations.

We have updated the Abstract, experimental discussion, and conclusion to better reflect these contributions and their evaluation scope.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.