- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RWM uses language-conditioned interactions between object pairs to predict short-term and terminal visual goals. These goals condition an action policy, while a separate consequence model scores candidate action chunks. Experiments cover goal retrieval, simulated manipulation, LIBERO, and real-robot execution.
Critical strengths
The ablations show that both learned interaction selection and language-conditioned messages improve goal prediction. The pickup results on unseen colors improve over the reported baselines, and the paper includes real-robot experiments.
Critical weaknesses
The goal interface is built around object-pair relations and short-term/terminal visual futures. Its applicability to contact-rich or long-horizon manipulation is unclear. Force, compliance, and constraints on intermediate action sequences may not be captured by the desired visual endpoints, even when the language instruction is detailed. More complex instructions also require the future generator to resolve temporal and compositional structure. The scope of the approach beyond visually specified object rearrangement needs to be made explicit.
The desired-future generator predicts visual goals without explicitly checking their physical reachability. Consequence-based selection chooses the candidate whose predicted endpoint is closest to the goal, but this relative ranking does not establish that any candidate can achieve it. The paper does not explain how unreachable goals or uniformly poor candidates are detected and handled.
The terminal goal is sampled once and retained throughout the episode, while short-term goals are repeatedly regenerated. The formulation does not explicitly enforce consistency between these goals. For tasks with multiple valid solutions, short-term predictions may correspond to a different solution from the sampled terminal goal. The paper should clarify how this consistency is maintained and how an incorrect or subsequently infeasible terminal goal is revised.
Questions for authors
See weaknesses
Thank you for the thoughtful questions regarding the scope and assumptions of RWM.
RWM is designed to address instruction-conditioned visual goal construction. RELATRON grounds language in ordered object interactions, and the shared future predictor generates two complementary predictions: a short-term future corresponding to the next action chunk and a terminal future representing task completion. These predictions condition the action expert together with current-scene features. Our experiments investigate this formulation primarily in visually specified manipulation, particularly under changes in object appearance and scene context.
Regarding physical reachability, the optional consequence predictor estimates the visual outcome of candidate action chunks and compares these predictions against the desired short-term future. Its purpose is to rank candidate actions according to their predicted agreement with the goal. It is therefore a relative outcome-ranking mechanism, rather than an explicit physical feasibility checker. In particular, it does not guarantee that the selected candidate can fully realize the desired goal.
Regarding consistency between short-term and terminal predictions, both are generated by the same Transformer with shared parameters and relational conditioning, while horizon embeddings distinguish their respective prediction targets. During execution, the short-term future is refreshed at each replanning step, whereas the terminal future is retained throughout the episode. This provides a shared prediction framework across the two horizons, although it does not explicitly impose a cross-horizon consistency constraint or revise the terminal goal when it becomes infeasible.
We have clarified the prediction mechanism and the intended scope of the framework in the revised manuscript. Extending this formulation with explicit physical feasibility estimation and stronger temporal consistency constraints would be valuable directions for future work.