acceptodds
Withdrawn from ICLR 2027

XTrajectory: Learning World Dynamics in Vision-Language Models through Unified Future Trajectory Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Future prediction is essential for agents that reason and act in the physical world, yet existing visual world models commonly represent future dynamics through pixels, latent states, or specialized motion modules. We investigate whether predictive physical dynamics can instead be acquired directly by a general-purpose vision-language model (VLM) through structured future trajectory prediction. We introduce XTrajectory, a unified framework that represents diverse future motion as autoregressively generated coordinate sequences through the VLM's native language interface, without introducing an additional trajectory decoder or dynamics head. XTrajectory unifies the future motion of visual points, physical objects, human hands, and robot end-effectors under a common history-to-future formulation, and learns this capability through geometrically verifiable reinforcement post-training. Experiments on existing trajectory benchmarks and our XTrajectory-Bench show improvements in future trajectory forecasting and strong transfer to unseen manipulation and physical-interaction scenarios. Evaluation on general multimodal benchmarks further reveals meaningful, though non-uniform, transfer beyond trajectory prediction. These results suggest that future trajectories provide a compact, interpretable, and directly verifiable interface for introducing predictive world dynamics into general-purpose VLMs.

Primary area:
reinforcement learning
Keywords:
Machine learning, Computer Vision, Vision Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cmiiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces XTrajectory, a framework that trains a general-purpose vision-language model to predict future 2D trajectories through its native language interface, without an additional trajectory decoder. It unifies the motion of visual points, objects, human hands, and robot end-effectors using a common coordinate-sequence representation. The authors construct a training corpus from seven sources and introduce XTrajectory-Bench, covering physical and egocentric motion prediction. The training approach combines geometrically verifiable rewards with TS-GSPO, which supplements sequence-level optimization with coordinate-segment objectives. Experiments show improvements over the base VLM, competitive transfer to existing manipulation trajectory benchmarks, and non-uniform transfer to other multimodal tasks.

Critical strengths
  1. Useful integration of heterogeneous trajectory prediction tasks. The main strength of this paper is that it brings physical motion, object motion, hand motion, and robot end-effector motion into a unified prediction framework. The authors convert seven datasets into a common format, with sufficient details in Appendix A for reproduction. Since these tasks are usually studied separately, I think a shared dataset and benchmark is a useful contribution. It allows different motion types to be studied and evaluated under the same setting, especially if the authors release the processed data.

  2. Consistent improvements over the base model across trajectory benchmarks. XTrajectory improves over Qwen3.5-9B on all five evaluated trajectory benchmarks, including ShareRobot and VABench, whose inputs omit the explicit motion history used during training. These results support the usefulness of trajectory post-training and demonstrate transfer beyond the training input format. The comparisons with specialized trajectory models and embodied VLMs also provide useful context for its performance.

Critical weaknesses
  1. The contributions of the reward design and TS-GSPO are not clearly isolated.

The reward combines coordinate accuracy, Fréchet distance, motion direction, and a bonus for reasoning longer than 300 characters. However, the paper does not ablate these components. It is unclear whether the added geometric terms outperform a simple coordinate-distance reward, or whether the reasoning length bonus actually helps. The thinking-mode experiments do not answer the latter question.

  1. The evaluation metrics and baseline comparisons need clarification.

Under the RMSE definition in Eq. 5, discrete Fréchet distance should satisfy dF≤T dRMSd_F \leq \sqrt{T}\,d_{\mathrm{RMS}} for trajectories of equal length evaluated on the same points. This bound also holds when averaging over the same samples. Appendices B.1 and B.3 specify five future points for Physics-IQ and xHOT3D. However, the quadratic baseline in Table 1 gives Fréchet/RMSE ratios of 2.317 and 2.353, both exceeding 5≈2.236\sqrt{5}\approx 2.236. Averaging RMSE over spatial dimensions could explain this difference, but that is not the definition given in Eq. 5. The authors should clarify how these metrics are calculated and averaged.

Separately, the SFT baseline trains only on coordinates, without reasoning or semantic labels. The RL method uses a different output format and training setup. This makes it difficult to attribute the reported capability retention to RL itself. Given the near-zero SFT scores on some multimodal benchmarks, it would also help to distinguish formatting failures from incorrect predictions.

Questions for authors
  1. Metric consistency: How exactly are RMSE and Fréchet distance calculated in Table 1? Why do the quadratic baseline's Fréchet/RMSE ratios exceed √5 on the two five-step datasets? If this is an implementation issue, do the corrected results change the rankings?

  2. Reward design: How does the full reward compare with a coordinate-distance-only reward? Does removing the reasoning-length bonus affect performance?

  3. TS-GSPO: What happens when the segment loss uses the global advantage instead of segment-specific advantages? This would help isolate the benefit of local credit assignment.

  4. SFT comparison: How was the SFT baseline tuned? Are its low multimodal scores caused by formatting errors or incorrect predictions? Would the conclusions hold under comparable output formats and supervision?

Reviewer_FRRDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces XTrajectory, which combines heterogeneous trajectory prediction through a shared 2D coordinate interface, which means that XTrajectory does not require an additional prediction head. Additionally, the work proposes TS-GSPO, extending GSPO to allow for finer trajectory credit assignment. Lastly, the work includes a benchmark to evaluate this work's specific setting.

Critical strengths
  • The overall approach seems to work well as Tables 1 and 2 show, and the addition of TS-GSPO also shows promising results in Table 4
  • The unification of trajectory prediction into a VLM as pure text prediction without an additional prediction head is neat
Critical weaknesses
  • Coverage of prior work seems limited: Many aspects that are central to the work's contribution have existing prior art that should be referred to. ManipLVM-R1 [1] and VLA-R1 [2] already train VLMs to generate 2D trajectories as text using rewards like RMSE and Fréchet distance. Although the work does cite MolmoMotion [3] it does not mention MolmoMotion-AR which already generates trajectories without an additional trajectory prediction head, directly from the language head.
  • Missing empirical exploration: The paper misses central aspects regarding the design space of the proposed method (e.g., how to form the predictions) as well as explorations that isolate joint / cross-domain training from domain-specific ones, as core contributions like TS-GSPO do not seem to rely on joint training.
  • Claiming Trajectory-Bench as one of the paper's three main contributions is questionable, as its primary purpose is to provide a reformatted setting of already existing benchmarks (Physics-IQ, Apple-pi, HOT3D), made to match the author's methodological choices. While this serves as a valid evaluation for their own task, the benchmark does not offer clear value to the community beyond this, making the claim as a main contribution unsubstantiated.

[1] Z. Song et al. ManipLVM-R1: Reinforcement Learning for Reasoning in Embodied Manipulation with Large Vision-Language Models [2] A. Ye et al. VLA-R1: Enhancing Reasoning in Vision-Language-Action Models [3] J. Zhang et al. MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction

Questions for authors

Could the authors please lay out how their work relates to prior works like ManipLVM-R1, VLA-R1, and MolmoMotion-AR? Can the authors elaborate on the significance of joint training and differentiate the performance improvements that their contributions offer in more detail? Can the authors provide an exploration of the relevant aspects of the design space that motivate their choices and differentiate them from previous approaches?

Reviewer_98WrOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents XTrajectory, a reinforcement learning framework for unified future trajectory prediction in general-purpose vision-language models (VLMs). The method formulates motion prediction across heterogeneous entities as a unified 2D coordinate generation task, emphasizing direct trajectory generation through the VLM's native autoregressive interface without introducing task-specific prediction heads. The authors integrate seven data sources to construct approximately 110K training samples and propose a geometrically verifiable reinforcement learning framework with TS-GSPO, which introduces step-level credit assignment to optimize individual future trajectory predictions. In addition, the authors construct XTrajectory-Bench, an evaluation benchmark comprising 822 samples adapted from existing datasets. Experimental results demonstrate improved trajectory prediction accuracy over the original Qwen3.5 model and competitive performance compared with existing methods.

Critical strengths

The authors integrated seven data sources, covering physical dynamics, human interaction, and robot operation, and constructed approximately 110K training samples. The construction of XTrajectory-Bench further achieved comparable evaluation across different domains, providing a powerful framework for researching trajectory prediction in general visual language models. Although I don't consider the unification of two-dimensional coordinates to be a particularly innovative aspect, it does establish a unified training and evaluation interface that can cover various types of movements.

Critical weaknesses
  1. The research motivation of this article is somewhat strained. For instance, MLP and diffusion output heads are already quite mature, common, and simple. It's hard to say that their bottlenecks are due to them being additional modules. This article also lacks comparisons regarding the accuracy, generalization, and efficiency of lightweight output heads. Even many previous works have already achieved the direct utilization of VLM to obtain trajectory coordinates.
  2. The proposed method TS-GSPO is not novel. It is somewhat similar to engineering combination and optimization, and similar methods have already appeared in previous studies. Moreover, apart from improving the accuracy of trajectory prediction, the authors have not fully demonstrated the necessity of TS-GSPO for learning transferable world dynamics.
Questions for authors
  1. In the experimental part, the authors used 110k heterogeneous data for training, but they did not conduct any experimental comparisons to clarify how data from different domains could assist in learning "transferable world dynamics".
  2. The Zero-shot results in Table 2 do not seem to demonstrate the superiority of this paper. If the authors mainly use trajectory prediction accuracy to prove that the model has achieved better world dynamics, then could the results of other models also support a similar conclusion? On the contrary, if trajectory prediction accuracy itself is not sufficient to prove better world dynamics, then what?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.