acceptodds
Withdrawn from ICLR 2027

TrajLoom: Dense Future Trajectory Generation from Video

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Predicting future motion is crucial in video understanding and controllable video generation. Dense point trajectories are a compact, expressive motion representation, but modeling their future evolution from observed video remains challenging. We propose a framework that predicts future trajectories and visibility from past trajectories and video context. Our method has three components: (1) Grid-Anchor Offset Encoding, which reduces location-dependent bias by representing each point as an offset from its pixel-center anchor; (2) TrajLoom-VAE, which learns a compact spatiotemporal latent space for dense trajectories with masked reconstruction and a spatiotemporal consistency regularizer; and (3) TrajLoom-Flow, which generates future trajectories in latent space via flow matching, with boundary cues and on-policy K-step fine-tuning for stable sampling. We also introduce TrajLoomBench, a unified benchmark spanning real and synthetic videos with a standardized setup aligned with video-generation benchmarks. Compared with state-of-the-art methods, our approach extends the prediction horizon from 24 to 81 frames while improving motion realism and stability across datasets. The predicted trajectories directly support downstream video generation and editing.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VojMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a dense point trajectory prediction model based on video diffusion framework. Three components are proposed, i.e., grid-anchor offset encoding module, trajectory-based VAE and the trajectory-based flow matching model. Representing each point as offset from pixel-center reduces the location-dependent bias. The trajectory-based VAE learns a compact latent space and the flow matching model generate future trajectory via diffusion. A benchmark is further designed for evaluation. Extensive experiments conducted on this benchmark validate the effectiveness of the proposal in improving motion realism.

Critical strengths
  1. Exploring history point trajectories to improve estimation is reasonable, and the proposed formulation is valuable.
  2. Combining Grid-Anchor Offset Encoding with the spatiotemporal consistency regularizer in TrajLoom-VAE effectively suppresses frame-to-frame jitter and local tearing.
  3. The proposed unified real-world and synthetic benchmark benefits the research community in trajectory prediction.
  4. The paper is well-written, logically structured and easy to follow.
Critical weaknesses
  1. The core contribution, i.e., the grid-anchor offset encoding is essentially a standard zero-centering or cumulative displacement parameterization widely used in point tracking and trajectory forecasting literature. It cannot be packed as a primary algorithm contribution.
  2. The spatial consistency regularization term (Eq. 2) enforces relative motion smoothness among neighboring grid indices. However, when trajectory evolves into the future forecast window (e.g., frames 82–162), two points that were spatial neighbors on the coarse grid at Frame 1 may have diverged drastically in physical space (e.g., one moving with a foreground object to the right while the other stays on the background to the left) or even undergone topological crossing. Enforcing a heavy spatial neighbor penalty could result in over-smoothing motion boundaries.
  3. In real-world video dynamics, point occlusion/visibility and 2D spatial trajectories are tightly coupled (e.g., an object rotating out of view or passing behind an occluder). Nevertheless, TrajLoom-Flow models only coordinate offsets during the generative flow-matching process, relegating visibility prediction to a separate, post-hoc 1D temporal convolution head. The performance should be evaluated and discussed.
  4. The main experimental comparisons against WHN and Tra-MoE suffer from a severe information asymmetry. While WHN and Tra-MoE predict future motion from a single static image and text (a highly ambiguous task with large directional and speed uncertainty), TrajLoom is conditioned on 81 frames of ground-truth history trajectories. The clarification and explanation should be included.
Questions for authors

Please see the weaknesses.

Reviewer_MdYCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TrajLoom, a framework for predicting dense future point trajectories and visibility from observed video and trajectory history. The method has three components: (1) Grid-Anchor Offset Encoding, which represents each trajectory point as an offset from its pixel-center anchor to reduce location-dependent bias; (2) TrajLoom-VAE, a spatiotemporal VAE with masked reconstruction and a consistency regularizer that learns compact trajectory latents; and (3) TrajLoom-Flow, a rectified-flow generator conditioned on history latents and video features, with boundary hints and on-policy K-step fine-tuning. The authors also introduce TrajLoomBench, a unified benchmark spanning real (Kinetics, RoboTAP) and synthetic (Kubric) videos plus held-out MagicData, and demonstrate downstream use with Wan-Move for trajectory-guided video generation. The headline claims are extending the prediction horizon from 24 to 81 frames while improving motion realism and stability over the WHN baseline.

Critical strengths

Well-motivated representation design with empirical support. The Grid-Anchor Offset Encoding is simple, principled, and backed by a clean variance-decomposition analysis (Fig. 3b) plus ablations (Table 6, Fig. 10) showing it improves both convergence and reconstruction. This is the most convincing and reusable contribution of the paper.

Extended horizon and joint generation. Generating 81-frame futures jointly (rather than autoregressively) with stable sampling is a genuine practical advance over the 24-frame WHN setting, and the downstream Wan-Move demonstration shows the trajectories are usable as motion control signals. The benchmark unification (TrajLoomBench) is a useful community contribution.

Critical weaknesses

Confounded primary comparison. The central quantitative claim—large improvements over WHN—compares methods with fundamentally different conditioning (history + video vs. single image). Without a history-conditioned baseline or a controlled ablation isolating the contribution of history conditioning, the headline numbers do not establish that the proposed components (offset encoding, VAE, flow) are responsible for the gains. This is the single most consequential weakness for my assessment.

Metric emphasis mismatch. The main results table foregrounds coherence diagnostics (FlowTV, DivCurlE) that do not measure prediction accuracy and can be gamed by over-smoothing, while the accuracy-oriented metrics (MSE, ES) appear only in the appendix. Combined with the confounded baseline, this makes the main-text evidence less persuasive than the appendix suggests.

Questions for authors

Can you provide a controlled comparison isolating the effect of history conditioning—e.g., a WHN-style model given the same history trajectories/video features, or an ablation of TrajLoom without history conditioning—so that the contribution of the proposed components is separable from the contribution of additional conditioning?

For Table 2, is "WHN (L)-VAE" your re-implementation or the original? Please clarify training data, capacity, and evaluation protocol to confirm the comparison is apples-to-apples.

Why are MSE and Energy Score not reported in the main table alongside FVMD/FlowTV/DivCurlE, given your own caveat that the latter are not accuracy measures? Could the main table be restructured to lead with prediction-error and distributional metrics?

Reviewer_5CJNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies dense future trajectory generation conditioned on observed video and trajectory history. It combines grid-anchor offset encoding, a consistency-regularized trajectory VAE, and latent rectified flow with boundary hints and on-policy fine-tuning. The framework supports joint 81-frame generation and downstream motion-guided video synthesis.

Critical strengths

The motion-continuation task is practically relevant, and the implementation is supported by several useful component ablations. The appendix includes prediction-error and distributional metrics, sampler comparisons, noise sensitivity, a user study, and implementation details. In particular, the full-budget on-policy ablation provides meaningful controlled evidence for the proposed sampling refinement.

Critical weaknesses
  1. Unequal conditioning confounds the main comparisons. TrajLoom uses 81 frames of video, explicit trajectory history, and visibility, whereas WHN and Tra-MoE are evaluated with image-based conditioning. The reported gains therefore do not isolate methodological improvements from the benefits of additional motion observations. Given WHN’s existing multi-frame extension, stronger history-matched baselines and modality ablations are needed.

  2. The distinctive methodological contribution remains unclear. History-conditioned forecasting, displacement representations, and latent trajectory generation have relevant precedents. Although Tables 6 and 7 support the proposed components, the paper does not clearly establish what new insight or capability its combination contributes beyond existing techniques and standard temporal conditioning.

Questions for authors

Q1. Comparisons under matched information. WHN already includes a multi-frame extension in Appendix A.2. Why is its single-image version used as the principal baseline? Could you evaluate a history-conditioned WHN-style flow model with matched training data, history frames, trajectory/visibility inputs, text conditioning, query points, and prediction windows? A constant-velocity baseline estimated solely from observed history, together with ablations separating video and trajectory conditioning, would also help establish the contribution beyond richer observations and simple extrapolation.

Q2. Strictly history-only conditioning and alignment. Are the input AllTracker trajectories computed independently using only frames 1–81, or extracted from the full 162-frame sequence and then truncated? How are test captions obtained, and do they contain information about future motion? Please also clarify how the conditioning image at the history–future boundary, query-point identities, and occluded points are aligned across methods. If history is extracted from the full sequence, could you provide a history-only extraction control to verify that the results do not depend on access to future frames?

Comments on the paper

1 comment

We thank the reviewers for their time, constructive feedback, and recognition of the value of dense motion continuation. We have decided to withdraw this submission. We would also like to clarify several points regarding the WHN comparison, our task formulation, and the evidence supporting the paper’s contributions.

WHN comparison and prediction horizon. Our experiments evaluate WHN (L) in the single-image setting implemented by its official public release. We verified that the released training code selects the first RGB frame and that the inference pipeline accepts one conditioning image per example. Although Appendix A.2 of WHN’s published paper describes a multi-frame extension, the public release we inspected does not implement the corresponding multi-frame conditioning pipeline. Our baseline therefore follows the released configuration. In the relevant comparisons, both methods are evaluated over a common prediction horizon to accommodate WHN (L)’s supported window. TrajLoom additionally demonstrates the capability to generate 81-frame futures jointly.

Why motion continuation uses history. TrajLoom studies the continuation of motion already observed in a scene. A single image generally cannot uniquely determine the ongoing direction or speed: the same visible scene state can correspond to different motions. Video and trajectory history provide evidence about those dynamics and define the motion that the generated future should continue. History conditioning is therefore central to this task formulation. The baseline comparisons evaluate the complete systems under their stated inputs, while the component ablations assess the proposed design choices within our history-conditioned framework. These include the grid-anchor representation, consistency regularization, boundary hints, and on-policy refinement, with the full-budget on-policy ablation providing controlled evidence for the sampling refinement. Together, these components enable long-horizon dense motion continuation and downstream motion-guided video generation.

Spatial consistency and motion boundaries. The spatial term in Eq. (2) matches reconstructed differences between neighboring trajectories to their ground-truth counterparts. When initially neighboring foreground and background points subsequently diverge, that divergence is represented in the target. The grid determines which trajectory pairs are compared; it does not require those points to remain spatial neighbors throughout the forecast. The regularizer therefore encourages faithful reconstruction of relative motion, including differences across motion boundaries.

Complementary evaluation metrics. MSE and Energy Score are reported in the appendix and complement the distributional and coherence measures in the main results. FlowTV and DivCurlE serve as diagnostics of motion structure and should be interpreted alongside the prediction-error and distributional metrics. Together with the component ablations, these evaluations support the paper’s central contribution: generating long-horizon dense trajectories that continue observed scene dynamics and provide useful control for downstream video generation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.