acceptodds
Under review as a conference paper at ICLR 2027

JtT: Rethinking Point Trajectory Diffusion for Diverse and Rigid Motion from an Image

14 views

Abstract

We study forecasting rigid-body motion from a single image: generating diverse and physically plausible future trajectories for arbitrary points in a scene, without building rigidity into the architecture or relying on geometric inputs such as depth. Like video generators, recent trajectory generators fail at this task, even in simple synthetic Kubric scenes, despite scaling up model size and training on real data. We show how to solve this with the right architecture and parameterization. Our model, JtT, is a simple diffusion transformer that operates directly in coordinate space and decodes arbitrary and dense query patterns, without the trajectory VAEs or fixed grids assumed in prior work. Through a systematic study of the design space of point trajectory diffusion, we find that diffusing per-frame velocities is the only parameterization that captures rigidity, that it must be paired with the right timestep schedule, and that adding full-resolution RGB at the query points resolves the boundary failures of prior methods. However, evaluating such 2D models rigorously is itself non-trivial. To our knowledge, we are the first to observe that physical plausibility and diversity do not improve together during training: larger models, as used in prior work, memorize the single ground-truth future per training image, becoming more rigid while covering fewer possible futures. Yet prior work has paid little attention to this, so we show how model rankings depend on model size, metric, and stopping time, and adopt a stricter, checkpoint-averaged evaluation protocol. Under this protocol, JtT improves over prior methods on both distributional and rigidity metrics and approaches ground-truth rigidity, with only about 50M parameters.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
point trajectory diffusion, flow matching

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VfCcOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents JtT, a diffusion transformer for predicting diverse and physically plausible point trajectories from a single image. The method combines velocity-based trajectory prediction, full-resolution RGB conditioning, and a support/non-support token architecture for flexible and dense queries. The paper also investigates the trade-off between rigidity and diversity during training.

Critical strengths

I find several strengths of this work:

  1. The paper provides a systematic investigation of important design choices in trajectory diffusion, with several interesting empirical findings.

  2. The experiments are solid, with comprehensive ablations and convincing improvements over existing methods.

  3. The analysis of the rigidity-diversity trade-off is insightful and highlights important limitations of existing evaluation practices.

Critical weaknesses

The quantitative evaluation is primarily conducted on synthetic Kubric datasets. Although qualitative results are provided on real-world datasets, the generalization to complex in-the-wild scenes, particularly those involving articulated or non-rigid motion, remains unclear.

Questions for authors
  1. How well does JtT generalize to complex in-the-wild scenarios? Is the method primarily effective for rigid objects, or can it also model articulated and deformable motion?
  2. Could JtT be extended to downstream applications such as drag-to-video generation, where sparse user-provided trajectories are used to infer dense, physically plausible motion?
6 comments

The quantitative evaluation is primarily conducted on synthetic Kubric datasets. Although qualitative results are provided on real-world datasets, the generalization to complex in-the-wild scenes, particularly those involving articulated or non-rigid motion, remains unclear.

How well does JtT generalize to complex in-the-wild scenarios? Is the method primarily effective for rigid objects, or can it also model articulated and deformable motion?

Could JtT be extended to downstream applications such as drag-to-video generation, where sparse user-provided trajectories are used to infer dense, physically plausible motion?

Thank you for the positive assessment of our work! We are glad that you find our systematic investigation of the design choices of trajectory diffusion valuable, and our experiments solid, with comprehensive ablations and convincing improvements over existing methods. We are also glad that you find our analysis of the rigidity-diversity trade-off insightful, in particular for evaluation practices.

Non-rigid motion is outside the scope of our work. Our primary contribution is a method that samples diverse motion and captures a physical invariant, rigidity, purely from data, without building it into the architecture. We therefore focus on rigid motion, where JtT sets the state of the art in distributional fidelity and rigidity, on synthetic data (Sec. 4.1) and on real videos (Sec. 4.2, new).

We agree that the submission evaluated JtT quantitatively only on synthetic data, and showed real and non-rigid motion only qualitatively. We therefore extended our evaluation to such data. We address all of your concerns, in three parts:

  1. Real videos with rigid and non-rigid bodies: JtT sets the state of the art on Physics 101 (Sec. 4.2, new).
  2. Articulated motion in the wild, with and without drag conditioning: JtT sets the state of the art on MammalMotion (App. I.1, new).
  3. Beyond mammals: JtT applies zero-shot to robots and to animals that are not mammals (App. I.1.2, new).

Please take a look at the new appendix on downstream applications (App. I, new), which discusses these results at length. It compares samples of JtT and MotionForecasting without and with the motion vector (Figs. 36 and 37 in App. I.1.1). It also shows forecasts for robots, birds, a tortoise, and a crab (Figs. 38 and 39 in App. I.1.2).

We have already updated the paper, and all changes are in blue. Due to character limitations, our response will consist of several separate comments.

AuthorsOpenReview ↗

Real videos with rigid and non-rigid bodies

We added a quantitative evaluation on real videos (Sec. 4.2, new). We compare JtT with prior work (WHN and GPTF) on real videos from Physics 101. They contain simple physical experiments in five scenarios. We evaluate on the 1,456 held-out validation videos. Following prior work, we report the MSE of a single sample in each scenario. This evaluation assesses neither distributional fidelity nor rigidity. We therefore extend it with Best-of-16, FVMD, LRTL, the 3D rigid fit, and the object motion. To measure rigidity, we pseudo-label instance masks and depth of the rigid objects (App. D.2, new).

Results. JtT sets the state of the art on this benchmark. It outperforms prior work in pointwise accuracy, distributional fidelity, and physical plausibility (Table 4, new). It achieves the lowest mean MSE, 17% below the best prior method (GPTF), and the lowest MSE in four of the five scenarios. It learns rigid motion particularly well. Its mean MSE is 34% below that of any prior method in fall, multi, and ramp, and its motion on rigid objects is 2.3× closer to rigid motion than that of WHN. It also lowers FVMD by 22% compared with WHN.

Non-rigid bodies. Physics 101 is not limited to rigid bodies. Its videos also show:

  • the hand of the experimenter, which releases the object;
  • water, into which the object drops (liquid);
  • a coil spring, on which the object hangs, often inside a plastic bag (spring);
  • soft objects made of dough or rubber (fall, multi, and ramp).

In the two scenarios built around non-rigid bodies, JtT has the lowest MSE of all methods in liquid, and in spring it improves on WHN and WAN, while GPTF is better (Table 4a). Qualitatively, JtT captures the interaction between rigid and non-rigid bodies, such as a hand releasing an object or a block entering water. It also reflects different kinds of forces acting on rigid bodies, such as gravity and buoyancy on a wooden block dropped into water (App. G).

Table 4 (new). Results on real videos (Physics 101). (a) MSE of a single sample, with prior work as reported in WHN and GPTF. (b) Our extended evaluation. LRTL, 3D rigid fit (px), and object motion (px per frame) are computed on the rigid objects, and the first row scores the ground-truth tracks.

(a) MSE (px²) ↓FallLiquidMultiRampSpringMean
WAN (1.3B)16.054.4821.8837.5370.4830.08
WHN19.786.0015.6536.3565.3128.62
GPTF27.245.7817.2531.0125.7621.41
JtT-B (Dense)14.04 (13.94)3.71 (3.61)12.53 (12.28)20.62 (20.50)38.18 (37.94)17.82 (17.78)
(b) ExtendedBest-of-16 ↓FVMD ↓LRTL ↓3D rigid fit ↓Object motion
Ground truth videos——4.200.618.27
WHN23.948,26810.283.197.38
JtT-B (Dense)15.22 (15.17)6,484 (6,467)3.57 (3.52)1.38 (1.35)7.53

AuthorsOpenReview ↗

Articulated motion in the wild, with and without drag conditioning

To address both of your questions, we move beyond Kubric and Physics 101. We show that JtT sets the state of the art on forecasting animal motion in the wild, both without and with drag conditioning. Since downstream applications are not the primary scope of our work, we added these results to a new appendix on downstream applications (App. I, new), with a pointer in Sec. 4.2. We copy its key parts below.

Task. We consider the task of forecasting animal motion (Thakkar et al., 2026). Given the image and the first four steps of the point trajectories of up to 500 query points on an animal, the task is to forecast the remaining 28 steps of each trajectory, and whether each query point is visible at each step. The input frame is cropped to 256×256 around the animal. The forecast can also be prompted with a motion vector, which gives the direction and distance the animal moves (arrows in Figs. 37 to 39). It is the mean displacement of the points from their first to their last visible step. At inference, any vector can be given, for example the opposite of the observed motion or twice its distance (Table 26). The motion vector thus acts as a single drag for the whole animal.

Data. We use the official MammalMotion data and annotations released by Thakkar et al. (2026). The training split has 1.6M windows, each having 32 timesteps at 15 fps.

Adapting JtT. We use JtT-B (Dense) with the architecture of Sec. 3.3 and per-frame velocities as the diffusion target (Sec. 3.1). Four changes adapt it to the task:

  1. Observed steps. The observed steps of each trajectory stay clean in its query token, and noise is added only to the 28 future steps.
  2. Motion vector. A linear layer embeds the motion vector, and we add its output to the timestep embedding, which modulates every block. We drop the motion vector for 30% of the training examples, so that a single model forecasts with and without it.
  3. Visibility. We generate the visibility of each point as a third channel, next to the two coordinates of its velocity.
  4. Image features. We use the same frozen DINOv3 ViT-L/16 encoder as MotionForecasting, but with features from layers 5, 11, 17, and 23. We also use a 7×7 RGB patch at the query points (App. C.1.2).

Evaluation. We use the protocol and metrics of MotionForecasting.

Results. JtT sets a new state of the art on the MammalMotion benchmark. It outperforms MotionForecasting in every metric, both without the motion vector and with it as drag conditioning (Tables 24 and 25). The largest gains are in distributional fidelity. Notably, without the motion vector (the setting closest to that of Sec. 3), JtT lowers FD(V) and FD(A) by 75% and FVMD by 63% on all species. Given the motion vector, JtT also outperforms MotionForecasting, slightly on all species and by 26% in ADE and FDE on Panthera. Qualitatively, without the motion vector, the samples of JtT are more diverse than those of MotionForecasting (Fig. 36). We also confirm numerically that JtT successfully incorporates drag conditioning (details in Table 26 of App. I.1, new).

Table 24 (new). Animal motion without the motion vector (MammalMotion). Distributional fidelity, with the metrics of MotionForecasting (lower is better). JtT: Average (Best) of the last five checkpoints.

SetMotionModelFD(V) ↓FD(A) ↓FVMD ↓
All DataHighMotionForecasting2.3602.38435.23
JtT-B (Dense)0.961 (0.942)0.970 (0.960)20.18 (20.03)
CombinedMotionForecasting0.5850.61912.32
JtT-B (Dense)0.148 (0.146)0.155 (0.153)4.52 (4.48)
PantheraHighMotionForecasting10.7597.08869.06
JtT-B (Dense)5.861 (5.529)5.021 (4.937)46.47 (44.90)
CombinedMotionForecasting1.9941.42620.98
JtT-B (Dense)0.947 (0.891)0.807 (0.795)11.62 (11.12)

Table 25 (new). Animal motion given the motion vector (MammalMotion). Best of five samples, with the metrics of MotionForecasting. JtT: Average (Best) of the last five checkpoints. ADE and FDE are in crop widths, and PWT is in %.

SetMotionModelADE ↓FDE ↓VMD ↓PWT ↑
All DataHighMotionForecasting0.04610.06245.4037.47
JtT-B (Dense)0.0441 (0.0440)0.0595 (0.0593)5.39 (5.38)38.07 (38.10)
CombinedMotionForecasting0.02160.02933.9963.90
JtT-B (Dense)0.0211 (0.0210)0.0285 (0.0284)3.97 (3.97)64.13 (64.16)
PantheraHighMotionForecasting0.07240.11375.2334.99
JtT-B (Dense)0.0473 (0.0472)0.0742 (0.0742)5.21 (5.21)37.25 (37.35)
CombinedMotionForecasting0.02910.04563.8664.15
JtT-B (Dense)0.0214 (0.0213)0.0337 (0.0337)3.83 (3.83)64.98 (65.02)
Reviewer_P1tLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper is about predicting how rigid objects will move, given just one image. The model outputs 2D trajectories for whatever query points you give it, and the goal is to get futures that are both diverse and physically plausible. The model, JTT, is a pretty small diffusion transformer. Based on my understanding, it works right in coordinate space, with no VAE. Each query point grabs the RGB value at its own pixel and looks at multi-layer DINOv2 features of the image. The queries get randomly split into a small support set and a big non-support set, so the same model can handle a few points or a dense grid.

A bunch of design choices is tested. The authors find that diffusing per-frame velocities works best for rigidity, that logit-normal timestep sampling matters, and that RGB at the query points fixes the errors near object edges. They also notice that bigger models overfit on Kubric-A: over training they get more rigid but less diverse, so which model looks best depends on when you stop. Because of this, they average results over five checkpoints. On the WHN benchmark, JTT beats earlier trajectory generators on FVMD and LRTL, and its LRTL is close to the ground truth. Real videos are only shown as examples.

Critical strengths

I like how simple the model is. It’s small and it doesn’t need a VAE, extra losses, or rollout fine-tuning. I also appreciate that the authors are upfront about training dynamics. Showing how the losses and metrics move over training for three model sizes is really useful for people working on this benchmark. And the ablations are easy to follow, with the effects of velocity parameterization and query-point RGB showing up consistently.

Critical weaknesses
  1. My biggest worry is novelty. The architecture is mostly put together from known works (Perceiver IO, CoTracker3). Predicting step-by-step displacements is pretty standard in trajectory forecasting (based on my understanding), and MotionForecasting already diffuses velocities. So to me this reads more like an empirical study on synthetic benchmark. What’s missing is the “why”: I can not find whether the paper really explains why velocities give rigid motion. The intro is quite long and says the same claims a few times, and I’d much rather see a short explanation of this main point there.

  2. Table 3 does show substantially better rigidity for velocities than offsets under logit-normal timestep sampling. However, the reason for this gap remains unclear to me. These representations are related by temporal accumulation, although applying diffusion in each space changes the effective noise structure and loss weighting. Could the authors clarify which of these factors drives the improvement?

  3. The lack of quantitative evaluation on real data limits the conclusions. The quantitative results appear to come entirely from synthetic Kubric scenes, while real-data results are qualitative. I wonder whether the query-point RGB gains partly depend on the relatively clear appearance boundaries in these scenes, and whether they would hold up with similar foreground and background colors or more complex textures. Quantitative evaluation on real videos, including an RGB ablation, would help address this concern.

Questions for authors

Please refer to Weaknesses. If these get cleared up, I’m happy to adjust my score.

11 comments

Thank you for the thorough and thoughtful assessment of our work. We are glad that you appreciate the simplicity of JtT (no VAE or auxiliary losses), our transparent analysis of training dynamics, and our clear and consistent ablations.

We acknowledge your concerns about the robustness of the RGB gains and about explaining why diffusing velocities yields rigid motion. Due to character limitations, we will address each concern in a separate message. In this response, we focus on your novelty concern and the RGB ablation. The velocity question and some real-data evaluations require additional experiments that are still in progress, and we will follow up with them shortly.

AuthorsOpenReview ↗

Clarification of novelty of our architecture

My biggest worry is novelty. The architecture is mostly put together from known works (Perceiver IO, CoTracker3).

It is true that our architecture builds on Perceiver IO and CoTracker3, and we acknowledge this in the paper (Sec. 2 and 3.3). However, we disagree that it is mostly put together from known works. JtT differs from both in three ways.

First, neither of these works is generative. Perceiver IO is a general-purpose architecture for structured inputs and outputs. In the paper, its application to motion is deterministic regression of dense optical flow between two frames. CoTracker3 is a correlation-based point tracker: it finds correspondences by searching correlation volumes computed from the frames it observes. In contrast, JtT is given only a single image, so there are no correspondences to search for. It must instead generate sparse trajectories for an arbitrary set of query points over many future frames, sampling many possible futures.

Second, we tokenize motion differently. CoTracker3 uses one token per point per frame with space-time factorized attention. In JtT, each token holds the whole velocity trajectory of a point, together with its position and RGB, so there is no space-time factorization.

Third, our bottleneck (the support set) is dynamic. Both prior works pass information through a fixed set of learned tokens: Perceiver IO initializes its latent array from a learnable matrix that does not depend on the input, and CoTracker3 uses a fixed number of learned virtual tracks (proxy tokens). In both cases, the number of these tokens is a hyperparameter set at training time, and the tokens are not outputs of the model. In contrast, our support set is a random subset of the actual query points (of size up to S_max), re-sampled at every training step. Any point can be promoted to or demoted from the support set, and support tokens are denoised and output like any other query. Non-support tokens read only from the support set and never attend to the image or to each other, so for a fixed support size, the cost of decoding grows only linearly with the number of queries. As a result, at inference, users can choose both the query pattern (Fig. 1) and the support size without retraining.

To the best of our knowledge, it has not been shown before that a point trajectory generator trained this way generalizes to many inference patterns, including dense ones, and outperforms grid-based methods such as WHN (Table 2) without ever being trained on a grid.

To highlight novelty of this architectural contribution, we will quantify the trade-off between compute and quality, by adding a study of how performance depends on the number of support points, which we will report during the discussion period.

AuthorsOpenReview ↗

Ablation of RGB on real data (Physics 101)

I wonder whether the query-point RGB gains partly depend on the relatively clear appearance boundaries in these scenes, and whether they would hold up with similar foreground and background colors or more complex textures. Quantitative evaluation on real videos, including an RGB ablation, would help address this concern.

Thank you for this suggestion. We agree that the gains from RGB should be verified on more complex textures and on real data. We therefore ran the RGB ablation on real videos from Physics 101 and on Kubric-D, which has more complex textures and scenes than Kubric-A. Both are in a new appendix section on ablations (App. C.1, "RGB at the query point"), with a pointer in Sec. 4. All changes in the paper are in blue.

Here we show that RGB at the query point helps on real videos and on more complex textures and scenes.

Setup. In both experiments, we train variants of our model that differ only in the RGB input at the query points. As in the paper, we use the checkpoint-averaged protocol. We evaluate an EMA checkpoint every 10K steps and report the average over the last five, with the best of them in parentheses. App. C.1.4 gives the full setup.

Quantifying boundary errors. On Kubric-A, we observed qualitatively that RGB at the query point fixes the boundary failures of prior methods (Fig. 4). Without RGB, the model sees the image only through DINOv2 features, one per image patch. When two objects share a patch, the model struggles to tell which of them a query point lies on. As a result, boundary points move with the wrong object. Background points are dragged along with moving objects, and points on one object move with a touching object (Figs. 8 and 9, new).

To quantify these errors, we introduce the object-assignment error (OAE). It is the fraction of boundary points (query points within 16 px of another object) whose generated trajectory moves with the neighboring object instead of their own. To decide this, we fit the motion of each object to the generated trajectories of its points. We then check which of the two motions the boundary point follows. App. C.1.5 gives the definition and further validation of this metric.

1. Ablation of RGB on real data (Physics 101, App. C.1.2)

We train three variants of JtT-B (Dense) on Physics 101: no RGB, a 3×3 patch and a 7×7 patch at the query points. We evaluate with the protocol of WHN, discussed in the paper. We compute the MSE (px²) of one sample to the CoTracker3 tracks over the first 25 frames, on all 1,456 validation videos. Like WHN, we sample with 10 Euler steps. Our evaluation code reproduces the reported MSE of the released WHN checkpoint (28.59 vs. 28.62). We train each variant for 200K steps and report the average over 160K–200K. RGB lowers the MSE in every scenario (Table 14, new):

RGB at queryFallLiquidMultiRampSpringMean ↓Best-of-16 ↓
None17.25 (16.88)4.74 (4.61)15.29 (14.82)24.63 (24.28)47.11 (46.63)21.80 (21.44)18.29 (18.08)
3×314.04 (13.94)3.71 (3.61)12.53 (12.28)20.62 (20.50)38.18 (37.94)17.82 (17.78)15.22 (15.17)
7×713.72 (13.55)3.42 (3.28)12.75 (12.61)20.80 (20.74)38.77 (38.39)17.89 (17.86)15.44 (15.33)

Removing RGB increases the error by about 22%. The error is higher in each of the five scenarios at every checkpoint from 30K steps on. The 3×3 and 7×7 patches perform on par.

Fig. 9 (new) shows why. Without RGB, points at the edges of moving objects and hands are dragged along or left behind. Physics 101 has no instance masks, so we segment the query points by their CoTracker3 tracks to compute the OAE. We compute it on the first sample of every validation video (Table 15, new):

OAE (%) ↓FallLiquidMultiRampSpringAll
None4.98 (4.87)6.63 (6.10)3.80 (3.46)3.14 (3.02)3.81 (3.72)3.65 (3.52)
3×32.48 (2.46)4.00 (3.76)2.87 (2.76)2.05 (2.00)3.11 (3.06)2.46 (2.40)
7×71.97 (1.88)3.18 (2.94)2.37 (2.33)2.11 (2.07)3.22 (3.17)2.38 (2.34)

Without RGB, the OAE is about 50% higher than with either patch.

Reviewer_gmWHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the prediction of diverse future motion trajectories for arbitrary query points, given a single input image. The study focuses on ensuring rigid-body consistency in generated motion, maintaining flexibility in query point selection, and achieving dense prediction. The authors propose JtT, a diffusion-based Transformer that generates trajectories directly within a 2D coordinate space. This approach employs frame-by-frame velocity parameterization, an improved timestep sampling strategy, and high-resolution RGB conditioning; it also supports an arbitrary number of query points through the stochastic partitioning of "support" and "non-support" tokens.

Furthermore, the paper provides a systematic analysis of the relationship between model scale, training stages, and generation quality. It reveals that distribution matching and rigid-body consistency may exhibit opposing trends during the training process; consequently, the authors employ an evaluation strategy based on averaging multiple model checkpoints. On the Kubric-A dataset, the JtT-S Dense model—with approximately 55 million parameters—achieves an FVMD(D) of 11,807 and an LRTL of 6.84, outperforming key baselines. The authors also provide qualitative validation using real-world video data.

Critical strengths
  1. The method is concise and achieves competitive results. Without requiring explicit rigid-body constraints or complex trajectory VAEs, JtT supports arbitrary queries and dense trajectory generation. As shown in Table 2, JtT-S Dense achieves an FVMD(D) of 11,807 and an LRTL of 6.84—significantly outperforming WHN-L (13,745 and 14.10, respectively)—which demonstrates the practical effectiveness of its design.

  2. The analysis of model design and training dynamics offers significant research value. Beyond comparing final performance, the paper investigates the impacts of parameterization, time-step sampling, and model scale. Figure 3 reveals a phenomenon where FVMD deteriorates during the later stages of training while LRTL continues to improve; this provides valuable insights for model selection and evaluation in generative trajectory prediction.

Critical weaknesses
  1. The core explanations regarding physical plausibility and diversity lack sufficient validation. The paper primarily uses LRTL to gauge rigid-body consistency and infers a decline in diversity based on changes in FVMD. However, LRTL is not an adequate measure of true physical plausibility, and FVMD is not directly equivalent to mode coverage. While training curves and qualitative analyses offer some supporting evidence, there is a lack of more direct, quantitative validation for the claim that the model increasingly memorizes a single future, leading to "mode collapse."

  2. Quantitative validation in real-world scenarios remains limited. The paper’s quantitative experiments are conducted primarily on the Kubric-A dataset. Although the authors provide extensive qualitative results on Physics 101 and Cityscapes, these do not directly verify distribution matching or physical plausibility in real-world settings. Consequently, it remains difficult to assess the method’s performance in more complex, real-world environments, which limits the generalizability of the conclusions.

Questions for authors

Q1. Regarding the decline in diversity during training: Figure 3 shows that, in the later stages of training, large-scale models exhibit a deterioration in FVMD while LRTL continues to improve. The authors interpret this as the model gradually memorizing a single future. Given that the Kubric-A dataset contains multiple plausible futures for the same initial scene, could the authors compare how well the generated samples cover these various futures across different training stages? This would help determine whether the degradation in FVMD is indeed a result of the model’s predictive modes collapsing toward a single outcome.

Q2. Regarding the validity of the rigid-body consistency metric: The paper notes that LRTL may yield low error values ​​for stationary trajectories. Does the improvement in LRTL—achieved via JtT—also manifest in objects that exhibit significant motion? Could the authors analyze the relationship between LRTL and prediction quality based on the magnitude of object motion? This would confirm that the observed improvement in the metric reflects more accurate rigid-body motion, rather than a tendency for the generated trajectories to become stationary or show reduced movement.

6 comments

Thank you for the thorough and thoughtful assessment of our work. We are glad that you recognize the simplicity and effectiveness of our method (JtT). We also appreciate that you see research value in our analysis of parameterization, timestep sampling, model scale, and training dynamics. In particular, we are glad you found the opposing trends of FVMD and LRTL during training (Fig. 3) useful for model selection and evaluation in generative trajectory prediction.

Due to character limitations, we address each of your concerns in a separate comment:

  1. Q1: Collapse to a single future:
  2. Direct assessment of rigidity:
  3. Quantitative validation on real data: a new section with results on Physics 101
  4. Q2: Relationship between prediction quality and object motion: an analysis of LRTL, rigidity, and accuracy by object displacement (App. D.3, new).

We also incorporated all these changes in our paper (marked in blue).

AuthorsOpenReview ↗

Collapse to a single future

While training curves and qualitative analyses offer some supporting evidence, there is a lack of more direct, quantitative validation for the claim that the model increasingly memorizes a single future, leading to "mode collapse." Regarding the decline in diversity during training: Figure 3 shows that, in the later stages of training, large-scale models exhibit a deterioration in FVMD while LRTL continues to improve. The authors interpret this as the model gradually memorizing a single future. Given that the Kubric-A dataset contains multiple plausible futures for the same initial scene, could the authors compare how well the generated samples cover these various futures across different training stages? This would help determine whether the degradation in FVMD is indeed a result of the model’s predictive modes collapsing toward a single outcome.

Thank you for this suggestion! We agree that mode collapse can be quantified more directly. Kubric-A provides multiple ground-truth futures for each test image, so we can measure directly how well the samples cover them. We therefore strengthened our analysis with three new metrics, computed at every checkpoint of every model.

New metrics (Metrics in Sec. 4; App. D.1, new). For each of the 16 test images of Kubric-A, we draw 64 samples and compare them with its 64 ground-truth futures.

  • Precision is the fraction of samples that land near some ground-truth future.
  • Recall is the fraction of ground-truth futures that land near some sample.
  • Sample variance quantifies diversity. We compute it for each initial condition and report it relative to that of the ground-truth futures.

Precision and recall follow [Kynkäänniemi et al. (2019)]. The ground-truth futures are scored in the same way, one half against the other, which gives the values of a perfect model. Table 17 also reports the medoid distance, the mean distance of the futures of an image to their medoid, relative to that of the ground truth.

Results (Fig. 3; Table 17; App. D.1, new). The new metrics confirm that the larger models memorize a single future. The sample variance shows this directly. For B, it more than halves from 60K to 130K steps. Its precision rises while its recall falls, so the samples stay close to ground-truth futures but cover fewer of them. The recall of B against the ground-truth futures falls from 78% at 40K to 11% at 130K steps. S collapses later and more mildly. XS does not collapse. Accordingly, FVMD increases, while LRTL keeps improving. In the appendix, we also provide qualitative evidence of the collapse (App. D.1, new).

We acknowledge that a large table like the one below is not easy to read. The paper shows recall, precision, and sample variance as plots over training, where the trends are much easier to see (bottom row of Fig. 3; Fig. 12, new).

Table 17 (new). Additional metrics at every checkpoint of Tables 9 and 10 on Kubric-A (mean over the 16 test images; ↑: higher is better). For the sample variance and the medoid distance, the ground-truth value is ideal. The first row scores the ground-truth futures in the same way. *: lowest validation loss.

ModelStepRecall (%) ↑Precision (%) ↑Sample variance (px²)Sample variance (% of GT)Medoid distance / ground truth
Ground truth–89.388.34,4291001.00
XS40K1.280.7619140.37
50K18.477.71,496340.59
60K57.056.12,887650.84
70K74.245.94,005900.99
80K75.647.34,317971.03
90K77.147.34,6141041.06
100K79.353.74,4721011.04
110K80.164.64,297971.03
120K80.368.84,150941.00
130K*78.575.23,871870.97
S40K90.218.47,1081601.29
50K92.022.17,4701691.33
60K90.640.65,8921331.19
70K79.169.54,239961.01
80K77.572.13,958890.98
90K71.977.53,610820.94
100K*70.181.23,494790.93
110K68.081.43,564800.93
120K66.883.83,467780.92
130K62.583.03,266740.88
B40K78.357.84,186951.00
50K60.579.32,956670.84
60K*59.280.32,872650.82
70K52.185.02,690610.79
80K48.886.92,559580.77
90K42.890.02,429550.74
100K37.391.22,295520.71
110K27.095.31,948440.64
120K18.095.71,689380.57
130K10.797.71,332300.48

We added this analysis to the main body of the paper. All changes are in blue.

Kynkäänniemi et al. (2019) : https://arxiv.org/abs/1904.06991

AuthorsOpenReview ↗

Direct assessment of rigidity

The core explanations regarding physical plausibility and diversity lack sufficient validation. The paper primarily uses LRTL to gauge rigid-body consistency and infers a decline in diversity based on changes in FVMD. However, LRTL is not an adequate measure of true physical plausibility, ...

We agree that rigidity could be measured more directly than with LRTL. We first note that LRTL was introduced by prior work (Karazija et al., 2024; WHN), and the paper explicitly calls it a proxy metric (Metrics in Sec. 4). Prior work evaluates rigidity with LRTL, so we report it to compare fairly on their benchmark and with their metrics. As we also state in the paper, rigidity is a 3D invariant, while the model predicts 2D trajectories. Directly evaluating rigidity is impossible from 2D outputs alone, since the image-plane projection of a rigid motion is not itself rigid. On real videos it is even harder, because they come without a ground-truth 3D reconstruction.

Your concern that rigidity could be evaluated more directly is nevertheless valid. We therefore introduce a direct measure based on 3D reprojection, the 3D rigid fit.

3D rigid fit (Metrics in Sec. 4; App. D.2, new). Given the 3D positions of the query points in the input frame and the camera, both provided by the simulator, we fit, for every object and frame, the rigid transform of its 3D points whose projection is closest to the generated trajectories. We report the mean reprojection distance that remains, in pixels. It is zero for any rigid motion. Like LRTL, the 3D rigid fit is zero for trajectories that do not move. We therefore also report the object motion, the root-mean-square displacement of the points on objects between consecutive frames, which should match that of the ground truth.

Results on synthetic data (Table 2; Sec. 4.1). The 3D rigid fit confirms that JtT captures the rigidity invariant best. Every JtT variant generates motion closer to true rigid motion than any prior work (5.95–7.31 px vs. 7.45–15.81 px), while generating diverse, non-static trajectories (object motion 6.47–8.24 px per frame).

Table 2. Motion generation quality on Kubric-A, following WHN (the last two columns are new). Prior methods: single checkpoint; JtT: Average (Best) of the last five checkpoints. 3D rigid fit in px; object motion in px per frame (not an error: closer to GT is better; JtT: average). GPTF has no released samples to score. †: reimplementation; *: estimated.

ModelSizeFVMD (D) ↓FVMD (S) ↓Best-of-K ↓LRTL (ALL) ↓3D rigid fit ↓Object motion
GT—4,9418,845110.856.680.008.58
Track2Act (from WHN)600M16,73522,509250.8015.8015.816.65
WHN (L, latent-space)250M13,74517,838127.0014.107.456.16
WHN (B, coordinate-space)50M14,51918,551137.8023.709.166.32
MotionForecasting†100M20,17826,379186.478.2013.416.31
GPTF30M*—17,78695.6014.60——
JtT
S (Sparse)50M12,546 (12,238)16,769 (16,351)160.25 (129.77)7.69 (6.91)7.31 (5.54)8.24
B (Sparse)200M13,514 (11,812)17,716 (15,829)124.37 (115.24)6.67 (6.13)5.95 (4.85)6.85
S (Dense)55M11,807 (11,591)15,695 (15,452)135.85 (128.69)6.84 (6.64)7.21 (6.71)6.63
B (Dense)230M13,266 (12,333)17,437 (15,958)130.44 (118.76)6.80 (6.13)6.09 (5.67)6.47

In summary, we extended all our main analysis on both synthetic and real data with the 3D rigid fit, which we believe measures rigidity directly. On Kubric-A, it is in Table 2, discussed in Sec. 4.1. More details are in the appendix. All changes are in blue.

Comments on the paper

1 comment

Dear Reviewers and AC,

Thank you for your time and constructive feedback. Below we summarize our main responses. We have already updated the paper and the supplementary material, and all changes in the paper are in blue.

The discussion. We had a productive discussion with two of the three reviewers. Reviewer P1tL wrote that our experiments address their concerns and that the clarification of novelty is convincing, and they raised their score. Reviewer VfCc wrote that our results "fully addressed" their concerns. Reviewer gmWH, who gave the only negative rating (weak reject), did not take part in the discussion. We posted a detailed response on November 16 and a reminder on November 18, but have received no reply. We believe that points 1–3 below resolve all of their concerns.

Strengths noted by the reviewers.

  • A simple, small model. JtT needs no trajectory VAE, auxiliary losses, rollout fine-tuning, or explicit rigidity constraints (gmWH, P1tL).
  • Flexible queries. The same model handles arbitrary query patterns and dense trajectory generation (gmWH).
  • Strong results. JtT improves convincingly over prior work (VfCc). For example, JtT-S (Dense) lowers LRTL from 14.10 to 6.84 compared with WHN-L, a model more than four times larger (gmWH).
  • A systematic study of the design space. The study of parameterization, timestep sampling, and model scale gives interesting empirical findings (VfCc) and has significant research value (gmWH). The experiments are solid, with comprehensive ablations (VfCc), and the ablations are easy to follow, with consistent effects of velocities and of RGB at the query point (P1tL).
  • Transparent analysis of training dynamics. Showing how losses and metrics move over training is useful for anyone working on this benchmark (P1tL). The finding that rigidity and diversity do not improve together gives valuable insights for model selection (gmWH) and highlights important limitations of current evaluation practices (VfCc).
  • Clear presentation. Reviewers find the paper clear and easy to follow (gmWH, VfCc).

Weaknesses raised, and how we addressed them.

  1. Quantitative evaluation on real data (gmWH, P1tL, VfCc). The submission showed real videos only qualitatively, because real data (e.g. Physics 101) lacks the annotations that a rigorous evaluation of rigidity (and distributional fidelity) needs. To address this, we pseudo-labeled all 1,456 validation videos with instance masks and depth (App. D.2), which we plan to release, and evaluated JtT on them (Sec. 4.2). JtT sets the state of the art there. Its mean MSE is 17% below the best prior method, its FVMD is 22% below that of WHN, and its motion on rigid objects is 2.3× closer to rigid motion. JtT also sets the state of the art on forecasting animal motion in the wild, with and without drag conditioning (App. I).
  2. Direct measures of rigidity and diversity (gmWH). LRTL and FVMD come from prior work. We added direct measures. The 3D rigid fit measures rigidity by 3D reprojection, and every JtT variant is closer to rigid motion than any prior method (App. D.2). Precision, recall, and sample variance against the 64 ground-truth futures of each image confirm that the larger models collapse to a single future (App. D.1, Fig. 3).
  3. LRTL could favor static trajectories (gmWH). We analyzed LRTL by how far objects move (App. D.3). JtT moves objects further than prior methods. Its gain does not come from less motion.
  4. Why velocities give rigid motion (P1tL). A new appendix answers this with an ablation of 16 training runs (App. J). Velocity noise exposes rigidity better in the noised input, and the velocity loss acts as a smoothness regularizer. Only together do they give samples that are rigid and diverse.
  5. Robustness of the RGB gains (P1tL). We added these ablations. RGB at the query point helps on real videos and on the more complex Kubric-D, even at low color contrast (App. C.1). P1tL wrote that these experiments address their concerns.
  6. Novelty of the architecture (P1tL). We clarified three differences from Perceiver IO and CoTracker3: JtT is generative, a token holds a whole trajectory, and its support set is a random subset of the queries rather than fixed learned tokens. P1tL found this convincing. A new study (Table 30 in App. C.3) shows that JtT decodes 1,856 query points through only 32-60 support points while remaining state of the art.
  7. Articulated and non-rigid motion, and drag (VfCc). Physics 101 includes non-rigid bodies such as water, arms, hands, springs and soft bodies. On animals, JtT turns a single drag into the motion of the whole animal, and it applies zero-shot to robots (App. I). VfCc found the improvement significant and is convinced of applications such as drag-to-video.
  8. Length of the introduction (P1tL). We shortened it and updated the paper.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.