- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents JtT, a diffusion transformer for predicting diverse and physically plausible point trajectories from a single image. The method combines velocity-based trajectory prediction, full-resolution RGB conditioning, and a support/non-support token architecture for flexible and dense queries. The paper also investigates the trade-off between rigidity and diversity during training.
Critical strengths
I find several strengths of this work:
-
The paper provides a systematic investigation of important design choices in trajectory diffusion, with several interesting empirical findings.
-
The experiments are solid, with comprehensive ablations and convincing improvements over existing methods.
-
The analysis of the rigidity-diversity trade-off is insightful and highlights important limitations of existing evaluation practices.
Critical weaknesses
The quantitative evaluation is primarily conducted on synthetic Kubric datasets. Although qualitative results are provided on real-world datasets, the generalization to complex in-the-wild scenes, particularly those involving articulated or non-rigid motion, remains unclear.
Questions for authors
- How well does JtT generalize to complex in-the-wild scenarios? Is the method primarily effective for rigid objects, or can it also model articulated and deformable motion?
- Could JtT be extended to downstream applications such as drag-to-video generation, where sparse user-provided trajectories are used to infer dense, physically plausible motion?
6 comments
Real videos with rigid and non-rigid bodies
We added a quantitative evaluation on real videos (Sec. 4.2, new). We compare JtT with prior work (WHN and GPTF) on real videos from Physics 101. They contain simple physical experiments in five scenarios. We evaluate on the 1,456 held-out validation videos. Following prior work, we report the MSE of a single sample in each scenario. This evaluation assesses neither distributional fidelity nor rigidity. We therefore extend it with Best-of-16, FVMD, LRTL, the 3D rigid fit, and the object motion. To measure rigidity, we pseudo-label instance masks and depth of the rigid objects (App. D.2, new).
Results. JtT sets the state of the art on this benchmark. It outperforms prior work in pointwise accuracy, distributional fidelity, and physical plausibility (Table 4, new). It achieves the lowest mean MSE, 17% below the best prior method (GPTF), and the lowest MSE in four of the five scenarios. It learns rigid motion particularly well. Its mean MSE is 34% below that of any prior method in fall, multi, and ramp, and its motion on rigid objects is 2.3× closer to rigid motion than that of WHN. It also lowers FVMD by 22% compared with WHN.
Non-rigid bodies. Physics 101 is not limited to rigid bodies. Its videos also show:
- the hand of the experimenter, which releases the object;
- water, into which the object drops (liquid);
- a coil spring, on which the object hangs, often inside a plastic bag (spring);
- soft objects made of dough or rubber (fall, multi, and ramp).
In the two scenarios built around non-rigid bodies, JtT has the lowest MSE of all methods in liquid, and in spring it improves on WHN and WAN, while GPTF is better (Table 4a). Qualitatively, JtT captures the interaction between rigid and non-rigid bodies, such as a hand releasing an object or a block entering water. It also reflects different kinds of forces acting on rigid bodies, such as gravity and buoyancy on a wooden block dropped into water (App. G).
Table 4 (new). Results on real videos (Physics 101). (a) MSE of a single sample, with prior work as reported in WHN and GPTF. (b) Our extended evaluation. LRTL, 3D rigid fit (px), and object motion (px per frame) are computed on the rigid objects, and the first row scores the ground-truth tracks.
| (a) MSE (px²) ↓ | Fall | Liquid | Multi | Ramp | Spring | Mean |
|---|---|---|---|---|---|---|
| WAN (1.3B) | 16.05 | 4.48 | 21.88 | 37.53 | 70.48 | 30.08 |
| WHN | 19.78 | 6.00 | 15.65 | 36.35 | 65.31 | 28.62 |
| GPTF | 27.24 | 5.78 | 17.25 | 31.01 | 25.76 | 21.41 |
| JtT-B (Dense) | 14.04 (13.94) | 3.71 (3.61) | 12.53 (12.28) | 20.62 (20.50) | 38.18 (37.94) | 17.82 (17.78) |
| (b) Extended | Best-of-16 ↓ | FVMD ↓ | LRTL ↓ | 3D rigid fit ↓ | Object motion |
|---|---|---|---|---|---|
| Ground truth videos | — | — | 4.20 | 0.61 | 8.27 |
| WHN | 23.94 | 8,268 | 10.28 | 3.19 | 7.38 |
| JtT-B (Dense) | 15.22 (15.17) | 6,484 (6,467) | 3.57 (3.52) | 1.38 (1.35) | 7.53 |
Articulated motion in the wild, with and without drag conditioning
To address both of your questions, we move beyond Kubric and Physics 101. We show that JtT sets the state of the art on forecasting animal motion in the wild, both without and with drag conditioning. Since downstream applications are not the primary scope of our work, we added these results to a new appendix on downstream applications (App. I, new), with a pointer in Sec. 4.2. We copy its key parts below.
Task. We consider the task of forecasting animal motion (Thakkar et al., 2026). Given the image and the first four steps of the point trajectories of up to 500 query points on an animal, the task is to forecast the remaining 28 steps of each trajectory, and whether each query point is visible at each step. The input frame is cropped to 256×256 around the animal. The forecast can also be prompted with a motion vector, which gives the direction and distance the animal moves (arrows in Figs. 37 to 39). It is the mean displacement of the points from their first to their last visible step. At inference, any vector can be given, for example the opposite of the observed motion or twice its distance (Table 26). The motion vector thus acts as a single drag for the whole animal.
Data. We use the official MammalMotion data and annotations released by Thakkar et al. (2026). The training split has 1.6M windows, each having 32 timesteps at 15 fps.
Adapting JtT. We use JtT-B (Dense) with the architecture of Sec. 3.3 and per-frame velocities as the diffusion target (Sec. 3.1). Four changes adapt it to the task:
- Observed steps. The observed steps of each trajectory stay clean in its query token, and noise is added only to the 28 future steps.
- Motion vector. A linear layer embeds the motion vector, and we add its output to the timestep embedding, which modulates every block. We drop the motion vector for 30% of the training examples, so that a single model forecasts with and without it.
- Visibility. We generate the visibility of each point as a third channel, next to the two coordinates of its velocity.
- Image features. We use the same frozen DINOv3 ViT-L/16 encoder as MotionForecasting, but with features from layers 5, 11, 17, and 23. We also use a 7×7 RGB patch at the query points (App. C.1.2).
Evaluation. We use the protocol and metrics of MotionForecasting.
Results. JtT sets a new state of the art on the MammalMotion benchmark. It outperforms MotionForecasting in every metric, both without the motion vector and with it as drag conditioning (Tables 24 and 25). The largest gains are in distributional fidelity. Notably, without the motion vector (the setting closest to that of Sec. 3), JtT lowers FD(V) and FD(A) by 75% and FVMD by 63% on all species. Given the motion vector, JtT also outperforms MotionForecasting, slightly on all species and by 26% in ADE and FDE on Panthera. Qualitatively, without the motion vector, the samples of JtT are more diverse than those of MotionForecasting (Fig. 36). We also confirm numerically that JtT successfully incorporates drag conditioning (details in Table 26 of App. I.1, new).
Table 24 (new). Animal motion without the motion vector (MammalMotion). Distributional fidelity, with the metrics of MotionForecasting (lower is better). JtT: Average (Best) of the last five checkpoints.
| Set | Motion | Model | FD(V) ↓ | FD(A) ↓ | FVMD ↓ |
|---|---|---|---|---|---|
| All Data | High | MotionForecasting | 2.360 | 2.384 | 35.23 |
| JtT-B (Dense) | 0.961 (0.942) | 0.970 (0.960) | 20.18 (20.03) | ||
| Combined | MotionForecasting | 0.585 | 0.619 | 12.32 | |
| JtT-B (Dense) | 0.148 (0.146) | 0.155 (0.153) | 4.52 (4.48) | ||
| Panthera | High | MotionForecasting | 10.759 | 7.088 | 69.06 |
| JtT-B (Dense) | 5.861 (5.529) | 5.021 (4.937) | 46.47 (44.90) | ||
| Combined | MotionForecasting | 1.994 | 1.426 | 20.98 | |
| JtT-B (Dense) | 0.947 (0.891) | 0.807 (0.795) | 11.62 (11.12) |
Table 25 (new). Animal motion given the motion vector (MammalMotion). Best of five samples, with the metrics of MotionForecasting. JtT: Average (Best) of the last five checkpoints. ADE and FDE are in crop widths, and PWT is in %.
| Set | Motion | Model | ADE ↓ | FDE ↓ | VMD ↓ | PWT ↑ |
|---|---|---|---|---|---|---|
| All Data | High | MotionForecasting | 0.0461 | 0.0624 | 5.40 | 37.47 |
| JtT-B (Dense) | 0.0441 (0.0440) | 0.0595 (0.0593) | 5.39 (5.38) | 38.07 (38.10) | ||
| Combined | MotionForecasting | 0.0216 | 0.0293 | 3.99 | 63.90 | |
| JtT-B (Dense) | 0.0211 (0.0210) | 0.0285 (0.0284) | 3.97 (3.97) | 64.13 (64.16) | ||
| Panthera | High | MotionForecasting | 0.0724 | 0.1137 | 5.23 | 34.99 |
| JtT-B (Dense) | 0.0473 (0.0472) | 0.0742 (0.0742) | 5.21 (5.21) | 37.25 (37.35) | ||
| Combined | MotionForecasting | 0.0291 | 0.0456 | 3.86 | 64.15 | |
| JtT-B (Dense) | 0.0214 (0.0213) | 0.0337 (0.0337) | 3.83 (3.83) | 64.98 (65.02) |
Thank you for the positive assessment of our work! We are glad that you find our systematic investigation of the design choices of trajectory diffusion valuable, and our experiments solid, with comprehensive ablations and convincing improvements over existing methods. We are also glad that you find our analysis of the rigidity-diversity trade-off insightful, in particular for evaluation practices.
Non-rigid motion is outside the scope of our work. Our primary contribution is a method that samples diverse motion and captures a physical invariant, rigidity, purely from data, without building it into the architecture. We therefore focus on rigid motion, where JtT sets the state of the art in distributional fidelity and rigidity, on synthetic data (Sec. 4.1) and on real videos (Sec. 4.2, new).
We agree that the submission evaluated JtT quantitatively only on synthetic data, and showed real and non-rigid motion only qualitatively. We therefore extended our evaluation to such data. We address all of your concerns, in three parts:
Please take a look at the new appendix on downstream applications (App. I, new), which discusses these results at length. It compares samples of JtT and MotionForecasting without and with the motion vector (Figs. 36 and 37 in App. I.1.1). It also shows forecasts for robots, birds, a tortoise, and a crab (Figs. 38 and 39 in App. I.1.2).
We have already updated the paper, and all changes are in blue. Due to character limitations, our response will consist of several separate comments.